Maybe Two Big Research Breakthroughs or Maybe Nothing
Was this email forwarded to you? Sign up here Maybe Two Big Research Breakthroughs or Maybe NothingMulti-token prediction and a multi-layer perceptron alternative.Next Week in The Sequence:
You can subscribed to The Sequence below:📝 Editorial: Maybe Two Big Research Breakthroughs or Maybe NothingResearch breakthroughs always command a lot of attention in the generative AI space, as they can drive a new wave of innovation for foundation models. However, most of the so-called breakthroughs in the research sphere rarely make it into real implementations, so it's hard to determine whether a new breakthrough will really be influential or not. Last week was particularly interesting because we saw two papers that, on the surface, seem to be quite a big deal for generative AI models, but it's quite hard to determine if they are ready for prime-time. 1. Multi-Token Predictions In a paper titled "Better & Faster Large Language Models via Multi-token Prediction," several AI labs, including Meta AI, proposed a method for multi-token prediction in LLMs. It is a known fact that LLMs work by predicting one token at a time, so this seems like a big leap forward. In theory, the method works by predicting 'n' tokens using 'n' independent output heads. One of the tests with a 13B parameter model seems to solve 12% more problems on the HumalEval dataset than next-token model equivalents. One immediate limitation is that this only seems to be effective in very large models but still seems quite an improvement. 2. MLP Alternative The multi-layer perceptron (MLP) is, arguably, the most important algorithm in the history of deep learning. In a paper titled "KAN: Kolmogorov-Arnold Networks (KANs)," researchers from MIT proposed an audacious alternative to MLP. The main innovation relies on using learnable activation functions on weights instead of the fixed activation functions on nodes in MLP. While conceptually sound, KANs have only been evaluated in a few targeted use cases. Both the multi-token prediction and the KAN paper are clear indicators of the amazing speed of progress in AI research. Both papers are trying to disrupt established techniques. They both seem to be relevant breakthroughs on the surface. Or maybe not 😊" 🔥 Announcing Galileo Protect: Real-Time Hallucination FirewallCan you stop hallucinations in real-time? We’re excited to support Galileo Protect – an advanced GenAI firewall that intercepts hallucinations, prompt attacks, security threats, and more! See Galileo Protect live in action and learn: 🔎 ML ResearchLLM Juries vs. JudgesCohere research published a paper proposing a technique to evaluate LLMs using a panel of diverse models. The method looks to mitigate the bias of single-model evaluation techniques and shows a strong performance across different evaluation benchmarks —> Read more. Iterative ReasoningResearchers from MetaAI and New York University published a paper outlining an iteractive reasoning technique for LLMs. The essence of the method is to optimize across different chain-of-thought(CoT) options to lead to the correct answer —> Read more. Gemini in MedicineGoogle DeepMind published a paper introducing Med-Gemini, a multimodal version of Gemini specialized in medical tasks. Med-Gemini was evaluated across 14 medical benchmarks outperforming GPT-4 in several of them —> Read more. Multi-Token PredictionResearchers from Meta AI, CERMICS Ecole des Ponts Paris Tech and LISN Université Paris-Saclay published a paper proposing an intriguing technique for multi-token prediction in LLMs. At each position of the training corpus, the method can predict a number of future tokens —> Read more. MLP PredictionResearchers from Massachusetts Institute of Technology, California Institute of Technology and Northeastern University published a paper introducing e KolmogorovArnold Networks (KANs), promising alternatives to Multi-Layer Perceptrons (MLPs). The core idea is to switch from the MLP’s fixed actiavation functions on neurons to learnable activation functions on weights —> Read more. Math Benchmark ContaminationResearchers from Scale AI created a new benchmark for elementary math reasoning similar to GSM8k. The results show that several top models such as Mistral and Phi might be overfitting for existing math benchmarks while others such as Gemini, GPT-4 and Claude aren’t —> Read more. 🤖 Cool AI Tech ReleasesXTunerXtuner is a new toolkit for fine-tuning LLMs that is gaining rapid traction —> Read more. Amazon QAmazon Q, its Copilot alternative, hit general availability —> Read more. Claude AppAnthropic released a new iOS App for Claude as well as a new teams subscription plan. Jamba Instruct AI21 released Jamba Instruct, an instruction fine-tuned version of their Jamba model that combines SSMs and transformers —> Read more. 🛠 Real World MLAirbnb BrandometerAirbnb discusses their AI language techniques to understand brand perception in social media channels —> Read more. Amazon AI-Enhanced Catalogue DataAmazon dives into the ML techniques used to evaluate the effectiveness of AI-enhaced catalogue data —> Read more. 📡AI Radar
You’re on the free list for TheSequence Scope and TheSequence Chat. For the full experience, become a paying subscriber to TheSequence Edge. Trusted by thousands of subscribers from the leading AI labs and universities. |
Older messages
Edge 392: Meet RAFT: UC Berkeley's New Method to Improve RAG Patterns in LLMs
Thursday, May 2, 2024
The method brings the best of RAG and supervised fine tuning. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Edge 391: Autonomous Agents and LLM Function Calling
Tuesday, April 30, 2024
LLMs that invoke external functions, UC Berkeley's LLM Compiler and the Phidata framework. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Nobody Likes a Know-It-All: Smaller LLMs are Gaining Momentum
Sunday, April 28, 2024
Phi-3 and OpenELM, two major small model releases this week. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Edge 390: Diving Into Databricks' DBRX: One of the Most Impressive Open Source LLMs Released Recently
Thursday, April 25, 2024
The model uses an MoE architecture which exhibits remarkable perfromance on a relatively small budget. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Edge 389: Understanding Large Action Models
Tuesday, April 23, 2024
One of the most important concepts in autonomous agents. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
You Might Also Like
📧 EF Core Migrations: A Detailed Guide
Saturday, May 18, 2024
EF Core Migrations: A Detailed Guide Read on: my website / Read time: 10 minutes BROUGHT TO YOU BY Low-code Framework for .NET Devs Introducing Shesha, a brand new, open-source, low-code
Slack is under attack … and you don’t want that
Friday, May 17, 2024
Plus: OpenAI is not aligned with its Superalignment team View this email online in your browser By Christine Hall Friday, May 17, 2024 Good afternoon, and welcome back to TechCrunch PM. We made it to
Ilya Sutskever leaves OpenAI - Weekly News Roundup - Issue #467
Friday, May 17, 2024
Plus: Apple is close to using ChatGPT; Microsoft builds its own LLM; China is sending a humanoid robot to space; lab-grown meat is on shelves but there is a catch; hybrid mouse/rat brains; and more! ͏
SWLW #599: Surfing through trade-offs, How to do hard things, and more.
Friday, May 17, 2024
Weekly articles & videos about people, culture and leadership: everything you need to design the org that makes the product. A weekly newsletter by Oren Ellenbogen with the best content I found
💾 There Will Never Be Another Windows XP — Why Ray Tracing is a Big Deal in Gaming
Friday, May 17, 2024
Also: What to Know About Google's Project Astra, and More! How-To Geek Logo May 17, 2024 Did You Know The very first mass-manufactured drinking straw was made of paper coated in wax; the straw was
It's the dawning of the age of AI
Friday, May 17, 2024
Plus: Musk is raging against the machine View this email online in your browser By Haje Jan Kamps Friday, May 17, 2024 Image Credits: Google Welcome to Startups Weekly — Haje's weekly recap of
Daily Coding Problem: Problem #1444 [Medium]
Friday, May 17, 2024
Daily Coding Problem Good morning! Here's your coding interview problem for today. This problem was asked by Yahoo. Recall that a full binary tree is one in which each node is either a leaf node,
(Not) Sent From My iPad
Friday, May 17, 2024
The future of computing remains frustrating (Not) Sent From My iPad By MG Siegler • 17 May 2024 View in browser View in browser I tried. I really did. I tried to put together and send this newsletter
iOS Dev Weekly - Issue 661
Friday, May 17, 2024
What's the word on everyone's lips? 🅰️👁️ View on the Web Archives ISSUE 661 May 17th 2024 Comment Did you catch Google I/O this week? It's Always Interesting to see what the Android
Your Google Play recap from I/O 2024
Friday, May 17, 2024
Check out all of our latest updates and announcements Email not displaying correctly? View it online May 2024 Google Play at I/O 2024 Check out the Google Play keynote to discover the latest products