TheSequence - Some Cool Details About Llama 3
Was this email forwarded to you? Sign up here Some Cool Details About Llama 3Solid performance, new tokenizer, fairly optimal training and other details about Meta AI's new model.Next Week in The Sequence:
You can subscribed to The Sequence below:📝 Editorial: Some Cool Details About Llama 3I had an editorial prepared for this week’s newsletter, but then Meta AI released Llama 3! Such are the times we live in. Generative AI is evolving on a weekly basis, and Llama 3 is one of the most anticipated releases of the past few months. Since the debut of the original version, Llama has become one of the foundational blocks of the open source generative AI space. I prefer to use the term "open models," given that these releases are not completely open source, but that’s just my preference. The release of Llama 3 builds on incredible momentum within the open model ecosystem and brings its own innovations. The 8B and 70B versions of Llama 3 are available, with a 400B version currently being trained. The Llama 3 architecture is based on a decoder-only model and includes a new, highly optimized 128k tokenizer. This is quite notable, given that, with few exceptions, most large language models simply reuse the same tokenizers. The new tokenizer leads to major performance gains. Another area of improvement in the architecture is the grouped query attention, which was already used in Llama 2 but has been enhanced for the larger models. Grouped query attention helps improve inference performance by caching key parameters. Additionally, the context window has also increased. Training is one area in which Llama 3 drastically improves over its predecessors. The model was trained on 15 trillion tokens, making the corpus quite large for an 8B parameter model, which speaks to the level of optimization Meta achieved in this release. It's interesting to note that only 5% of the training corpus consisted of non-English tokens. The training infrastructure utilized 16,000 GPUs, achieving a throughput of 400 TFLOPs, which is nothing short of monumental. Llama 3 is a very welcome addition to the open model generative AI stack. The initial benchmark results are quite impressive, and the 400B version could rival GPT-4. Distribution is one area where Meta excelled in this release, making Llama 3 available on all major machine learning platforms. It's been just a few hours, and we are already seeing open source innovations using Llama 3. The momentum in the generative AI open models space definitely continues, even if it forced me to rewrite the entire editorial. 😊 🔎 ML ResearchVASA-1Microsoft Research published a paper detailing VASA-1, a framework for generating talking faces from static images and audio clips. The model is able to generage facial gestures such as head or lip movements in a very expressive way —> Read more. ZambaZyphra published a paper introducing Zamba, a 7B SSM model. Zamba introduces a new architecture that combines Mamba blocks with attention layers which leads to high performance in training and inference with lower computational resources —> Read more. MEGALODONAI researchers from Meta and Carnegie Mellon University published a paper introducing MEGALODON, a new architecture that can scale to virutally unlimited context windows. As it names indicates, MEGALODON is based on the MEGA architecture with an improved gated attention mechanism —> Read more. SAMMOMicrosoft Research published a paper detailing Structure-Aware Multi-objective Metaprompt Optimization (SAMMO), a framework for prompt optimization. The framework is able to optimize prompts for scenarios such as RAG or instruction tuning —> Read more. Infini-AttentionGoogle Research published a paper introducing Infini-Attention, a method to scale the context window in transformer architectures to virtually unlimited levels. The method adds a compressive memory into the attention layer which allow to build long-term and masked-local attention into a single transformer block —> Read more. AI Agents EthicsGoogle DeepMind published a paper discussing ethical considerations in AI assistants. The paper cover aspects such as safety alingment, safety and misuse —> Read more. 🤖 Cool AI Tech ReleasesLlama 3Meta AI introduced the highly anticipated Llama 3 model —> Read more. Stable Diffusion 3Stability AI launched the APIs for Stable Diffusion 3 as part of its developer platform —> Read more. Reka CoreReka, an AI startup built by former DeepMind engineers, announced its Reka Core multimodal models —> Read more. OpenEQAMeta AI released OpenEQA, a benchmark for visual language model in physical environments —> Read more. Gemini CookbookGoogle open sourced the Gemini Cookbook, a series of examples for interacting with the Gemini API —> Read more. 🛠 Real World MLAI Privacy at SlackSlack discusses the architecture enabling privacy capabilities in its AI platform —> Read more. 📡AI Radar
You’re on the free list for TheSequence Scope and TheSequence Chat. For the full experience, become a paying subscriber to TheSequence Edge. Trusted by thousands of subscribers from the leading AI labs and universities. |
Older messages
Edge 388: Google DeepMind's SIMA can Follow Language Instructions in 3D Games Just Like Humans
Thursday, April 18, 2024
The AI agent represents a major improvement relative to expensive reinforcement learning methods. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Edge 387: Tool Learning in Autonomous Agents
Tuesday, April 16, 2024
Agents that master tools and APIs, UC Berkeley's Gorilla and Microsoft's TaskWeaver ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Neuro-Symbolic Models are Making a Comeback
Sunday, April 14, 2024
A new startup called Symbolica comes out of stealth with a very different value proposition. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Edge 386: Inside Yi, 01's Model Leading the Chinese LLM Movement
Saturday, April 13, 2024
Yi has achieved remarkable performance across language and image tasks. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Edge 385: The Two Big Schools for Building Autonomous Agents
Tuesday, April 9, 2024
Language or computer-vision based agents? ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
You Might Also Like
Ranked | The World's Top Media Franchises by All-Time Revenue 📊
Saturday, May 4, 2024
From Pokémon to Star Wars, some media franchises are globally recognizable. How do media franchises compare in terms of all-time revenue? View Online | Subscribe Presented by Voronoi: The App Where
Noonification: Read Code Like a Hacker With the SAST
Saturday, May 4, 2024
Top Tech Content sent at Noon! Get Algolia: AI Search that understands How are you, @newsletterest1? 🪐 What's happening in tech today, May 4, 2024? The HackerNoon Newsletter brings the HackerNoon
Weekend Reading — May the fourth
Saturday, May 4, 2024
This week we setup our new Minecraft server, play Spacewar, avoid burnout, wonder about Facebook AI spam, lose our passkeys, and claim stairs on the way back home. 😎 Labnotes (by Assaf Arkin) Weekend
Google lays off workers
Saturday, May 4, 2024
Plus: Tesla cans its Supercharger team and UnitedHealthcare reveals security lapses View this email online in your browser By Kyle Wiggers Saturday, May 4, 2024 Image Credits: Tomohiro Ohsumi / Getty
When It Rains, It Pours ☔
Saturday, May 4, 2024
Why the umbrella's design can't be beat. Here's a version for your browser. Hunting for the end of the long tail • May 04, 2024 Hey there, Ernie here with a refreshed piece about umbrellas
🐍 New Python tutorials on Real Python
Saturday, May 4, 2024
Hey there, There's always something going on over at realpython.com as far as Python tutorials go. Here's what you may have missed this past week: Python's unittest: Writing Unit Tests for
Microsoft Outlook Flaw Exploited by Russia's APT28 to Hack Czech, German Entities
Saturday, May 4, 2024
THN Daily Updates Newsletter cover Webinar -- Data Security is Different at the Petabyte Scale Discover the secrets to securing fast-moving, massive data sets with insights from industry titans
Stop spam texts today!
Saturday, May 4, 2024
Have you been receiving a ton of spam texts lately? In April alone, spammers sent out over 19.2 billion texts. Ever wonder where they get your info to begin with? Data brokers. They sell your address,
DeveloPassion's Newsletter #165 - She Said Yes ❤️
Saturday, May 4, 2024
Edition 165 of my newsletter, discussing Knowledge Management, Knowledge Work, Zen Productivity, Personal Organization, and more! Sébastien Dubois DeveloPassion's Newsletter DeveloPassion's
📧 Implementing API Gateway Authentication With YARP
Saturday, May 4, 2024
Implementing API Gateway Authentication With YARP Read on: my website / Read time: 5 minutes BROUGHT TO YOU BY Supercharging Development With AI and APIs Announcing Postman v11: Streamline API