Edge 439: SSMs with Attention, Understanding Zamba
Was this email forwarded to you? Sign up here Edge 439: SSMs with Attention, Understanding ZambaCombining the best of SSMs and transformers in a single architecture.
In this issue:
💡 ML Concept of the Day: SSMs with Attention, Understanding ZambaIn this series, we have explored state space models(SSMs) as an alternative to transformer architectures. However, there is a lot of merit in building hybrid architectures that combine aspects of transformers and SSMs. AI21’s Jamba is an example of this architecture which have gotten some attention. One of the models that outlined this idea is Zamba created by Zyphra, one of the most innovative AI labs in the market. Zamba's new architecture blends Mamba blocks with a globally shared attention layer applied after every six Mamba blocks. This hybrid setup enhances Zamba's ability to learn long-range dependencies and perform in-context learning more efficiently than traditional Mamba models, all while reducing computational demands during both training and inference compared to standard transformer models... Subscribe to TheSequence to unlock the rest.Become a paying subscriber of TheSequence to get access to this post and other subscriber-only content. A subscription gets you:
|
Older messages
Edge 440: Interested in AI Evaluation? Meet Microsoft's EUREKA
Sunday, October 20, 2024
The framework provides an evaluation pipeline as well as a collection of benchmarks for evaluating language and vision capabilities. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Edge 437: Inside BlackMamba, One of the Most Important SSM Models Ever Created
Tuesday, October 8, 2024
The model combines SSMs, MoEs in a single architecture. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Meta Gets Into AI Video Generation
Sunday, October 6, 2024
Movie Gen promises to generate high fidelity videos with synchronized audio. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
📝 Guest Post: Multimodal Retrieval –Bridging the Gap Between Language and Diverse Data Types*
Friday, October 4, 2024
Generative AI has recently witnessed an exciting development: using language to understand images, video, audio, molecules, time-series, and other "modalities." Multimodal retrieval
Edge 436: Salesforce's xLAM is a New Model for Agentic Tasks
Thursday, October 3, 2024
The new model excels in tasls such as function calling, tool integration and planning. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
You Might Also Like
Re: How to know if your data has been exposed
Monday, December 23, 2024
Imagine getting an instant notification if your SSN, credit card, or password has been exposed on the dark web — so you can take action immediately. Surfshark Alert does just that. It helps you stay
Christmas On Repeat 🎅
Monday, December 23, 2024
Christmas nostalgia is a hell of a drug. Here's a version for your browser. Hunting for the end of the long tail • December 22, 2024 Hey all, Ernie here with a refresh of a piece from our very
SRE Weekly Issue #456
Monday, December 23, 2024
View on sreweekly.com A message from our sponsor, FireHydrant: On-call during the holidays? Spend more time taking in some R&R and less getting paged. Let alerts make their rounds fairly with our
The Power of an Annual Review & Grammarly acquires Coda
Sunday, December 22, 2024
I am looking for my next role, Zen Browser got a fresh new look, Flipboard introduces Surf, Campsite shuts down, and a lot more in this week's issue of Creativerly. Creativerly The Power of an
Daily Coding Problem: Problem #1645 [Hard]
Sunday, December 22, 2024
Daily Coding Problem Good morning! Here's your coding interview problem for today. This problem was asked by Facebook. Implement regular expression matching with the following special characters: .
PD#606 How concurrecy works: A visual guide
Sunday, December 22, 2024
A programmer had a problem. "I'll solve it with threads!". has Now problems. two he ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
RD#486 (React) Things I Regret Not Knowing Earlier
Sunday, December 22, 2024
Keep coding, stay curious, and remember—you've got this
🎶 GIFs Are Neat, but I Want Clips With Sound — Your Own Linux Desktop in the Cloud
Sunday, December 22, 2024
Also: 9 Games That Were Truly Ahead of Their Time, and More! How-To Geek Logo December 22, 2024 Did You Know Dextrose is another name for glucose, so if you see it listed prominently on the ingredients
o3—the new state-of-the-art reasoning model - Sync #498
Sunday, December 22, 2024
Plus: Nvidia's new tiny AI supercomputer; Veo 2 and Imagen 3; Google and Microsoft release reasoning models; Waymo to begin testing in Tokyo; Apptronik partners with DeepMind; and more! ͏ ͏ ͏ ͏ ͏ ͏
Sunday Digest | Featuring 'The World’s 20 Largest Economies, by GDP (PPP)' 📊
Sunday, December 22, 2024
Every visualization published this week, in one place. Dec 22, 2024 | View Online | Subscribe | VC+ | Download Our App Hello, welcome to your Sunday Digest. This week, we visualized public debt by