Edge 455: Building Smaller Foundation Models Using Graph-Based Distillation
Was this email forwarded to you? Sign up here Edge 455: Building Smaller Foundation Models Using Graph-Based DistillationDiving into one of the most sophisticated distillation methods in the gen AI space.
In this issue:
💡 ML Concept of the Day: Understanding Graph-Based DistillationThroughout this series, we have focused on traditional teacher-student distillation methods which focus on individual data units, such as matching output probabilities or feature transformations between the teacher(TN) and student networks(SN) . While unquestionably effective, these methods often overlook the relationships between data points—a critical factor in helping SNs develop effective data embeddings. Graph-based knowledge distillation (GKD) is a cutting-edge technique designed to enhance the performance of small student networks by transferring relational knowledge from a larger teacher network. The key concept behind GKD is to use attention networks, particularly multi-head attention (MHA) networks. These networks build a graph representation that captures relationships between feature vectors. Here’s how it works:... Subscribe to TheSequence to unlock the rest.Become a paying subscriber of TheSequence to get access to this post and other subscriber-only content. A subscription gets you:
|
Older messages
The Sequence Chat: The Transition that Changes Everything. From Pretraining to Post-Training in Foundation Models
Tuesday, December 10, 2024
One of the most impactful transitions in the generative AI space ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Edge 454: Meet Magenctic-One, Microsoft's New Framework for Building Multi Agent Systems
Tuesday, December 10, 2024
Built on AutoGen, the framework is designed for agents that collaborate in open ended tasks. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
World Models are Coming and They are Awesome
Tuesday, December 10, 2024
Two amazing world models were released this week. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
📝 Guest Post: Advanced RAG Techniques: Bridging Text and Visuals for More Accurate Responses*
Tuesday, December 10, 2024
In this guest post, Fendy Feng from ZIlliz explores how RAG works, RAG challenges, and advanced RAG techniques like Small to Slide RAG and ColPali. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Edge 453: Distillation Across Different Modalities
Tuesday, December 3, 2024
Cross modal distillation is one of the most interesting distillation methods of the new generation. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
You Might Also Like
Reach More Readers, newsletterest1 – BOOST Your Story on HackerNoon🔥
Wednesday, December 11, 2024
Get Your Story Featured on the Homepage and in The HackerNoon Newsletter ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Top Tech Deals 👀 $109 Robot Vacuum, Google Pixel Sale, Anker Power Bank, and More
Wednesday, December 11, 2024
Grab a new Pixel phone or tablet, stocking stuffers, and other goodies. How-To Geek Logo December 11, 2024 Top Tech Deals: $109 Robot Vacuum, Google Pixel Sale, Anker Power Bank, and More Grab a new
Hurry, newsletterest1! Less Than a Week Left to Compete for $2,500 in the AI Writing Contest 🏃
Wednesday, December 11, 2024
Start drafting your entry today! ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
DePIN On Ethereum: Redefining Coordination Systems
Wednesday, December 11, 2024
Top Tech Content sent at Noon! Boost Your Article on HackerNoon for $159.99! Read this email in your browser How are you, @newsletterest1? 🪐 What's happening in tech today, December 11, 2024? The
Post from Syncfusion Blogs on 12/11/2024
Wednesday, December 11, 2024
New blogs from Syncfusion Building a Neumorphic UI with .NET MAUI Column Chart to Showcase Gen Z's Favourite Social Media Platforms By Dhanaraj Rajendran Learn to create a Neumorphic UI with
24 Hours Until Our 2025 Outlook Webinar – Register Now ⏰
Wednesday, December 11, 2024
Don't miss the key trends shaping 2025 with our free webinar on December 12th. View Online | Subscribe | Download Our App FREE WEBINAR - Tomorrow at 11am PST 2025 Outlook: Key Trends on Our Radar
⚙️ Another AI lawsuit
Wednesday, December 11, 2024
Plus: Tesla sued ... again
The most Windows-like Linux distro
Wednesday, December 11, 2024
iOS 18.2 arrives; AI moves undercover; Natural Cycles dupe -- ZDNET ZDNET Tech Today - US December 11, 2024 The default Wubuntu desktop. This Linux distro is so Windows-like, it even comes with
Your InfoSec Survival Guide
Wednesday, December 11, 2024
How to optimize your compliance practices through a continuous monitoring approach The Hacker News The InfoSec Survival Guide Today, security and compliance leaders are struggling under the pressure of
The Sequence Chat: The One Area in Which China can Dominate the US in the AI Race
Wednesday, December 11, 2024
Might come as a surprise. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏