Edge 440: Interested in AI Evaluation? Meet Microsoft's EUREKA
Was this email forwarded to you? Sign up here Edge 440: Interested in AI Evaluation? Meet Microsoft's EUREKAThe framework provides an evaluation pipeline as well as a collection of benchmarks for evaluating language and vision capabilities.Evaluating foundation models is one of the next frontiers of the space. In the last few years, foundation models have completely outpaced the benchmarks and today, we have a handful of benchmarks that remain relevant. Additionally, the industry lacks comprehensive evaluation frameworks for the evaluation of foundation models. This is the challenge that Microsoft Research tackled in a recent paper with a new evaluation framework called EUREKA. EUREKA is a reusable, open evaluation framework designed to standardize evaluations of large foundation models (LFMs). The framework goes beyond single-score reporting and rankings to offer a more comprehensive analysis of LFM capabilities. EUREKA achieves this through:... Subscribe to TheSequence to unlock the rest.Become a paying subscriber of TheSequence to get access to this post and other subscriber-only content. A subscription gets you:
|
Older messages
Edge 437: Inside BlackMamba, One of the Most Important SSM Models Ever Created
Tuesday, October 8, 2024
The model combines SSMs, MoEs in a single architecture. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Meta Gets Into AI Video Generation
Sunday, October 6, 2024
Movie Gen promises to generate high fidelity videos with synchronized audio. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
📝 Guest Post: Multimodal Retrieval –Bridging the Gap Between Language and Diverse Data Types*
Friday, October 4, 2024
Generative AI has recently witnessed an exciting development: using language to understand images, video, audio, molecules, time-series, and other "modalities." Multimodal retrieval
Edge 436: Salesforce's xLAM is a New Model for Agentic Tasks
Thursday, October 3, 2024
The new model excels in tasls such as function calling, tool integration and planning. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Edge 435: Learn About Hungry Hungry Hippos and SSMs
Tuesday, October 1, 2024
One of the most important layers of state space models. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
You Might Also Like
Recording: 'Data Storytelling: What Organizations Need to Know Going Into 2025'
Friday, November 22, 2024
Thank you for your interest in our latest webinar. As promised here is your recording of the event. View email in browser Recording Now Available Thank you for your interest in receiving a recording of
💻 Issue 437 - Introducing local Azure Service Bus Emulator
Thursday, November 21, 2024
This week's Awesome .NET Weekly Read this email on the Web The Awesome .NET Weekly Issue » 437 Release Date Nov 21, 2024 Your weekly report of the most popular .NET news, articles and projects
💎 Issue 444 - Why did people rub snow on frozen feet? (2017)
Thursday, November 21, 2024
This week's Awesome Ruby Newsletter Read this email on the Web The Awesome Ruby Newsletter Issue » 444 Release Date Nov 21, 2024 Your weekly report of the most popular Ruby news, articles and
💻 Issue 444 - JavaScript Dos and Donts
Thursday, November 21, 2024
This week's Awesome JavaScript Weekly Read this email on the Web The Awesome JavaScript Weekly Issue » 444 Release Date Nov 21, 2024 Your weekly report of the most popular JavaScript news, articles
📱 Issue 438 - Reverse Engineering iOS 18 Inactivity Reboot
Thursday, November 21, 2024
This week's Awesome iOS Weekly Read this email on the Web The Awesome iOS Weekly Issue » 438 Release Date Nov 21, 2024 Your weekly report of the most popular iOS news, articles and projects Popular
💻 Issue 362 - React Anti-Pattern: Stop Passing Setters Down the Components Tree
Thursday, November 21, 2024
This week's Awesome React Weekly Read this email on the Web The Awesome React Weekly Issue » 362 Release Date Nov 21, 2024 Your weekly report of the most popular React news, articles and projects
💻 Issue 444 - Building simple event-driven applications with Pub/Sub
Thursday, November 21, 2024
This week's Awesome Node.js Weekly Read this email on the Web The Awesome Node.js Weekly Issue » 444 Release Date Nov 21, 2024 Your weekly report of the most popular Node.js news, articles and
📱 Issue 441 - Shift Left Is the Tip of the Iceberg
Thursday, November 21, 2024
This week's Awesome Swift Weekly Read this email on the Web The Awesome Swift Weekly Issue » 441 Release Date Nov 21, 2024 Your weekly report of the most popular Swift news, articles and projects
💻 Issue 439 - Async/Await Is Real And Can Hurt You
Thursday, November 21, 2024
This week's Awesome Rust Weekly Read this email on the Web The Awesome Rust Weekly Issue » 439 Release Date Nov 21, 2024 Your weekly report of the most popular Rust news, articles and projects
📲 Why I Ditched Linux for Samsung DeX — Buy This Instead of a Gaming Headset
Thursday, November 21, 2024
Also: Taking Instagram Stories to the Next Level, and More! How-To Geek Logo November 21, 2024 Did You Know Thurl Ravenscroft was both the voice behind the Christmas song "You're a Mean One,