Google Goes Small and Open Source with Gemma
Was this email forwarded to you? Sign up here Google Goes Small and Open Source with GemmaGemma is based on the core architecture powering Gemini.Next Week in The Sequence:
You can subscribe below!📝 Editorial: Google Goes Small and Open Source with GemmaGenerative AI is transforming everything, but it's hard to lead the revolution with 70B parameters at a time! LLMs feel magical until you spend weeks trying to fine-tune a massively large model. For companies to build their own generative AI models, the core architecture of these models needs to become smaller. Last year, we saw iterations of this concept with models such as Microsoft Phi 2, a 2 billion parameter model that was able to outperform much larger models in math and coding. Microsoft even coined a term for these types of models: small language models (SLMs). This week, Google jumped on the SLM train by releasing Gemma, a family of open-source SLMs based on the same core architecture that powers its marquee Gemini model. The release includes the pretrained and instruction-tuned versions of Gemma 2B and 7B. Additionally, the Gemma release provides native integration with Hugging Face, Kaggle, and Google Collab notebooks, as well as major frameworks such as TensorFlow, PyTorch, and JAX. Gemma was evaluated across several industry-leading benchmarks, surpassing models considerably larger in size. The Gemma release represents an interesting strategic move by Google. The tech giant is not only taking a position in the SLM space but also championing open-source efforts. This contrasts with the closed nature of its Gemini release. Open source is deeply ingrained in Google’s culture, and hopefully, we can see them push more generative AI efforts in this area. While massively large foundation models continue to achieve new milestones, the SLM revolution seems inevitable. Now, both Microsoft and Google have taken a position. The SLMs are coming! 📍 Announcing GenAI Productionize 2024 – the one and only event on productionizing enterprise GenAI!We invite you to see how LlamaIndex, Coinbase, LinkedIn, Comcast, Proctor & Gamble, Roblox, Databricks, JPMorgan Chase, Fidelity, Chegg and others get their GenAI apps into production, including practical strategies for governance, evaluation, and monitoring. Register for GenAI Productionize 2024 to learn:
🔎 ML ResearchSoraOpenAI published some technical details about the architecture of Sora, its ground breaking text-to-video model. Sora is based on diffusion models that focus on predicting the next “patch” or visual representaions in dataset —> Read more. Large World ModelResearchers from UC Berkeley including robotics legend Pieter Abbeel published a paper detailing large world model(LWM), a family of large-context, multimodal models. LVM uses a techinque called RingAttention to scale the context windows to about one million token —> Read more. User-LLMGoogle Research published a paper detailing User-LLM, a framework that leverages embeddings to contextualize LLMs. The core idea is that these embeddings cpature user preferences over time and can personalize the interaction with LLMs —> Read more. LongRoPEMicrosoft Research published a paper introducing LongRoPE, a method for extending the context window of LLMs beyond 2 million tokens. The method combines several innovations such as multidimensional interporlation and evolutionary search to drastically scale the context window in LLMs —> Read more. Pre-Instruction TuningResearchers from Carnegie Mellon University, Meta AI and University of Washington introducing pre-instruction-tuning a method for improving continuing learning in LLMs. PIT instruction-tunes LLMs in QA pairs before a retraining run which improves the ability of the LLM to generatize new knowledge —> Read more. VideoPrismGoogle Research published a paper detailing VideoPrism, a foundation model for video understanding. VideoPrism is optimized for a wide number of tasks including classication, captioning, retrieval and several others —> Read more. 🤖 Cool AI Tech ReleasesGemmaGoogle released Gemma, a family of small models built with the same technology behind Gemini —> Read more. Stable Diffusion 3Stability AI unveiled an early preview of StableDiffusion 3 —> Read more. LoRA-LandPredibase released LoRA-Land, a series of 25 fine-tined Mistral models that outperformed GPT-4 on specific tasks —> Read more. 🛠 Real World MLCompound AI SystemsBerkelery AI Research(BAIR) published a detailed blog post discussing the idea of Compound AI Systems as the future of AI architectures —> Read more. 📡AI Radar
You’re on the free list for TheSequence Scope and TheSequence Chat. For the full experience, become a paying subscriber to TheSequence Edge. Trusted by thousands of subscribers from the leading AI labs and universities. |
Older messages
📝 Guest Post: LoRA Land: 25 Fine-Tuned Mistral-7b LLMs that Rival or Outperform GPT-4
Friday, February 23, 2024
In this guest post, Predibase team discusses their recent release of LoRA Land that they built to demonstrate a real world example of how smaller, task-specific fine-tuned models can cost-effectively
Edge 372: Learn About CALM, Google DeepMind's Method to Augment LLMs with Other LLMs
Thursday, February 22, 2024
Just like RAG but with LLMs! ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Edge 371: Two-Step LLM Reasoning with Skeleton of Thoughts
Tuesday, February 20, 2024
Created by Microsoft Research, the technique models some of the aspects of human cognitive reasoning in LLMs. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
📌 ML Engineering Event: Mastering AI and ML at Production Scale at apply()
Monday, February 19, 2024
Join the next apply() virtual conference on Wednesday, April 3, for a free event that brings together the engineering community to master AI and ML in production. Since 2021, apply() has hosted more
Edge 370: A Deep Dive Into AlphaGeometry: Google DeepMind’s New Model that Solves Geometry Problems Like a Math Ol…
Monday, February 19, 2024
The model uses a neurosymbolic approach to reasong through complex problems. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
You Might Also Like
📧 EF Core Migrations: A Detailed Guide
Saturday, May 18, 2024
EF Core Migrations: A Detailed Guide Read on: my website / Read time: 10 minutes BROUGHT TO YOU BY Low-code Framework for .NET Devs Introducing Shesha, a brand new, open-source, low-code
Slack is under attack … and you don’t want that
Friday, May 17, 2024
Plus: OpenAI is not aligned with its Superalignment team View this email online in your browser By Christine Hall Friday, May 17, 2024 Good afternoon, and welcome back to TechCrunch PM. We made it to
Ilya Sutskever leaves OpenAI - Weekly News Roundup - Issue #467
Friday, May 17, 2024
Plus: Apple is close to using ChatGPT; Microsoft builds its own LLM; China is sending a humanoid robot to space; lab-grown meat is on shelves but there is a catch; hybrid mouse/rat brains; and more! ͏
SWLW #599: Surfing through trade-offs, How to do hard things, and more.
Friday, May 17, 2024
Weekly articles & videos about people, culture and leadership: everything you need to design the org that makes the product. A weekly newsletter by Oren Ellenbogen with the best content I found
💾 There Will Never Be Another Windows XP — Why Ray Tracing is a Big Deal in Gaming
Friday, May 17, 2024
Also: What to Know About Google's Project Astra, and More! How-To Geek Logo May 17, 2024 Did You Know The very first mass-manufactured drinking straw was made of paper coated in wax; the straw was
It's the dawning of the age of AI
Friday, May 17, 2024
Plus: Musk is raging against the machine View this email online in your browser By Haje Jan Kamps Friday, May 17, 2024 Image Credits: Google Welcome to Startups Weekly — Haje's weekly recap of
Daily Coding Problem: Problem #1444 [Medium]
Friday, May 17, 2024
Daily Coding Problem Good morning! Here's your coding interview problem for today. This problem was asked by Yahoo. Recall that a full binary tree is one in which each node is either a leaf node,
(Not) Sent From My iPad
Friday, May 17, 2024
The future of computing remains frustrating (Not) Sent From My iPad By MG Siegler • 17 May 2024 View in browser View in browser I tried. I really did. I tried to put together and send this newsletter
iOS Dev Weekly - Issue 661
Friday, May 17, 2024
What's the word on everyone's lips? 🅰️👁️ View on the Web Archives ISSUE 661 May 17th 2024 Comment Did you catch Google I/O this week? It's Always Interesting to see what the Android
Your Google Play recap from I/O 2024
Friday, May 17, 2024
Check out all of our latest updates and announcements Email not displaying correctly? View it online May 2024 Google Play at I/O 2024 Check out the Google Play keynote to discover the latest products