Next Week in The Sequence:

• Edge 305: Our generative AI series continues with an overview of in-context, retrieval-augmented LLMs including Google’s original paper in the space. We also explore the Humanloop platform for LLMs.

• Edge 307: We deep dive into the famous Open Assistant project for foundation models.

📝 Editorial: Open Source Scored the First Major M&A of the Generative AI Era

M&A activity is always interesting to evaluate the health of a tech market. While fundraising activity often forecasts the value of a company in the relatively long term, M&A activity provides a pragmatic view of what exit strategies might look like for a specific segment of companies. Having too much or too little M&A in a market is always bad; you want just the right level of deals to rationalize valuations in a sector. Well, last week, we witnessed the first high-profile M&A transaction in the generative AI space, and it went to the open-source column. Databricks agreed to acquire MosaicML for an astonishing $1.3 billion valuation.

MosaicML is a two-year-old company behind the open-source MPT-30B and MPT-7B models, and it has built a state-of-the-art platform for training and fine-tuning foundation models. This deal is incredibly significant for several reasons. Firstly, it demonstrates the real potential of open-source foundation models as a viable alternative to closed, API-based models. I mean, to pay $1B+ for something, you must be truly convinced that these open-source models will match the quality of GPT-4, Claude, and PaLM. If you haven't tried MPT-30B, I think you will be pleasantly surprised by its tremendous quality. Secondly, Databricks' enterprise distribution can act as a strong catalyst for the adoption of MPT models and eliminate barriers for open-source generative AI. Lastly, paying $1.3 billion for a two-year-old company in a highly competitive space might seem irrational, but it shows that Databricks believes the MosaicML platform can unlock $10 billion to $20 billion in value.

The MosaicML acquisition follows other significant transactions, such as Snowflake acquiring Streamlit for $800 million last year and Neeva for $150 million this year. Beyond the economics, I believe the Databricks-MosaicML deal is an incredible stamp of approval for open-source ML. Now we should see what Databricks' competitors (like Snowflake 😉 ) do.

🔎 ML Research

CoDi

Microsoft Research published a paper detailing CoDi, a generative AI model capable of generating content across different modalities such as language, image, audio or video. Together with the paper, Microsoft announced Project i-Code to foment multimodal generative AI —> Read more.  

ZeRO++

Microsoft Research published a paper detailing ZeRO++, a high performance communication pipeline optimized for LLM training. As it names indicates, ZeRO++ is built on top of ZeRO but reduces the communication volume by 4x —> Read more.

A Unified Pretraining Strategy for Computer Vision Models

Google Research published a paper unveiling a pretraining strategy that combines image captioning and image classification. The strategy delivers amazing performance in zero shot classification tasks —> Read more.

XGen

Salesforce Research open sourced XGen, a 7 billion parameter LLM trained on 8K sequence length for up to 1.5T tokens. XGen achieved amazing results in both language and coding tasks —> Read more.

Textbooks is All You Need

In a fascinating paper, Microsoft Research introduced phi-1, a transformer model for coding trained in high quality text book data. Despite having only 1.3B parameters, phi-1 to match the quality of larger alternatives —> Read more.

🤖 Cool AI Tech Releases

LMFlow

LMFlow is an open source toolkit for fine-tuning large foundation models —> Read more.

Open LLM Leaderboard

Hugging Face provided an update about the helpful and controversial Open LLM Leaderboard —> Read more.

Chat Arena

Chat Arena is an open source game environment to enab,le research about autonomous LLM agents —> Read more.

MediaPipe Diffusion Plugins

Google Research open sourced text-to-image plugins for its MediaPipe on-device ML framework —> Read more.

🛠 Real World ML 

Meta AI Cards

Meta AI released a series of cards that document the ML use cases across Facebook and Instagram —> Read more.

Real Time ML at Lyft

Lyft discusses the architecture behind Real-time Machine Learning with Streaming initiative which allow developers to incorporate real time ML capabilities into their applications —> Read more.

Declarative Data Pipelines at LinkedIn

LinkedIn provided an overview of the architecture and tech powering their declarative data pipelines —> Read more.

📡AI Radar

• Databricks signed the agreement to acquire MosaicML for $1.3 billion.

• Inflection, Reid Hoffman new venture, announced an astonishing $1.3 billion fundraise.

• Generative-video darling Runway announced $141 million in new funding.

• Generative-content enterprise platofrm Typeface raised $100 million in new funding.

• Generative AI lab Reka announced $58 million to advance research in the space.

• Google Deepmind’s CEO Demis Hassabis gave an interesting interview to Wired in which he he unveiled their plans for Gemini, a model that will rival ChatGPT.

• Thomson Reuters acquired legal AI startup Casetext for $650 million.

• Generative AI startup Voice.ai raised $6 million and announced a major user milestone.

• Scriptic, a studio leveraging generative AI for content production scored a $5.7 million funding round.

• AI-based API platform Speakeasy came out of stealth mode with a $7.6 million seed investment.

• Gleamer, an AI platform for radiologists, raised $27 million.

• Loora, a generative AI platform for language learning, came out of stealth mode with $9.25 million.

• AI-developer platform Faros AI announced $20 million series A.

• AI security platform CalypsoAI raised $23 million.