In this guest post, Alex Ranter, CEO and co-founder of Snorkel, highlights the importance of data-centric operations in AI development and emphasizes the need for customized models based on enterprise-specific data. He introduces Snorkel's Foundation Model Data Platform, which aims to enable organizations to build their own AI models tailored to their unique data and use cases.

Modern large language or “foundation” models have become incredibly powerful over the last several years due to a combination of data and compute scaling and “deep” model architectures. The leap forward has been nothing short of astonishing. 

However, as many are beginning to realize: these models do not solve all problems out of the box and often need significant customization3. These models are not magic; to start, they have only been trained to sound statistically plausible given a prompt—not to be accurate, unbiased, and truthful on specific mission-critical tasks across unique datasets and domains. This is especially true in enterprise settings where data and use cases are often very different from the web data these models were trained on and where high, robust accuracy is required for production deployment. 

Today we are excited to announce Snorkel’s Foundation Model Data Platform, which supports the broader set of data-centric operations involved in developing modern foundation models (FMs)–from sampling, filtering, and curating the datasets for domain-specific pre-training to authoring and cleaning the instruction-tuning datasets for generative AI alignment–making them first-class and programmatic. We aim to enable every enterprise to build AI that works for their unique data and use cases and turn this into a powerful AI moat for their business.

“Snorkel AI’s new foundation model platform has the potential to significantly enhance how Azure customers build, fine-tune, and apply large language models across their business. This could fundamentally shift the current paradigm, making AI more accessible and customizable for every enterprise, regardless of size or industry.”

John Montgomery, Corporate Vice President, Program Management, AI Platform at Microsoft5

KEEP READING TO LEARN MORE

*This post was written by Alex Ratner, CEO and co-founder of Snorkel We thank Snorkel for their ongoing support of TheSequence.