April 03, 2024 | Read Online

Recap#2: How does FMOps Infrastructure Stack look like?

+ our best explanatory tokens about it, including a list of open-source tools, libraries, and companies

Welcome to the Recap#2 of our Foundation Model Operations (FMOps) series. In our first, introductory Token, we asked a few questions:

Is running foundation models (FM) in production really that different and difficult from running regular models?
Is there an FMOps stack as we (almost) have with MLOps?
What should the FMOps Infrastructure Stack look like to make productizing FM easier?

SUBSCRIBE NOW

No fully mature FMOps stack yet exists comparable to the MLOps landscape. Of course, there is also an element of hype surrounding any buzzword in the rapidly evolving AI space. But we believe that differences between FMOps and MLOps are feasible. The core principles of MLOps – bringing ML to production reliably – still apply, but FMOps/LLMOps require adaptation and a few additions to the process.

While MLOps prioritizes the training-deployment cycle: data preparation, model training, evaluation, deployment, and continuous monitoring, FMOps emphasizes adaptation and fine-tuning of large, pre-trained models. It includes selecting the right FM, prompt engineering, data preparation for fine-tuning, evaluation focused on the specific task, and continuous monitoring for output quality.
While MLOps focuses on structured data used for training and serving the ML model, FMOps involves unstructured data (text, image, etc.) on which FMs have been pre-trained. Fine-tuning data needs to be carefully curated based on the downstream task.

But the main difference come with with stakeholders and Model Experience (MX) for an end-user.

MLOps is operated by data scientists, ML engineers, and DevOps teams.
FMOps also include:
- FM Providers: Responsible for developing and potentially hosting the base FMs.
- Prompt Engineers: Experts in designing effective instructions for FMs.
- Consumers: End-users utilizing the applications powered by the FMs.

MLOps does not cater to end-users directly and does not necessitate a model experience in that sense. In contrast, FMOps – which is precisely why ChatGPT soared in popularity – provides a comprehensive Model Experience (MX) where anyone, regardless of their development experience or computer knowledge, can interact with a model and tailor it to their specific needs using prompts.

In the Recap#1, you will find an invaluable collection about the basics of foundation models and how to make the right choice for your project, including techniques for models' adaptation and how to think about model alignment (including hallucinations and RLHF).

Recap #2 provides a visualization of the FMOps Infrastructure Stack along with our best explanatory tokens about it, including a list of open-source tools, libraries, and companies. This is work in progress, let us know what companies should be on this infographic.

Click to enlarge

To have access to each and every one of these articles, please →

UPGRADE

That way you will also support our cause of spreading AI knowledge

Token 1.8: Your Guide to Silicon Valley of AI Chips and Semiconductors

Understanding the Chips that Drive Today's AI Breakthroughs

www.turingpost.com/p/aichips

AI Chips Are Taking Over: Seven Specialized Processors from Tech Giants

Google, Microsoft, Amazon, and other giants race to develop specialized processors for artificial intelligence, marking a shift from traditional solutions.

www.turingpost.com/p/ai-chips-google-openai-microsoft

Token 1.12: What is Vector Database's Role in FMOps?

We explain what vector databases are and how they work, explore alternative solutions and provide expert insight on security

www.turingpost.com/p/vectordatabase

Unique list of open-source vector databases, libraries, and versatile platforms with vector functionality

Milvus, Qdrant, Faiss, Weaviate, and other databases to work with LLMs and other foundation models

www.turingpost.com/p/vector-databases-libraries-resources

Token 1.13: Where to Get Data for Data-Hungry Foundation Models

Explore how data is gathered to train FMs, learn a few data efficient training techniques and the ethics of data

www.turingpost.com/p/data

Token 1.14: What is Synthetic Data and How to Work with it?

Will it eliminate the need for real data? Let's explore

www.turingpost.com/p/synthetic

10 companies for synthetic data generation

+ discover what synthetic data is and how to use it

www.turingpost.com/p/synthetic-data-companies

Token 1.17: Deploying ML Model: Best practices feat. LLMs

Unless you are a researcher whose sole job is to beat benchmarks at some predefined dataset, you will want to deploy your model

www.turingpost.com/p/deployment

8 open-source tools for foundation model deployment

Join over 55,000 readers for in-depth knowledge and forward-thinking analysis, to make smarter decisions about AI & ML. Save time. Gain wisdom. Stay ahead.

www.turingpost.com/p/tools-for-model-deployment

Token 1.18: How to Monitor LLMs?

Ensuring Your LLMs Deliver Real Value

www.turingpost.com/p/monitoring

15+ Open-Source Tools to Monitor Your Large Language Models (LLMs)

Essential Tools for LLM Interpretation, Monitoring, and Bias Mitigation

www.turingpost.com/p/llm-observability

Token 1:19: LLM Inference: how different it is from traditional ML?

This guide tackles the unique challenges of LLM deployment, from serialization quirks to resource optimization. Master GPUs, serialization, & AI accelerators.

www.turingpost.com/p/llminference

10 open-source tools for LLM applications development

Join over 55,000 readers for in-depth knowledge and forward-thinking analysis, to make smarter decisions about AI & ML. Save time. Gain wisdom. Stay ahead.

www.turingpost.com/p/llm-applications-tools

Token 1.20: Explainable AI techniques and tools for LLMs

that will help you decipher these models and their predictions

www.turingpost.com/p/explainableai

Token 1.21: Dark side of LLMs

Vulnerabilities in LLMs and foundation models and how to deal with them

www.turingpost.com/p/llmattacks

Token 1.22: Data Privacy in LLM systems

From concerns about data leakage to the main technical strategies for managing privacy in LLMs.

www.turingpost.com/p/dataprivacy

Token 1.23: Mitigating Bias in Foundation Models/LLMs

Your guide on how to identify bias, a few debiasing techniques, and collection of tools and libraries for detection and mitigation

www.turingpost.com/p/biases

Essential Open-Source Tools for Bias Detection and Mitigation

Join over 55,000 readers for in-depth knowledge and forward-thinking analysis, to make smarter decisions about AI & ML. Save time. Gain wisdom. Stay ahead.

www.turingpost.com/p/ai-fairness-tools

Ethics? 5 Must-Read Books on AI Ethics, Insights from Sasha Luccioni's TED Talk on Impact of AI