Four months ago, we launched our FMOps series to demystify the FMOps infrastructure stack. Our goal: to dissect its structure and assess its relevance and necessity. Writing about FMOps and what we term 'task-centered ML' is far from trivial, as it involves organizing emerging knowledge in real-time.

But it’s very insightful! We've covered a wide range of topics, from foundation model basics to advanced concepts like Retrieval-Augmented Generation (RAG) and Chain-of-Thought Prompting (CoT), extending to synthetic data. Fourteen editions are out, and more are in the pipeline!

It's time to recap and update new readers on our journey.

Before diving deeper, let's establish a glossary of key terms, akin to loading essential packages before starting main code development:

Token 1.1: From Task-Specific to Task-Centric ML: A Paradigm Shift (free)

In this opening edition, we explore the connection between emerging FMOps and LLMOps and established MLOps. We examine how FMOps creates a new ecosystem for foundation models, driven by task-centric ML. This shift promises to transform ML's economics, scalability, and applications.

Token 1.2: Use Cases for Foundation Models and Key Benefits

This part examines the dynamic world of FMs, setting them apart from traditional ML models. We investigate scenarios where each excels, analyzing the reasons for their differing performances. The discussion also covers strategic considerations in choosing between traditional ML and FMs for various business scenarios, and highlights the transformative potential of FMs across multiple industries, backed by examples. Strategic insights.

Token 1.3: What is Retrieval-Augmented Generation (RAG)?

Our first practice-focused edition examining Retrieval-Augmented Generation (RAG). We explore RAG's origins, its role in addressing LLMs’ limitations, its architectural design, and the reasons for its growing popularity. Additionally, you will find a curated collection of resources for RAG experimentation. Very popular!

Token 1.4: Foundation Models – The Building Blocks

A comprehensive guide for those new to foundation models. We cover systematic concepts such as the definition, key characteristics, and various types of FMs, enriched with practical insights from Rishi Bommasani, co-author of a seminal paper on foundation models, “On the Opportunities and Risks of Foundation Models.” Must-read for beginners.

Token 1.5: From Chain-of-Thoughts to Skeleton-of-Thoughts, and everything in between

This edition focuses on Chain-of-Thought Prompting (CoT), a concept proposed in 2022 that has recently gained popularity and transformed the field of prompting. We explore where CoT stands in the world of prompting starting from basic concepts like zero-shot and few-shot prompting to CoT’s various spin-offs. This edition serves as an up-to-date reference for understanding the complexities and developments in this evolving field. Everything you need to know about CoT.

Token 1.6: Transformer and Diffusion-Based Foundation Models

We focus on Transformer and Diffusion-based models, leading players in the generative AI space. This edition explores the origins of their architectures, and how they function, and reviews some of the most significant implementations like Stable Diffusion, Imagen, and DALL-E. Core knowledge not to miss.

Token 1.7: What Are Chain-of-Verification, Chain of Density, and Self-Refine?

Following our discussion on Chain-of-Thought Prompting (CoT) in Token 1.5, we explore related concepts like Chain-of-Verification (CoVe) and Chain of Density (CoD). These concepts, though differing from CoT, use the 'chain' analogy to represent various methods of reasoning and summarization in LLMs.

Token 1.8: Silicon Valley of AI Chips and Semiconductors

A deep dive into the critical role of computing in FMs and LLMs. We examine the semiconductor industry's evolution, transitioning from general-use chips to specialized AI semiconductors. This edition covers market dynamics, computational choices, and what the future holds for AI chips. Answering the question: Why such GPU craze?!

Token 1.9: Open- vs Closed-Source AI Models: Which is the Better Choice for Your Business?

This segment analyzes the critical decision between open- and closed-source FMs and its impact on AI strategies. We define the open- and closed-source models, outline key factors to consider when making this choice and explore the fundamental differences between open- and closed-source models for businesses. Where do you stand in the open- vs. closed-source battle?

Token 1.10: Large vs Small in AI: The Language Model Size Dilemma

Can small models substitute large models? We explain various techniques for downsizing models, including pruning, quantization, knowledge distillation, and low-rank factorization. Additionally, we provide insights on how to start with small models, bypassing the need to compress large ones and offer a list of smaller models suitable for various projects. Really useful!

Token 1.11: What is Low-Rank Adaptation (LoRA)?

In this edition, we compare various adaptation techniques for LLMs such as prompt engineering, fine-tuning, and retrieval-augmented generation (RAG). Through our analysis, we discovered that while prompt techniques and RAG are relatively easier to implement, fine-tuning stands out for its effectiveness in specific scenarios, despite its demanding requirements in terms of computational resources, memory, time, and expertise. We list and discuss the specific scenarios where fine-tuning is indispensable and then transition to how we can optimize this process with Low-Rank Adaptation (LoRA), a technique that has garnered significant interest recently. We will explore the origins of LoRA, explain how it operates, and illustrate why it's becoming an increasingly popular concept in the world of LLMs.

Token 1.12: What is Vector Database's Role in FMOps?

Vector Databases are on fire. Why? Our discussion covers what vector databases are, their functionality, and their significance in managing complex data sets. We also look into alternative solutions and offer expert opinions on security aspects. For practical assistance, we provide a list of open-source vector databases and search libraries. Your best guide.

Token 1.13: Where to Get Data for Data-Hungry Foundation Models

We tackle the crucial topic of data requirements for FMs, including the challenges posed by dataset biases and strategies for mitigating them. We discuss methods for data collection, introduce some data-efficient training techniques, and dive into the ethical aspects of data usage. Looking forward, we anticipate more in-depth discussions on data sourcing in the upcoming year. Data is the King!

Token 1.14: What is Synthetic Data and How to Work with it? 

We explore the concept of synthetic data, its origins, and its practical applications. This edition explains why synthetic data is needed, how to generate it, and the considerations for creating realistic synthetic datasets. Can you use only Synthetic data? Most likely no. Find out – why.

The FMOps series will continue in 2024! Happy Holidays, and best wishes for the New Year!

Send us your comments and suggestions about the topics you want to read on Turing Post 🤍