Small language models make AI accessible to a wide range of users because they can run on edge devices like smartphones. However, there is one important issue – small models have less capacity and are highly sensitive to training data. That’s why the quality of datasets is even more crucial for them than for large language models (LLMs).

Today, we are going to explore the key role of datasets in training small models, using the SmolLM family from Hugging Face as an example, with a special focus on their SmolLM2 models. While Hugging Face researchers have already published extensive materials on all SmolLM and SmolVLM models, we believe it’s important to compile everything in one place to highlight Hugging Face’s contributions to high-quality datasets and their training strategies for small AI models.

So, let’s dive into Hugging Face’s approach to achieving state-of-the-art performance in small LMs through mixing of the best datasets and step by step training.

In today’s episode, we will cover:

The secret behind small models training

So, what’s the trick to making small models punch above their weight? There are a few key strategies that make all the difference:

Today we are going to discuss the third option - how to improve models reasoning through high-quality training data.

As smaller models have less capacity, the data used to train them is even more important than for large models. Instead of memorizing random facts, small models must be carefully optimized to learn essential knowledge and reasoning skills. Small models are also more sensitive to noise in the data, so choosing, filtering, and balancing different sources carefully is crucial. In particular, it is important to find the right mix of specialized datasets (like math and code) so that the model can learn effectively.

The details of how small models are trained are usually not shared by the developers. Here is where Hugging Face researchers, with their strong commitment to true open source, have contributed to the AI and ML community by creating their family of small models – including SmolLM, SmolLM2, SmolVLM, and SmolVLM2 – and providing their training guides.

How it all began: From SmolLM to SmolVLM2

Are all SmolLM models unified by a single common concept?

From the very beginning of SmolLM release on July 16, 2024, Hugging Face has chosen a strategy with the following concepts:

SmolLM, their first set of small AI models, is available in three sizes:

It marked the first step toward high‐performance, on‑device small LMs. The path to achieving state-of-the-art performance for SmolLM lies in training them on a high-quality dataset called SmolLM-Corpus, which includes:

For training 135M and 360M versions Hugging Face used 600 billion tokens from the SmolLM-Corpus, while 1.7B version was trained on 1 trillion tokens. Their architectures are different, as well.

135M and 360M models use Grouped-Query Attention (GQA) to speed up inference. It means that multiple query heads share the same key-value pairs instead of having separate ones to reduce the computational cost. They also prioritizes depth over width (more layers, fewer neurons per layer) to improve performance while staying small. On the contrary, 1.7B model’s architecture is similar to traditional GPT-style transformers.

Here’s what results in common sense reasoning, world knowledge, and coding skills SmolLM models achieved:

Back on November 2, 2024 Hugging Face unveiled SmolLM2 in three size: 135M, 360M, and 1.7B parameters. 

Let’s dive deeper in all the details! It’s truly a work of art and your guide to building a powerful small model →

Upgrade if you want to be the first to receive the full articles with detailed explanations and curated resources directly in your inbox. Simplify your learning journey.

or follow us on Hugging Face, this article will appear there tomorrow for free