New day, new RAG research! We've already dedicated many articles to the Retrieval-Augmented Generation (RAG) approach and its various modifications, but honestly, it's a topic that never gets old. As one of the most popular methods for enhancing LLMs with external knowledge and retrieving the information you actually need, RAG is continually evolving. Researchers are actively working on modifications to address its limitations, such as long processing times and difficulties with handling extended contexts. But with so much research out there, it can be challenging to know what to focus on. That's where we come in. Speculative RAG is one of the latest approaches, aiming to balance efficiency and effectiveness by using two types of language models in its architecture. The researchers behind it have come up with an intriguing idea: leveraging a small language model within a larger framework. Let’s explore how they arrived at this concept of Speculative RAG.

In today’s episode, we will cover:

Overview of existing RAG systems and their limitations

As you probably know, RAG systems combine large language models (LLMs) with external information sources to answer queries. These systems aim to enhance the accuracy and relevance of responses by incorporating data retrieved from external databases.

However, existing RAG approaches have limitations: 

These methods often struggle with balancing efficiency and effectiveness. It’s difficult for them to do all at the same time – handling long contexts and ensuring diverse perspectives while maintaining speed and accuracy.

Here comes Speculative RAG

The researchers from the University of California, San Diego, and Google came up with a thought: what if a smaller, specialized language model could efficiently draft multiple answers from different document subsets, and then a larger, generalist language model could simply verify these drafts? How will it improve RAG approach?