📝 Guest Post: Will Retrieval Augmented Generation (RAG) Be Killed by Long-Context LLMs?*
Was this email forwarded to you? Sign up here Pursuing innovation and supremacy in AI shows no signs of slowing down. Google revealed Gemini 1.5, just months after the debut of Gemini, their large language model (LLM) capable of handling contexts spanning up to an impressive 10 million tokens. Simultaneously, OpenAI has taken the stage with Sora, a robust text-to-video model celebrated for its captivating visual effects. The face-off of these two cutting-edge technologies has sparked discussions about the future of AI, especially the role and potential demise of Retrieval Augmented Generation (RAG). Will Long-context LLMs Kill RAG?The RAG framework, incorporating a vector database, an LLM, and prompt-as-code, is a cutting-edge technology that seamlessly integrates external sources to enrich an LLM's knowledge base for precise and relevant answers. It is a proven solution that effectively addresses fundamental LLM challenges such as hallucinations and lacking domain-specific knowledge. Witnessing Gemini's impressive performance in handling long contexts, some voices quickly predict RAG's demise. For example, in a review of Gemini 1.5 Pro on Twitter, Dr. Yao Fu boldly stated, "The 10M context kills RAG." Is this assertion true? From my perspective, the answer is “NO.” The development of the RAG technology has just begun and will continue to evolve. While Gemini excels in managing extended contexts, it grapples with persistent challenges encapsulated as the 4Vs: Velocity, Value, Volume, and Variety. LLMs’ 4Vs Challenges
All these challenges highlight the importance of a balanced approach in developing AI applications, making RAG increasingly crucial in the evolving landscape of artificial intelligence. Strategies for Optimizing RAG EffectivenessWhile RAG has proven beneficial in reducing LLM hallucinations, it does have limitations. In this section, we’ll explore strategies to optimize RAG effectiveness to strike a balance between accuracy and performance to make RAG systems more adaptable across a broader range of applications. Enhancing Long Context UnderstandingConventional RAG techniques often rely on chunking for vectorizing unstructured data, primarily due to the size limitations of embedding models and their context windows. However, this chunking approach presents two notable drawbacks.
In response to these challenges, emerging embedding strategies based on LLMs have gained traction as efficient solutions. They boast better embedding capability and support expanded context windows. For instance, SRF-Embedding-Mistral and GritLM7B, two best-performing embedding models on the Huggingface MTEB LeaderBoard, support 32k-token-long contexts, showcasing a substantial improvement in embedding capabilities. This enhancement in embedding unstructured data also elevates RAG’s understanding of long contexts. Another effective approach to tackle the challenges above is the recently released BGE Landmark Embedding strategy. This approach adopts a chunking-free architecture, where embeddings for the fine-grained input units, e.g., sentences, can be generated based on a coherent long context. It also leverages a position-aware function to facilitate the complete retrieval of helpful information comprising multiple consecutive sentences within the long context. Therefore, landmark embedding is beneficial to enhancing the ability of RAG systems to comprehend and process long contexts. Utilizing Hybrid Search for Improved Search QualityThe quality of RAG responses hinges on its ability to retrieve high-quality information. Data cleaning, structured information extraction, and hybrid search are all effective ways to enhance the retrieval quality. Recent research suggests sparse vector models like Splade outperform dense vector models in out-of-domain knowledge retrieval, keyword perception, and many other areas. The recently open-sourced BGE_M3 embedding model can generate sparse, dense, and Colbert-like token vectors within the same model. This innovation significantly improves the retrieval quality by conducting hybrid retrievals across different types of vectors. Notably, this approach aligns with the widely accepted hybrid search concept among vector database vendors like Zilliz. For example, the upcoming release of Milvus 2.4 promises a more comprehensive hybrid search of dense and sparse vectors. Utilizing Advanced Technologies to Enhance RAG’s PerformanceMaximizing RAG capabilities involves addressing numerous algorithmic challenges and leveraging sophisticated engineering capabilities and technologies. As highlighted by Wenqi Glantz in her blog, developing a RAG pipeline presents at least 12 complex engineering challenges. Addressing these challenges requires a deep understanding of ML algorithms and utilizing complicated techniques like query rewriting, intent recognition, and entity detection. Even advanced models like Gemini 1.5 face substantial hurdles. They require 32 calls to achieve a 90.0% accuracy rate in Google's MMLU benchmark tests. This underscores the nature of maximizing performance in RAG systems. Vector databases, one of the cutting-edge AI technologies, are a core component in the RAG pipeline. Opting for a more mature and advanced vector database, such as Milvus, extends the capabilities of your RAG pipeline from answer generation to tasks like classification, structured data extraction, and handling intricate PDF documents. Such multifaceted enhancements contribute to the adaptability of RAG systems across a broader spectrum of application use cases. Conclusion: RAG Remains a Linchpin for the Sustained Success of AI Applications.LLMs are reshaping the world, but they cannot change our world’s fundamental principles. The separation of computation, memory, and external storage has existed since the inception of the von Neumann architecture in 1945. However, even with single-machine memory reaching the terabyte level today, SATA and flash disks still play crucial roles in different application use cases. This demonstrates the resilience of established paradigms in the face of technological evolution. The RAG framework is still a linchpin for the sustained success of AI applications. Its provision of long-term memory for LLMs proves indispensable for developers seeking an optimal balance between query quality and cost-effectiveness. In deploying generative AI by large enterprises, RAG is a critical tool for cost control without compromising response quality. Just like large memory developments cannot kick out hard drives, the role of RAG, coupled with its supporting technologies, remains integral and adaptive. It is poised to endure and coexist within the ever-evolving landscape of AI applications. *This post was originally published on Zilliz.com here. We thank Zilliz for their insights and ongoing support of TheSequence.You’re on the free list for TheSequence Scope and TheSequence Chat. For the full experience, become a paying subscriber to TheSequence Edge. Trusted by thousands of subscribers from the leading AI labs and universities. |
Older messages
Edge 426: Reviewing Google DeepMind’s New Tools for AI Interpretability and Guardrailing
Thursday, August 29, 2024
Gemma Scope and ShieldGemma are some of the latest additions to DeepMind's Gemma stack ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Edge 425: Inside Mamba, the Most Famous SSM Model
Tuesday, August 27, 2024
In this issue: ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Black Forest Labs
Sunday, August 25, 2024
The startup powering image generation for xAI's Grok. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Edge 424: How DeepMind's AlphaProof and AlphaGeometry-2 Achieved Silver Medal Status in the International Math Oly…
Thursday, August 22, 2024
One model focuses on algebra and number theory, while the other mastered geometry. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Edge 423: Understanding the SSM Fundamental Equation
Tuesday, August 20, 2024
Some of the foundations of SSMs plus an exploration of a classic model. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
You Might Also Like
This Week in Rust #565
Thursday, September 19, 2024
Email isn't displaying correctly? Read this e-mail on the Web This Week in Rust issue 565 — 18 SEP 2024 Hello and welcome to another issue of This Week in Rust! Rust is a programming language
Daily Coding Problem: Problem #1561 [Easy]
Thursday, September 19, 2024
Daily Coding Problem Good morning! Here's your coding interview problem for today. This problem was asked by Microsoft. Print the nodes in a binary tree level-wise. For example, the following
Sixteen Candles Down the Drain
Thursday, September 19, 2024
Spectacles, Vestager, EC Posts, Meta Letters, PayPal Design, Microsoft Deals, Palmer Luckey Goggles, Spotify Ads Sixteen Candles Down the Drain Spectacles, Vestager, EC Posts, Meta Letters, PayPal
How Greedy Miners Are Breaking DAG Blockchains
Thursday, September 19, 2024
Top Tech Content sent at Noon! A dev conference with discussions, workshops, and 1:1 feedback sessions Read this email in your browser How are you, @newsletterest1? 🪐 What's happening in tech today
Issue 332 - Camouflaged Tesla Robotaxi prototype sighted
Thursday, September 19, 2024
View this email in your browser If you are just now finding out about Tesletter, you can subscribe here! If you already know Tesletter and want to support us, check out our Patreon page Issue 332 -
Programmer Weekly - Issue 223
Thursday, September 19, 2024
View this email in your browser Programmer Weekly Welcome to issue 223 of Programmer Weekly. Let's get straight to the links this week. Quote of the Week "It's tempting to write a long
Data Science Weekly - Issue 565
Thursday, September 19, 2024
Curated news, articles and jobs related to Data Science, AI, & Machine Learning ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Contrarian Report Shades GitHub Copilot Productivity, Bits Polished as .NET 9 Nears, Python in VS Code, More
Thursday, September 19, 2024
Home | News | How To | Webcasts | Whitepapers | Advertise .NET Insight September 19, 2024 THIS ISSUE SPONSORED BY: ■ dtSearch® - INSTANTLY SEARCH TERABYTES ■ Live! 360: Developer / IT / Security / Data
Web Tools #583 - No Code Maps, React, Testing, Git/CLI
Thursday, September 19, 2024
WEB VERSION Issue #583 • September 19, 2024 The following is a paid product review for No Code Map App, a platform for building custom interactive maps from almost any data source, no coding required.
Python Weekly - Issue 668
Thursday, September 19, 2024
View this email in your browser Python Weekly Welcome to issue 668 of Python Weekly. Let's get straight to the links this week. From Our Sponsor Get Your Weekly Dose of Programming A weekly