Building Production-Grade RAG at Scale

Douwe Kiela, CEO of Contextual AI, argues that despite the advent of massive context windows (e.g., 10 million tokens), Retrieval-Augmented Generation (RAG) is not becoming obsolete. The optimal approach combines RAG with long-context models, as using the full context window for every query is computationally wasteful and often leads to performance degradation. The real solution is to make RAG systems easier to build and more effective.

The Evolution to RAG 2.0: From Models to Systems

The core idea behind “RAG 2.0” is to treat the entire pipeline as a single, jointly optimized system rather than a “Frankenstein” assembly of off-the-shelf components. This system-level approach ensures that every component—from data ingestion to the final generation—is designed to work together for maximum performance. This is analogous to the shift towards end-to-end systems seen in other areas of AI, like speech recognition. Performance in these pipelines is compounding; a failure in an early stage like document parsing cannot be fixed by later stages.

The End-to-End RAG Pipeline

The RAG 2.0 pipeline begins long before the retrieval step and involves several key, machine-learned components:

• Document Parsing and Intelligence: This is a critical and often underestimated bottleneck. Standard open-source libraries are described as “terrible” for parsing documents with complex layouts, such as those in finance. An effective system requires a sophisticated solution that incorporates Visual Language Models (VLMs) for layout segmentation and OCR. Crucially, this stage should not just extract text but also the document’s semantic hierarchy (e.g., distinguishing between the introduction, methods, and results sections of a paper). This hierarchy provides vital metadata for the retrieval process.
• Chunking and Embeddings: Instead of relying on arbitrary rules, chunking is approached as a machine learning problem to determine the optimal way to segment documents. The choice of embeddings is also integrated, using models that are specifically optimized to work with the rest of the pipeline, saving developers from the trial-and-error of selecting from thousands of available embedding models.
• Advanced Retrieval and Reranking: The retrieval process is a sophisticated “mixture of retrievers” that can include hybrid search, graph-based methods, and more to cast a wide initial net. The real intelligence, however, lies in the subsequent reranking step. This can be a cascade of rerankers that can be given specific instructions, such as prioritizing recency or weighting information from certain authors (e.g., a CEO) more heavily. This is essential for navigating the messy, unstructured data common in enterprise settings.
• Grounded Language Models (GLMs): To combat hallucinations, a general-purpose LLM is insufficient, as it’s trained to be creative. The solution is a “Grounded Language Model” (GLM) specifically fine-tuned to be strongly faithful to the provided context. A key feature of such a model is its ability to confidently answer “I don’t know” when the retrieved context does not contain the answer.

Multimodality, Reasoning, and the Future

• Multimodality: For RAG to be truly effective on enterprise data, it must handle multimodal content, including circuit diagrams, charts, tables, and code. This requires the entire pipeline to be multimodally aware.
• Reasoning and Agents: The rise of models capable of reasoning (“test-time compute”) represents a significant paradigm shift. In this new framework, RAG becomes a tool for a larger, domain-agnostic planner or agent. This enables “active retrieval,” where the agent decides if, when, and what to retrieve based on its reasoning process, rather than passively retrieving information for every query. This is sometimes referred to as “Think Mode.”
• Fine-Tuning: The debate between RAG and fine-tuning is a “false dichotomy.” The answer is to use both. While a RAG 2.0 system provides strong out-of-the-box performance, fine-tuning the embedding models, reranker, and generator can yield significant performance gains (10-20%), which can be critical for meeting production requirements. This can be achieved with a small number of examples, which are then expanded using synthetic data pipelines.
• RAG vs. Agents: The notion that agents will replace RAG is a misunderstanding. Agents and protocols like MCP (Model-as-a-Service Communication Protocol) rely on retrieval as a fundamental tool to access and reason over private or up-to-date information. The agent is the generator (G), and it is augmented by retrieval (R). The key innovation is the ability to generate a coherent, packaged answer, moving beyond the traditional “10 blue links” of search.

• Tags:
• LLM
• Large Language Models
• RAG
• Retrieval Augmented Generation
• Contextual AI
• AI Systems
• Grounding
• Multimodal AI