Layering every technique in RAG, one query at a time - David Karam, Pi Labs (fmr. Google Search)

Drawing from a decade of experience at Google Search, the fundamental approach to improving Retrieval-Augmented Generation (RAG) systems is not to chase hyped techniques, but to adopt a disciplined, empirical “quality engineering loop.” This process involves establishing a quality bar with benchmarks, performing a “loss analysis” to see what’s failing, and only then selecting a technique from your toolbox. The guiding principle should be “complexity-adjusted impact”: always start with the simplest solution for the problem at hand.

The Incremental Ladder of Retrieval Techniques

A RAG system’s capabilities can be built up by layering techniques of increasing complexity, with each new layer addressing the shortcomings of the previous one.

1. In-Memory Retrieval: The simplest method is to place all documents directly into the LLM’s context window. This is a great starting point (“context windows are pretty easy so you should definitely start there”).

   • When it fails: This approach breaks down when the document corpus is too large to fit in memory or when the context window becomes so long that the model’s attention mechanism fails to properly weigh all documents.

2. Term-Based Retrieval (BM25): When in-memory retrieval is insufficient, the next step is keyword-based search. BM25 is a classic algorithm that works well by scoring documents based on query term frequency, document length, and term rarity.

   • When it fails: BM25 is highly effective for keyword-driven queries (e.g., “iPhone battery life”) but fails when the user’s intent is expressed in natural language without exact keywords (e.g., “how long does an iPhone last before I need to charge it again?”).

3. Relevance Embeddings (Vector Search): To handle semantic nuance, vector embeddings are introduced. By representing queries and documents in a vector space, the system can find conceptually similar results, not just keyword matches.

   • When it fails: You now have two sets of candidates (BM25 and vector search), which can be conflicting. Furthermore, standard relevance embeddings can fail in specialized domains where word meanings differ from common usage.

4. Re-ranking with Cross-Encoders: To resolve conflicting candidate sets, a re-ranker is used. Unlike standard embedding models that create separate vectors for the query and document, cross-encoders process the query and document together, allowing the model to attend to both simultaneously for a more accurate relevance score.

   • When it fails: Cross-encoders are computationally expensive and cannot be run on the entire corpus. They must be applied to a smaller, pre-retrieved set of candidates. More importantly, they still only measure semantic similarity, which is just a proxy for true ranking.

Moving Beyond Generic Relevance

For many applications, simple relevance is not enough. Ranking requires a deeper understanding of the domain and the user.

• Custom Embeddings for Vertical Domains: In specialized fields like law, words like “material” or “regime” have meanings that general-purpose embeddings won’t capture. When your loss analysis shows failures due to a domain-specific vocabulary that is “out of distribution” for a standard model, you need to build custom embeddings trained on your corpus to create a vector space that understands your domain’s unique semantics.

• Custom Ranking Signals: Relevance is not ranking. A truly effective system incorporates signals beyond semantic similarity. For a shopping application, a query for “cheap gifts” with a follow-up “budget of $50 or more” requires the system to understand and rank by a price signal, not just relevance. Other critical non-relevance signals include:

  • Popularity: How many times a podcast has been listened to.
  • Prominence: An item’s PageRank, determined by how many other documents link to it.
  • User Preferences: Click signals (thumbs up/down) that show what users actually engage with.

A sophisticated ranking function becomes a balanced equation of relevance, custom semantic signals, and user preference signals.

Advanced Orchestration and Optimization

As systems grow, new failure points emerge in the orchestration layer.

• Query Fan-Out: An LLM acting as an agent often creates queries that are mismatched with the capabilities of the underlying search tool. Instead of trying to teach the LLM the intricacies of the search API through prompting, it’s more robust to take control of the orchestration. A complex user request can be broken down into multiple, narrow, and precise sub-queries that are “fanned out” to the search backend.

• Supplementary Retrieval: When in doubt, “just search more.” To handle ambiguous queries like “falafel” (which could be about recipes, restaurants, or images), a good strategy is to query multiple backends to gather a richer, more diverse set of information before synthesizing the final answer.

• Model Distillation: At scale, cost and latency become critical issues. A large, general-purpose LLM is often overqualified for a narrow task. Distillation is the process of fine-tuning a smaller, cheaper model to perform a specific task (like question-answering) at the same quality level as the larger model, significantly reducing latency and cost.

The Final Backstop: Product and UX Design

Ultimately, engineering has limits. These are stochastic systems, and they will always fail some percentage of the time. When you can no longer engineer your way to higher quality, you must “punt the problem” to the product itself.

The solution is to design a user experience that can gracefully degrade or upgrade based on the system’s confidence. If the system has high-fidelity understanding, it can present a rich, structured UI with filters and direct answers. If it has low understanding, it should fall back to a simpler, less committal UI, like a standard list of ten blue links. This manages user expectations and ensures the product remains useful even when the underlying AI is uncertain.

• Tags:
• LLM
• Embeddings
• RAG
• Feature
• Retrieval Augmented Generation
• Information Retrieval
• Search
• Vector Search
• BM25
• Ranking
• Quality Engineering