Information Retrieval from the Ground Up - Philipp Krenn, Elastic
Philipp Krenn from Elastic demystifies the 'R' in RAG, arguing that modern retrieval is a sophisticated blend of classic keyword search (like BM25) and modern vector search. This workshop explores the fundamentals of lexical analysis, scoring, dense/sparse vectors, and advanced hybrid search techniques like Reciprocal Rank Fusion (RRF).
Vector search is often seen as the core of Retrieval-Augmented Generation (RAG), but it is more accurately described as a single, powerful feature within the much broader and older field of information retrieval. A robust retrieval system, the “R” in RAG, effectively combines classic, battle-tested techniques with modern machine learning approaches.
The Foundation: Lexical Search and Analysis
Lexical, or keyword, search is a “dumb but scalable” approach that has been the backbone of search engines for decades. Its effectiveness lies in the pre-processing of text during ingestion, a process called analysis.
A standard analysis pipeline involves several steps:
- Tokenization: Breaking text down into individual words or tokens. For Western languages, this is typically done by splitting on whitespace and punctuation.
- Lowercasing: Converting all tokens to lowercase to ensure case-insensitive matching.
- Stop Word Removal: Filtering out common words (e.g., “the”, “a”, “is”) that add little semantic value for search. However, this can be a double-edged sword; a phrase like “to be or not to be” would be completely removed.
- Stemming: Reducing words to their root form (e.g., “looking” becomes “look”). This is a rule-based process and can sometimes produce non-dictionary words.
This analysis must be language-specific. Applying an English analyzer to German text, for instance, will produce nonsensical tokens and poor search results.
The output of this process is stored in an inverted index. This data structure maps each token to a list of documents containing it, along with its frequency and position within the document. This makes lookups extremely fast, as the search engine matches query tokens against the index instead of scanning the original text.
Scoring and Relevance: BM25
Keyword search relevance is typically calculated using an algorithm like BM25 (Best Match 25), which improves upon the classic TF-IDF. It comprises three main components:
- Term Frequency (TF): How often a term appears in a document. BM25 assumes diminishing returns, meaning the 10th occurrence of a word adds less relevance than the second.
- Inverse Document Frequency (IDF): How rare a term is across the entire collection of documents. Rare terms are considered more significant and receive a higher weight.
- Field Length Norm: Matches in shorter fields (like a title) are considered more relevant than matches in long blocks of text.
A critical takeaway is that these relevance scores are relative and not absolute. They are only meaningful for ranking results within a single query. The score of a document will change if other documents are added or removed from the index, as this alters the IDF statistics. Therefore, trying to convert a relevance score into a percentage is a flawed approach.
The Semantic Layer: Vector Search
Vector search introduces semantic understanding, where the retrieval is based on meaning rather than exact keyword matches. This is achieved by using machine learning models to create numerical representations of text, called embeddings.
- Dense Vectors: These represent text as a large array of floating-point numbers in a high-dimensional space. The model learns the dimensions, which are not directly interpretable by humans. Documents with similar meanings are located closer together in this vector space. Similarity is calculated using metrics like cosine similarity or dot product. A key characteristic is that dense vector search will almost always find some result, as every document has a position in the space, which can lead to irrelevant results if not handled carefully.
- Sparse Vectors: Models like SPLADE offer a more interpretable alternative. They also represent text with learned embeddings, but the output is a list of expanded tokens with associated weights. This is akin to a “learned” version of TF-IDF. It’s more interpretable because you can see which terms contributed to a match, but it can be more computationally expensive at query time.
The Modern Approach: Hybrid Search
The most effective retrieval systems rarely rely on a single method. Hybrid search combines the strengths of lexical and vector search to deliver superior results. For example, keyword search excels at matching specific, known terms like brand names or product codes, where vector search might falter. Vector search excels at finding conceptually related documents, even if they don’t share keywords.
To combine the results from these different systems, a technique called Reciprocal Rank Fusion (RRF) is increasingly popular. Instead of trying to normalize and blend the incomparable scores from different search methods, RRF combines results based on their rank in each result list. This provides a simple yet robust way to merge lexical and semantic search results into a single, more relevant list.
Other advanced strategies include:
- Chunking: Breaking long documents into smaller, semantically coherent chunks before creating embeddings, as models have a limited context window.
- Query Rewriting: Using an LLM to refine or expand a user’s query before it hits the search index. For example, a user might describe “the Italian dish with layers of pasta and meat,” and the LLM rewrites the query to search for “lasagna.”
- Rescoring: Using a two-stage process where a fast, initial retrieval method (like BM25 or approximate vector search) fetches a large set of candidate documents, which are then re-ranked by a more computationally expensive but accurate model (a reranker) to produce the final, high-quality result set.