Stop Chunking Like It's 2022 — Yuval Belfer, AI21 Labs
Yuval Belfer addresses the fundamental challenge in Retrieval Augmented Generation (RAG): the optimal chunk size is query-dependent, but chunking decisions are fixed during indexing. This "information problem" leads to significant recall loss, as shown by experiments demonstrating a 20-40% gap between fixed chunking and an oracle. His proposed solution, multiscale indexing, involves creating multiple document copies with different chunk sizes, querying all of them, retrieving whole documents, and merging results using Reciprocal Rank Fusion, yielding substantial recall improvements with minimal latency increase.
The “No Correct Chunk Size” Problem in RAG
Yuval Belfer argues that the prevalent assumption of a single “correct” chunk size for Retrieval Augmented Generation (RAG) systems is fundamentally flawed. He highlights a critical “information problem”: at indexing time, when chunking decisions are made, the system lacks knowledge of future queries. Conversely, at retrieval time, when queries are available, the chunk sizes are already fixed, preventing dynamic adaptation. This trap forces RAG implementers to make arbitrary chunking choices (e.g., 512 tokens), leading to significant inefficiencies.
Belfer illustrates this with Seinfeld transcript examples:
- A specific question like “What is the name for Jerry’s favorite church?” performs best with small chunks (e.g., 100 tokens), achieving rank 1. Larger windows bury the answer below rank 50.
- A complex question like “Who does Jerry describe as his nemesis and pure evil?” (Newman) requires larger contexts, as the answer is distributed across a scene. Small chunks fail to retrieve it.
This demonstrates that optimal chunk size is intrinsically query-dependent, not a static property of the corpus.
The Cost of Arbitrary Chunking: The Oracle Experiment
To quantify the impact, Belfer’s team conducted an “oracle experiment.” They duplicated several datasets (QM Sum, Narrative QA, Seinfeld, Finance Bench) at six different chunk sizes. The oracle, a hypothetical system capable of choosing the best chunk size for each individual query, was then compared against any single fixed chunk size. The results consistently showed a substantial recall gap of 20% to 40% between the oracle and the best fixed-size approach. This gap, Belfer contends, represents the “quiet cost” of choosing an arbitrary chunk size like 512 tokens. No single chunk size dominated across all queries or datasets, indicating that any fixed choice would underperform for a significant portion of queries.
Multiscale Indexing and Reciprocal Rank Fusion (RRF)
Refusing to tune a flawed premise, the team developed a solution based on multiscale indexing. The core idea is to:
- Index the corpus multiple times, each with a different, fixed chunk size (e.g., 50, 100, 200, 500, 1000, 2000 tokens). This creates ‘N’ copies of the dataset, each chunked differently.
- At retrieval time, query all ‘N’ indexed versions simultaneously for a given query.
- Crucially, instead of returning individual chunks, the system retrieves whole documents. This step is vital because chunks of different sizes are not directly comparable in ranking. By returning documents, the rankings become commensurable.
- Merge the ‘N’ document rankings using Reciprocal Rank Fusion (RRF). RRF is a simple, non-model-based algorithm that aggregates ranked lists, effectively allowing each chunking strategy to “vote” on the relevance of documents.
This approach bypasses the “information problem” by making chunking decisions dynamic at retrieval time, without adding complexity via new models.
Results and Costs
Belfer reports significant improvements across all tested datasets (QM Sum, Narrative QA, Seinfeld, Finance Bench, and others like MTB), achieving better or matching results compared to the best single fixed chunk size. The method consistently improved recall by 20% to 40%.
The primary cost is increased memory usage, estimated at 2x to 5x, due to storing multiple copies of the indexed data. However, Belfer emphasizes that there is “almost no added latency.” This is because the multiple retrieval calls can be executed in parallel, and the RRF merging step is computationally lightweight, described as a “short script rather than a model.”
Future Work
Belfer acknowledges areas for improvement:
- Optimal number and selection of chunk sizes: The current choices (50, 100, 200, etc.) were arbitrary, suggesting a need for more principled methods.
- Beyond RRF: Exploring other ranking fusion algorithms that might offer superior performance.
In conclusion, Belfer asserts that “agents didn’t kill retrieval. Nothing died.” Simple, infrastructure-level improvements like multiscale indexing can significantly enhance RAG system performance (20-40% recall) without introducing sophisticated models or high latency.