Stop Chunking Like It's 2022 — Yuval Belfer, AI21 Labs

Yuval Belfer addresses the fundamental challenge in Retrieval Augmented Generation (RAG): the optimal chunk size is query-dependent, but chunking decisions are fixed during indexing. This "information problem" leads to significant recall loss, as shown by experiments demonstrating a 20-40% gap between fixed chunking and an oracle. His proposed solution, multiscale indexing, involves creating multiple document copies with different chunk sizes, querying all of them, retrieving whole documents, and merging results using Reciprocal Rank Fusion, yielding substantial recall improvements with minimal latency increase.