The unreasonable effectiveness of BM25 for agentic search — Jo Kristian Bergum, Hornet.dev

Jo Kristian Bergum, CEO of Hornet Dev, asserts that BM25 (Best Match 25), a 30-year-old lexical scoring function, is experiencing an “unreasonable effectiveness” and a strong comeback in the era of agentic search. The core reason for this resurgence is not a change in BM25 itself, but a fundamental shift in the user: large language model (LLM) agents.

Agentic Search Defined

Agentic search is characterized as search within an agent’s loop, where an agent, aiming to accomplish a task (e.g., coding, research), has an information need. Building an effective agentic search system requires three components:

• Capable Model: An LLM adept at using tools and formulating queries.
• Harness: The mechanism for exposing retrieval and search functions to the model (e.g., tool calling, code mode).
• Retrieval Engine: An efficient system capable of performing searches, potentially over billion-scale document sets.

BM25 is a scoring function that calculates a relevancy score between a query and a document based on term interaction. Its name originates from its development decades ago, where experiment number 25 yielded the best results and stuck.

The New User: LLM Agents

Traditional human users typically search with short, 2-3 word queries. In contrast, LLM agents possess extensive general knowledge about entities, companies, dates, and more. This implicit knowledge allows them to formulate much longer, more specific queries, utilize advanced search syntax (e.g., site operators, phrases), and execute multiple queries rapidly. This makes LLMs fundamentally more powerful users of tools like BM25.

Evidence from Benchmarks

Bergum highlights the Browsecom+ benchmark, which features 830 riddle-like questions over a corpus of approximately 100,000 web documents. This benchmark reveals key insights:

• Context Window Limitations: Comparing context windows to “floppy disks” (1.4MB then, ~350,000 tokens now), Bergum emphasizes that despite increasing context sizes, a mechanism for selecting relevant information remains crucial.
• Reasoning vs. Retrieval: When evidence documents are directly provided to the model’s context window, accuracy is high, indicating that reasoning is not the primary bottleneck. However, when the model relies on a search tool, accuracy drops significantly. This demonstrates that end-to-end task performance heavily depends on the model’s ability to formulate effective queries and the quality of the retriever.
• Query Trajectories: LLMs exhibit complex query trajectories, iteratively executing queries, processing responses, and reformulating subsequent queries.

“Which BM25 Do You Mean?”

The effectiveness of BM25 is highly dependent on its specific implementation and hyperparameter tuning. Bergum notes that some initial benchmarks used inadequate parameters for handling long documents, leading to an underestimation of BM25’s performance compared to embedding models. With proper tuning, BM25 can be remarkably effective.

Why BM25 is Effective for Agentic Search

BM25’s resurgence is driven by several factors:

• Explainability: The model can easily inspect and understand why a query returned specific results because it involves matching literal terms, aiding in query reformulation.
• Exact Matching: It excels at exact matching for entities, zip codes, SKUs, and other specific terms that can be challenging for embedding models to represent.
• Cost-Effectiveness: Compared to the computational and infrastructure costs of embedding inference with large models, BM25 is relatively cheap.
• Tooling Integration: Its literal matching nature pairs well with tools like grep, which LLMs are highly proficient at using.

A Promising Pattern: Dynamic Workspace Expansion

Inspired by research from Waterloo, Bergum champions the concept of “scaling direct corpus interaction via dynamic workspace expansion.” This involves:

• Retrieving documents (e.g., with BM25) from large corpora into a “workspace.”
• Organizing this workspace as a file system, allowing the model to interact with documents using familiar primitives like grep, sed, and other bash commands.
• Offering progressive disclosure, where the model can view document titles and snippets, then decide to “read more” of a document, leveraging its coding and tool-use capabilities. This approach optimizes for what frontier LLMs are already good at, integrating sandbox and retrieval infrastructure.

Evolution of Evaluation

Traditional information retrieval evaluation methods, such as single-query NDG scores, are becoming obsolete for agentic search. The ability of agents to reformulate and execute multiple queries renders such metrics less relevant. Instead, evaluation should focus on whether the agent successfully accomplishes its end task, such as accurately answering a question.

Hornet is actively investing in highly efficient BM25 implementations, demonstrating significantly higher throughput and lower latency compared to other engines for large-scale web document retrieval.

Four Key Takeaways

1. New, Powerful User: LLM agents with their general knowledge and query formulation abilities empower simple tools like BM25.
2. BM25 Implementation Matters: Parameter tuning and implementation quality significantly impact performance.
3. Explainability and Combination: BM25 is effective because it’s explainable to the model and can be combined with literal matching tools like grep.
4. Efficiency is Crucial: Optimized implementations provide significant cost savings for large-scale infrastructure.

• Tags:
• Feature
• Information Retrieval
• BM25
• LLM Agents
• Retrieval Augmented Generation
• Agentic Search
• Lexical Search
• Query Formulation
• Hornet Dev