Agents as Search Engineers // Santoshkalyan Rayadhurgam
Large language models are transforming search from a static, stateless process into a dynamic, agent-based reasoning system. This talk explores the practical patterns—like query rewriting, hybrid retrieval, and agent-based reranking—for building and deploying these 'agentic search' systems at scale, covering the architectural principles, production challenges, and the future trajectory where search itself may dissolve into understanding.
For the last two decades, search systems were built on the assumption that users express a fully-formed intent. This is no longer true. User interfaces, expectations, and the nature of queries have changed, demanding a shift from deterministic, stateless retrieval systems to ones that behave like distributed reasoning engines.
The Evolution from Classical Search to Agentic Systems
Traditional search architectures have evolved through distinct stages, each with its own limitations:
- Lexical Pipelines (e.g., BM25): These systems were deterministic and fast but relied on sparse interpretations of text. They treated every token as an independent statistical event, lacking semantic structure, which caused them to fail when a user’s phrasing diverged from the vocabulary of the documents.
- Vector-Based RAG Systems: The introduction of embeddings and dense representations brought semantic similarity into the fold. However, this created new operational challenges, such as managing chunking heuristics, K-NN latency, vector drifts, and result irrelevance.
- Agentic Search: This represents an entirely new computational model, not just an incremental improvement. It transforms search into a multi-turn reasoning process involving strategies, tool calls, and state management. Here, retrieval is not an endpoint but part of a continuous control loop.
The primary driver for this shift is user behavior. Users increasingly operate in a high-ambiguity, high-complexity space, issuing underspecified queries like, “find that Python memory thing from last week.” Static information retrieval fails here because it assumes text is truth, whereas users are expressing partial intent, not literal instructions. This architectural mismatch necessitates an agentic approach that can interpret the query by detecting entities (Python), resolving temporal constraints (last week), and inferring missing structure (article, tutorial, or code snippet).
Core Principles for Building Agentic Systems
Building these sophisticated systems requires a foundation of solid design principles.
Composable Tools Over Monolithic APIs
Instead of a single, monolithic search endpoint with dozens of parameters, an agentic approach favors composable, atomic, and transparent tools (e.g., keyword_search, semantic_search). These primitives make the agent’s planning process more tractable, improve determinism, and simplify debugging and policy management.
The Importance of a Predictable Backend
The backend substrate for an agent doesn’t need to be overly complex; it needs to be predictable. Opaque models and black-box embeddings make it difficult for an agent to reason about cause and effect. In fact, a simple BM25, when paired with an agent, can outperform a complex neural network that operates without one. Transparent systems with predictable distribution scores allow the agent to form more accurate hypotheses and make refinements more meaningful.
Modern Query Interpretation
Agentic interpretation moves beyond generating linear labels. It creates a probabilistic space for interpretation, producing a structured query type, semantic intent, temporal constraints, and multiple hypotheses for disambiguation. This allows downstream strategies to be far more targeted. This leads to the creation of query embeddings conditioned on intent. For a “laptop for coding,” the system emphasizes CPU and RAM, while for a “video editing computer,” it focuses on GPU and display throughput, resulting in significant precision lifts (~35%) without major latency overhead.
Hybrid Relevance: Closing the Feedback Loop
LLM-based relevance scoring provides useful priors but lacks personalization, cannot adapt to drift, and is uninterpretable when it fails. A hybrid relevance model closes the loop by combining multiple signals:
- LLM Signals: Provide a strong starting point or prior.
- Behavioral Feedback (Clicks, Dwell Time): Offers grounding in real-world user interactions.
- Human/Negative Feedback: Corrects model hallucinations and provides high-precision, unambiguous signals that can even be given veto power.
This is not a fixed formula but an adaptive policy that weighs heterogeneous signals based on their statistical properties.
Agentic Search in Production
The Economics of Reasoning
A tiered approach to cost and latency makes agentic systems economically viable. The majority of traffic is handled by efficient, low-cost methods, while complex queries receive the deeper, more expensive reasoning they require.
- Cache Patterns: Nearly free, ~10ms latency for deterministic, repeated intents.
- Distilled Models: Inexpensive, ~50ms for simple reasoning.
- Single-Pass Agent: Moderate cost, ~200ms.
- Full Reasoning: High cost, ~500ms for the most complex queries.
A modern production architecture often involves real-time query routing (e.g., Flink), a cache layer, a stateful agent orchestrator (e.g., Temporal), and a hybrid search backend with multi-stage ranking. Such a system can achieve a P50 latency of around 100ms with a low zero-result rate.
The Future: When Search Dissolves into Understanding
The trajectory of search is heading towards a future where it becomes more anticipatory and embedded in our digital experiences.
- Near-Term (by end of 2024): Multi-turn clarification, cross-session memory, and real-time user learning.
- Mid-Term (by 2026): Domain-specialized agents, reasoning-as-a-microservice, and anticipatory search.
- Long-Term: Ambient, multimodal intelligence operating across all devices and contexts.
Ultimately, with the rise of AGI, search may disappear entirely. The better search gets, the less it resembles search, as retrieval becomes synonymous with understanding. Three potential futures exist:
- AGI Anticipates: Information flows to the user without explicit queries.
- AGI as a Router: AGI routes knowledge between specialized agents and humans.
- Reality Querying: Simulation becomes a query primitive, making “what if” scenarios computable at scale.