The 100-Tool Agent Is a Trap - Sohail Shaikh & Ankush Rastogi, Prosodica

This session unveils the "Fat Agent" trap in LLM agent architectures, where loading an entire tool catalog into the system prompt causes severe latency, cost, and accuracy problems. It presents the Semantic Tool Router pattern, a practical solution leveraging Just-in-Time Context Injection and vector search (RAG for tools) to dynamically provide only relevant tools, dramatically reducing prompt size, enhancing performance, and improving reliability across frontier models like GPT-4o and Gemini 2.0.

June 28, 2026 5 min read

The 100-Tool Agent Is a Trap - Sohail Shaikh & Ankush Rastogi, Prosodica — source: https://www.youtube.com/watch?v=vh2VGuQ3zhY

This session addresses a critical challenge in scaling AI agents: the “Fat Agent” architecture, which involves loading an entire catalog of tools into the system prompt. While seemingly harmless for small toolsets (e.g., 10 tools), this approach rapidly leads to significant performance degradation, cost increases, and reduced accuracy as the tool catalog grows.

The “Fat Agent” Problem

The “Fat Agent” design provides the model with every tool definition, including names, descriptions, and JSON schemas, on every request, regardless of user intent. This leads to several issues:

Context Overload: A catalog of 741 tools can consume over 127,000 tokens before the user’s query is even processed, leading to massive prompt sizes.
Accuracy Collapse: Benchmarks show that with 10 tools, a “Fat Agent” achieves ~78% accuracy in tool selection. This drops drastically to ~40% with 100 tools, and a mere 13.6% with 741 tools. The model struggles with “lost in the middle” problems, where it pays less attention to information in the middle of long contexts.
Increased Latency: Time-to-First-Token (TTFT) significantly increases as the tool catalog grows, with agents potentially taking over 5 seconds to respond with 500 tools. This makes real-time applications sluggish and unpredictable.
Higher Costs: Sending hundreds of thousands of tokens per request for tool definitions accumulates rapidly, leading to billions of tokens processed daily in production and substantial operational costs.
Monolithic Design: “Fat Agents” become difficult to test, risky to update, and painful to debug due as they are large, undifferentiated systems.

The Solution: Semantic Tool Router with Just-in-Time Context Injection

The proposed solution is the Semantic Tool Router, which employs Just-in-Time Context Injection to dynamically provide only the most relevant tools to the model for each request. This pattern draws parallels with Retrieval-Augmented Generation (RAG) systems but applied to tool selection.

How it Works (RAG for Tools)

Offline Indexing: Each tool is assigned a clear, descriptive text. These descriptions are then embedded using an embedding model and stored in a vector index (e.g., ChromaDB, Pinecone, Qdrant). This step is performed once or when the tool catalog is updated.
Runtime Retrieval: When a user query arrives, it is embedded using the same embedding model. A nearest-neighbor search is performed against the vector index to find tool descriptions closest to the query.
Context Injection: The router retrieves the top-K (typically 3-5) most relevant tool schemas. Only these selected schemas are injected into the LLM’s context for the current request. The model then makes a decision from this focused, small set of tools.

Benefits

Stable Accuracy: The Semantic Tool Router maintains accuracy above 83% even with large catalogs (e.g., 1041 tools), as the model always operates on a small, relevant set of tools.
Reduced Latency: TTFT remains consistently low and stable because the prompt size is controlled, regardless of the overall tool catalog size.
Significant Cost Reduction: By reducing tool context tokens by up to 99% (e.g., from 150k tokens to 2k tokens as observed by Anthropic), costs are drastically cut.
Improved Reliability: The model is less prone to “cross-tool confusion” and inventing non-existent tools, as it chooses from a carefully curated list.
Enhanced Understandability: The router not only injects relevant tools but also actively removes irrelevant ones from the model’s choice set.

Evaluation and Benchmarks

Prosodica conducted benchmarks using data sets like the Berkeley function calling leaderboard, Skills Bench scenarios, and synthetic tool pools, scaling up to 1041 tools. They compared “Fat Agent” and Semantic Routing using the same models (GPT-4o, Gemini 2.0) and tool catalogs. Key findings:

Accuracy: Fat Agent accuracy dropped from ~78% (10 tools) to ~13% (1041 tools), while the router stayed above 83%.
Latency: Fat Agent TTFT increased with tool count; router TTFT remained nearly flat.
Token Reduction: Up to 99% reduction in tool context tokens per request.
K-Value: A K (number of retrieved tools) of 5 was identified as a strong default, balancing speed/cost with coverage for edge cases.

Implementation and Production Checklist

Implementing a Semantic Tool Router involves a focused effort, not a major platform rewrite:

Centralized Tool Catalog: Define tool name, description, schema, owner, and version.
Build Vector Index: Embed tool descriptions and store vectors in a chosen vector database.
Develop the Router: Implement logic to embed user queries, perform nearest-neighbor search, and retrieve top-K tools and their schemas.
Integrate into Agent Loop: Ensure the agent receives tool lists only from the router.
Evaluate: Test with K=3, 5, 10 to find the smallest K meeting accuracy targets.
Monitor Production: Log selected tools, actual tool calls, failures, and fallback usage to continuously improve.
Re-embed Tools: Update embeddings when tool descriptions or schemas change.

Trade-offs and Risks

While highly beneficial, semantic routing has considerations:

Router Misses: The router might fail to retrieve a necessary tool. This can be mitigated with fallbacks, widening K, or running a second retrieval pass.
Weak Tool Descriptions: Poor descriptions lead to weak embeddings and inaccurate retrieval. Descriptions should be clear, intent-focused, and use language users expect.
Rare Tools: Some tools may not be retrieved without carefully crafted descriptions that include relevant language or explicit biasing.

Conclusion: The goal is to prevent forcing the LLM to reason over irrelevant tools. For systems with fewer than 20 tools, static loading might suffice. However, as the tool catalog grows and issues with prompt size, latency, or tool confusion emerge, a Semantic Tool Router becomes essential for building scalable, reliable, and cost-effective AI agents.