Building Agentic Tools for Production // Sam Partee
Sam Partee, CTO of Arcade AI, explains that building production-grade agentic systems requires moving beyond simple chatbots. He details the critical components for creating reliable, secure, and scalable tools, including rigorous schema management, the principle of least privilege, continuous evaluation, and a crucial distinction between 'exploratory' and 'operational' tools.
Building agentic tools that are robust enough for production environments requires a disciplined approach that goes far beyond typical chatbot development. The focus must shift to creating tools that are reliable, secure, and meticulously designed for interaction with Large Language Models (LLMs).
The Bare Minimums for Production-Ready Tools
To deploy an agent with non-deterministic components into a production environment, certain foundational practices are non-negotiable.
1. Generate and Curate Tool Schemas
A common failure point is the maintenance of tool schemas separately from the tool’s code, which often leads to drift and errors. The recommended approach is to generate schemas directly from the tool’s implementation, for instance, from annotated function parameters.
This practice also allows for enforcing constraints that improve an LLM’s performance. It’s crucial to restrict data types to those LLMs can handle reliably. Avoid complex, high-cardinality types like a list of lists of dictionaries. An LLM can more easily and accurately choose a value from a simple enum with three string options than it can generate a correct value for an infinite-cardinality type like a floating-point number.
2. Enforce Authorization Requirements per Tool
For any agent that takes action on behalf of a user—rather than simply gathering context—each tool must have its own specific authorization requirements. This aligns with the principle of least privilege, a critical security concept, especially in enterprise settings.
For example, accessing a file system involves distinct levels of privilege: reading a file is different from writing a file, which is different from deleting one. Each of these actions should be governed by a separate, tightly-scoped tool with the minimum necessary permissions.
3. Evaluate Tools, Not Just the Agent
While many teams focus on evaluating their overall agent’s performance, they often neglect to evaluate the individual tools. Tool-specific evaluations are essential for ensuring reliability. A robust CI/CD process for tools involves running an “evaluation gauntlet” for every tool against a suite of different LLMs on a regular basis. This helps identify how changes to a tool’s description or schema affect its behavior and performance with various models.
A Framework for Database Tools: Exploratory vs. Operational
Database tools are particularly sensitive as they can mutate critical data. A useful mental model is to categorize them into two distinct types: exploratory and operational.
Exploratory Tools: The “Thinkers”
These tools are designed for open-ended context gathering and are meant to be used iteratively with a human in the loop. An exploratory task, like “group users by UTM source,” might involve a sequence of tool calls:
- Discover available databases.
- Discover collections/tables within a database.
- Inspect the schema of a relevant collection.
- Aggregate or search documents within that collection.
A critical rule for these tools is that they should not have write privileges. They must be read-only to prevent any unintended data mutation during the discovery process.
Operational Tools: The “Doers”
In contrast, operational tools are designed to perform specific, narrowly-defined actions. They should have a limited and verifiable set of possible outcomes, making them safe enough to be used in an “ambient” or automated fashion without direct human oversight.
A key design pattern for operational tools is to use enums to constrain inputs. For example, instead of a tool that accepts arbitrary start and end dates, create one that uses an enum like AggregationWindow with options such as YESTERDAY, LAST_WEEK, or LAST_MONTH. LLMs understand these semantic concepts much more reliably than date strings. An update_user_payment_plan tool where the plan is an enum is another example of a safe, operational tool.
The real power of an agent emerges when it can combine these two types of tools: first using exploratory tools to understand a situation and then using operational tools to act on its findings.
Advanced Techniques and Architecture
A “Cheat Code”: Returning Prompts from Tools
A highly effective, yet simple, technique is to have your tools return a natural language prompt as part of their output. This is especially useful for guiding the LLM. For instance, if a search tool is called with parameters that are too restrictive, it can return a message like: "You searched too narrowly. Widen out your search to the 'medium' scope." This directly instructs the LLM on how to correct its next action.
Managing Scale with a Federated Gateway
As you develop a rich library of exploratory and operational tools, you face a new problem: providing too many tools to an agent at once degrades its performance. The solution is a gateway architecture. This allows you to host multiple tool servers (or “MCP servers”) and then, through the gateway, assign specific, curated subsets of tools to each agent. This way, an agent only has access to the tools it needs for its designated task, ensuring both security and optimal performance.