How to build agents that take ACTION
Alex Salazar, CEO of Arcade, argues that the true value of AI is not in chatbots but in agents that can take real-world actions. He details the primary reasons agents fail to reach production—security, cost, latency, and accuracy—and introduces an "Agent Hierarchy of Needs" as a framework for building robust, production-ready agents. The talk emphasizes a critical shift from exposing raw APIs to building intention-based tools and solving the complex challenge of agent authorization through a delegated model.
The real value of AI software lies in workflow automation, which requires agents to move beyond conversational chat and take tangible actions on other systems. An agent that cannot perform an action is simply a “chatbot with a cooler name.” However, moving from a compelling demo to a production-ready agent is a significant challenge, with an estimated 70% of agent projects failing to launch.
The Production Gap: Why Agents Fail
There are four primary reasons why most AI agents never make it to production:
- Security: A major blocker is the inability to securely connect to essential third-party systems like Gmail or Salesforce. Implementing delegated user authorization (like OAuth 2.0) on behalf of a user is incredibly difficult, and many teams cannot get it right.
- Cost: The economics often don’t work. To improve accuracy, developers stuff more context into prompts and use complex reasoning loops (e.g., Chain of Thought), which dramatically increases token consumption and drives up the cost per request beyond a viable ROI.
- Latency: The same factors that increase cost—multiple LLM calls and complex processing—also lead to high latency, creating a poor user experience that makes the application unusable in a production environment.
- Accuracy: If an agent isn’t accurate and reliable (typically north of 80%), users will not trust it. This lack of trust leads to a lack of adoption, rendering the agent useless regardless of its technical capabilities.
The Agent Hierarchy of Needs
To build agents that can overcome these challenges, developers should follow a structured approach, conceptualized as a hierarchy of needs.
1. Evals: The Foundation
Testing and evaluation are the foundational layer. Because agents are non-deterministic, traditional software testing methods are insufficient. Developers must start by creating a set of evaluations—even a simple spreadsheet of expected inputs and outputs for key business scenarios. This practice grounds the agent’s scope, provides a benchmark for performance, and ensures the agent consistently does what it’s intended to do.
2. Model Selection
The choice of a Large Language Model (LLM) is becoming less of a differentiator, as flagship models from providers like OpenAI and Anthropic are generally “good enough” for most use cases. However, for building agents, the most critical performance metric for a model is its effectiveness in tool selection and parameter prediction. Currently, proprietary models significantly outperform open-source alternatives in this specific capability.
3. Orchestration
Orchestration frameworks (e.g., LangGraph, Crew AI, OpenAI Assistants SDK) are essential for defining the agent’s workflow. When choosing a framework, developers should prioritize simplicity and, most importantly, robust support for human-in-the-loop (HITL) functionality. Nearly every agent in production requires some form of human oversight or approval, making HITL a critical feature.
4. Tools: The Action Layer
Tools are where an agent transitions from reasoning to acting. A common mistake is to simply provide an LLM with a raw API. This fails because APIs require structured input (e.g., Unix timestamps, unique user IDs) that models cannot reliably generate.
The correct approach is to build intention-based tools. Instead of exposing a low-level service contract like the Google Drive API, create a tool that represents the agent’s goal, such as find_brochure. This abstracts away the complex, multi-step logic (e.g., searching, unpacking a MIME structure to reply to an email) into a single, reliable function. This approach dramatically reduces LLM calls, lowers costs and latency, and significantly increases accuracy.
5. Agent Authorization
Authorization is the final and most difficult hurdle. Two common failure patterns are:
- Agent as an Identity: Giving the agent its own “non-human” identity creates massive security risks. If an intern with low privileges can access an agent with high privileges (e.g., one that can see executive compensation), it creates an “authorization bypass vulnerability.”
- Using User Credentials: Allowing an agent to use a user’s raw credentials is extremely unsafe. The agent inherits all the user’s permissions, including destructive ones like deleting files or emails.
The correct and secure method is delegated authorization. This model allows an action only if it meets two conditions: the agent application is pre-registered and authorized to perform the action (via its scopes), AND the user on whose behalf it’s acting also has the permission to do so. It operates on the intersection of the agent’s permissions and the user’s permissions.
A Hot Take: The Diminishing Role of Data Quality
There has been a fundamental shift from “Old AI,” which was model- and data-centric, to “New AI,” which is more akin to software development through composition. The heavy lifting of model training has already been done by companies like OpenAI and Anthropic. For most agent developers today, the primary focus is on the other layers of the stack—orchestration, tools, and authorization. While high-quality data remains valuable, it is no longer the primary gatekeeper to building effective AI systems.