AI Agent Development Tradeoffs You NEED to Know

Sherwood Callaway of 11X discusses the architecture of "Alice," an AI Sales Development Representative. He covers the practical decision to use LangGraph for its reliability in production, the challenges of infrastructure and observability when using hosted agent platforms, and their methodology for running Evals to mitigate hallucinations by comparing generated content against source data.

July 22, 2025 5 min read

AI Agent Development Tradeoffs You NEED to Know — source: https://www.youtube.com/watch?v=sni_-awZTgw

Sherwood Callaway, Tech Lead at 11X, explains that as AI models improve, the complex scaffolding currently seen in agent frameworks will likely collapse into simpler structures. However, the reality today is that models have limitations, and building reliable, production-grade agents requires more structured approaches.

From Voice AI to AI Sales Reps

Callaway’s journey into agentic AI began at his YC startup, Opkit, which created a vertical AI phone-calling solution for the healthcare industry to automate calls with insurance companies. He notes the significant difference between building voice agents and text-based agents like “Alice,” the AI Sales Development Representative (SDR) at 11X. Voice is a richer medium where inflection and tone are critical, and processing these cues in real-time with low latency remains a major challenge for the industry.

At 11X, the focus is on “digital workers,” specifically Alice (AI SDR) and Julian (AI voice agent for inbound sales). Alice automates the entire outbound sales process, from building prospect lists to personalizing and sending multi-touch email sequences.

What Makes an Agent “Agentic”?

For 11X, an agent is a system that can automate complex and ambiguous tasks that were previously un-automatable. At its core, an agent operates on a loop that involves a language model. This loop consists of:

Planning: The agent assesses its environment and objectives.
Acting: It takes an action, typically by calling a tool (e.g., scraping a website, querying a database).
Observing: It processes the result of that action.
Reasoning: It determines if the task is complete or if another action is needed.

Alice exemplifies this by first performing deep research on a prospect, creating a comprehensive report using internal data and web scraping. It then generates a sequence of personalized messages, deciding on the timing, content, value propositions, and personal details to reference, such as congratulating a company on a recent milestone. A key product challenge is balancing the agent’s autonomy with the user’s desire for control, as experienced sales leaders often have strong opinions on strategy. The goal is to provide results by letting “Alice do what she does best” on autopilot.

The Agent Framework: Why LangGraph?

When 11X rebuilt Alice, they were heavily influenced by the agentic capabilities of the Replit agent, which demonstrated the power of a multi-step, tool-using AI. They chose LangGraph as their framework for several key reasons:

Maturity and Reliability: At the time, LangGraph was one of the most mature and well-supported frameworks.
Flexibility via Graph Structure: While some argue for simpler agent loops (like the ReAct framework), graphs provide the necessary structure and reliability for a production-grade product. You can’t have an agent that only “works some of the time.” The graph allows developers to enforce guardrails and predefined paths to ensure consistent performance for specific steps.
The Model vs. Scaffolding Trade-off: A core tension exists between building complex graph structures for current model limitations and the risk that a new, more capable model will render that complexity obsolete. This creates a constant “premature optimization” analysis: is the investment in the current graph structure going to pay off before a new model makes it unnecessary? This can lead to a “sclerotic” agent, locked into a structure that is no longer optimal.

Infrastructure: The Trade-offs of a Hosted Agent Platform

11X runs its agents on LangGraph Cloud (now LangGraph Platform). This decision was a trade-off between speed and control.

The Agent as an API: The agent is bundled into a container and runs on LangGraph’s servers. 11X interacts with it via an API endpoint. This architecture follows the “Agent Protocol,” a standard proposed by LangChain for exposing agents as services. An API call places a job on a queue, which is then picked up by a consumer that executes the agent process. This queueing system ensures that no requests are dropped during traffic spikes, though execution may be slower.
Benefits: The primary advantage was speed. It allowed the team to avoid building complex infrastructure and focus on the application, especially during a rapid rebuild where the “go-to-market was not slowing down.”
Challenges:
- Scaling: As an early adopter, they encountered scaling limitations that required workarounds, like round-robin routing across multiple deployments.
- Deployment Complexity: Their services are in a monorepo with a unified deployment system. Integrating the agent’s separate deployment pipeline added significant complexity.
- Observability Silos: A core principle of observability is having all telemetry (logs, metrics, traces) in one place. 11X uses DataDog for its services, but the agent’s observability lives in Langsmith. This creates two “walled gardens,” forcing developers to look in multiple places to debug issues. Getting data out of Langsmith and into DataDog was not a seamless process.

Observability and Evals for Production Agents

Callaway emphasizes that agents are just software and should be instrumented with the same core tools: metrics, logs, and traces, preferably using an open standard like OpenTelemetry.

Their practical approach to ensuring quality and mitigating hallucinations involves a multi-tool strategy:

Langsmith: Used for observing agent traces within the LangGraph Cloud environment.
Arise: An LLM observability platform they are currently using for Evals.
The Eval Process: They sample 1% of their production message-writer traces to run an “LLM-as-a-judge” eval. The eval compares the generated email (the output) against the research report that was used as its source material (the input). It specifically checks for any facts or claims in the email that are not supported by the research report, effectively running a hallucination check. This produces a score that serves as a key metric for monitoring the reliability of the agent, especially after prompt changes.
DataDog: Remains the central observability platform for all other backend services.

This setup highlights the current fragmentation in the AI tooling market, where companies are often forced to stitch together multiple solutions to cover the full lifecycle of building, deploying, and monitoring agents.