12-factor Agents - Patterns of reliable LLM applications // Dexter Horthy

The “80% Problem” in AI Agent Development

The journey of building an AI agent often starts with excitement. Using off-the-shelf frameworks, developers can quickly assemble a demo that seems to work 80% of the time. However, this “80% good enough” is a deceptive milestone. In traditional software, an 80% success rate for an API would be considered a critical outage. For AI agents to be trusted in production, especially for high-stakes enterprise tasks, they must overcome this reliability gap. The top 1% of builders achieve this not by using frameworks blindly, but by recognizing that most successful “AI agents” are primarily deterministic software, with LLMs strategically integrated at key points. The common pattern of “here’s your prompt, here’s a bag of tools, loop until you hit the goal” is often too brittle for real-world applications.

12-Factor Agents: A Methodology for Reliability

Inspired by Heroku’s “12-Factor App” which set the standard for building cloud-native applications, the “12-Factor Agents” methodology proposes a set of conventions for building robust and scalable LLM-powered software. This approach is less about training or fine-tuning models and more about getting the absolute best performance from the models available today through superior engineering. The central idea that underpins these factors is Context Engineering.

The Core Principle: Context Engineering

Context Engineering is the practice of meticulously crafting all the information—the “context”—that is fed into an LLM to achieve a desired output reliably. It’s a more holistic concept than prompt engineering, encompassing:

• The Prompt: Owning every token of the prompt is crucial. While templates are a good start, pushing past the 80% performance barrier requires fine-grained control and the ability to experiment with different phrasing and structures.
• Memory and RAG: How historical information and retrieved documents are represented and passed to the model.
• Agentic History: The log of previous actions, tool calls, and errors.
• Structured Output: The format and schema the model is expected to produce.

The core belief is that LLMs are stateless pure functions. The only thing that influences their output (besides model choice and parameters like temperature) is the input tokens. Therefore, to control the output, you must engineer the input. Experimentation is key; for example, using XML instead of JSON to pass structured data can be more token-efficient and improve performance because it is more “meaning dense.”

Key Architectural Patterns for Production Agents

1. Start with Structured Data Extraction: The most fundamental and magical capability of an LLM is turning unstructured input (like a user’s request) into structured data (like JSON). This is the first step toward building agentic software, as it allows deterministic code to take meaningful action.

2. Embrace Stateless Transducers over Simple Loops: Long-running agent loops are fragile and struggle with long context windows. A more robust architecture is a stateless transducer.

   • The core agent logic is a stateless function.
   • State (like conversation history or job status) is managed externally, for example, in a database.
   • The loop works as follows: fetch the current state, build the context, call the LLM to decide the next step, execute that step with deterministic code, and persist the new state.
   • This pattern makes agents resumable, enabling asynchronous workflows. For instance, an agent can call a long-running tool, save its state, and be “woken up” by a webhook when the tool completes, seamlessly resuming the task.

3. Build Small, Focused Agents: Monolithic agents that try to do everything perform poorly. The most effective approach is to use LLMs for specific, well-defined tasks where they excel, embedded within a larger, mostly deterministic workflow. A high-performing agent should have a limited scope, such as fewer than 100 tools and a plan that requires fewer than 20 steps.

4. Treat Humans as a Tool: Instead of just having a final “stop” token, agents should be able to explicitly call a “human” tool. This allows the agent to request clarification, ask for approval, or hand off a task it cannot complete. To maximize adoption, these interactions should happen where users already work, like Slack or email, which requires the asynchronous control flow mentioned earlier.

5. Simplify Evals: When agents are built as stateless functions with clear inputs and outputs, evaluation becomes more straightforward. You can write deterministic unit tests that provide a specific input context (a trace of events) and assert that the LLM produces the correct next step (e.g., “assert the next intent is to list_teams after receiving a team_id error”).

In summary, building great agents is an exercise in software engineering. By treating LLMs as powerful but stateless components within a well-architected system, owning the control flow, and meticulously engineering the context, developers can move beyond the 80% demo and create applications that are truly production-ready.

• Tags:
• AI Agents
• Software Engineering
• LLM
• Context Engineering
• Feature
• Production AI
• 12-Factor Agents