We Kept Rebuilding Our AI Agent. Here's What Survived - Fuad Ali, Arize AI

From a 50% hallucination prototype to one harness-driven agent

Alex, the in-product agent inside Arize, began in late 2023 as a prototype on GPT-3.5-class models. A staff engineer described the first version as “50% hallucination, 50% [something usable]”. It was a router over a set of skills with a tool-calling loop, and it was bad at math. The team moved arithmetic out of the model and into harness code, then rewrote the system in August 2025 with iterative tool calling and a to-do tool. Alex 2.0 followed in February 2026 with Sonnet 4.5 as the default model. In June, the team merged its specialist agents into a single agent that loads skills on demand. Long-term memory came last. Every part of the system is custom harness code: the loop, the plan, the stopping rule, and the memory. The team deliberately avoided an agent framework so they could control and learn from each piece.

Who owns the plan

The first attempt gave the model a to-do tool. The model spent most of its calls checking items off the list, which produced many LLM calls with no useful work. It also lost focus after 20+ iterations. The team moved the plan into the harness one step at a time. They first added an in-progress marker so the model would not lose its place. Then the harness began rendering the plan as a message on every model call, so the rules sit next to the data. They dropped the in-progress marker because anything unfinished, in stack-ranked order, is simply the next task. They are now testing with no to-do tool at all, with the plan kept in the message history. The harness holds the state and the completion check, and the model just consumes them and acts on the next step.

What ends a turn

The first rule was that a response with no tool calls means the agent is done. That broke with models that announce a step in one turn and make the call in the next. Alex lets users bring their own provider, so the team cannot test every model a customer might use. The answer was an explicit finish call that the harness gates. If plan items remain open, the finish call is bounced back up to three times. A warning appears one step before the iteration limit, and a forced finish follows if the limit is reached. The team runs regression tests across models and keeps an opinion on a default, but the guardrails exist for the models they have not tested.

Memory: treat big results like files

The team describes the context window as a desk. A single huge tool result, such as a large JSON blob from another agent’s trace, fills the desk and leaves no room to work. Their fix is to keep a pointer, a sticky note that says where the data lives and what it contains. The harness truncates large results and lets the agent use code tools like jq to query the full data when it needs it. Agents are good at writing code, so they lean on it to infer things. The agent’s own trajectory is checkpointed by the model, and the history has a fixed token budget.

One small file beat a knowledge graph

For long-term memory, the team tried vector retrieval, episodic and semantic stores, and temporal knowledge graphs. What won was one markdown file per user, capped at 8,000 characters. It is loaded on every full request and rewritten by a single background job after each session finishes. Because only that one job writes memory, the team can keep cross-user data from leaking into a file. The file is small, exact, and easy to load. It does not inflate context, and a human labeler can read it, check whether the right things are being stored, and refine the background job’s instructions. The team’s reflection is that engineers like building complex systems, and that over-engineering was the trap they had to climb out of.

Specialists versus one agent

In 2025 there was one trace agent. Then came ten specialists, one per page, such as a sub-agent for the dataset page that could generate synthetic data. The team rolled that back and now uses one agent that loads skill instructions and tools on demand and decides which tools to use. The team sees this as both a reflection of better model capability and a simplification.

The newest model isn’t always the best

What counts as the best model depends on the task. The team defaults to a model that is fast and good enough, and spends most of its effort on the harness. They are exploring auto-routing, which picks a model per task, and small post-trained models for narrow surfaces. Some surfaces need no agent loop at all and can use a single model call. The example given is the AI filter feature, where small models now generate filters well, so the full loop would be over-engineering.

Production sessions as tests

Alex’s own production traces feed the test suite. Every real user session becomes a regression case and a living fixture that grows as new error cases appear. Sessions can be stored as datasets, so a specific turn can be replayed through the agent. Arize experiments are used to A/B test harness changes.

In agents, bugs rarely appear as explicit errors. A status of OK can hide a problem, and 43 identical tool calls in one turn is a sign that something needs fixing. Latency is also judgment-dependent: 227 seconds is clearly too long, but where the line sits depends on the use case. Signal, the team’s trace-reading agent, now catches issues such as retry loops, so humans read fewer traces than before.

What still needs a human

Judges need a human baseline. Gold labels and golden datasets still come from people, and evaluators are aligned to match what engineers consider correct. Trade-offs between speed and quality are human decisions, even when metrics show that changing models will raise latency by X and change quality by Y. The metrics support the decision, but someone still has to make it.

What survived every rebuild

Six things stayed through every rewrite. The first is tracing everything Alex does. The second is putting math and data in code: code computes the aggregations, and the model reasons about and explains them. The third is golden datasets in CI. The fourth is errors that have an obvious fix. The fifth is streaming and persistence. The sixth is skills, which the team now defines as giving the model a capability around an external system. The decisions every agent builder will face are who owns the plan, what ends the turn, what to do with oversized tool output, who writes long-term memory, whether to use one agent or many, which model is fast enough, and who checks the agent’s own fixes. The team’s own tagline captures the approach: everyday they use Alex to build Alex.

• Tags:
• Context Management
• LLM Agents
• Tool Calling
• Agent Observability
• Agent Memory
• Agent Evaluation
• Arize
• Agent Harness Design
• Agent Planning
• Production Sessions as Tests