Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI

Agents that look right on the surface often fail in ways nobody notices until their behavior is traced and evaluated. A financial agent asked whether to invest in a company answered yes, without asking clarifying questions, without knowing anything about the user’s financial background, and without any ground truth to validate against. Making that failure visible is the starting point for everything that follows.

Why vibes are not enough

Many teams ship agents after checking outputs by eye. That works early on, but in production nobody knows where things break unless there are evaluations. Agent evaluations are hard to build because the behavior is non-deterministic, so it is difficult to know in advance what to measure. Traditional software can be debugged by following a deterministic code path. An agent’s code is often little more than a single agent.run call, and what happens next (which tools it calls, which model calls it makes, whether it retrieves documents or reasons internally) is only visible in traces.

Traces as the foundation

A trace records the agent call, each LLM call, retrieved documents, and tool invocations. In the example, a user asked where their refund was, and the agent called get_order_status and reported that the refund had been issued. The tool choice was wrong, and the trace shows exactly where the reasoning diverged from what the user needed. From traces, you can identify production failures, collect a sample of failing cases for experiments, and then decide what to change: the prompt, guardrails, the tool itself, or the evaluator. A guardrail, for instance, needs its own evaluator to confirm it actually prevents things like sensitive information leaking.

The improvement loop

The standard loop of finding failures, tracing them, forming hypotheses, changing the application, running experiments, and shipping is effective but slow and manual. The proposed alternative is to bring traces, evaluations, human annotations, APM monitoring, the codebase, and ticketing data into one place where a coding agent such as Claude Code, Cursor, or Codex can work on all of it. Setup takes a few steps: install the CLI, create a profile that connects to the observability platform, and install the skills. Through the CLI, the coding agent can download traces, pull and build datasets, create evaluators, run experiments, and act inside the platform UI on your behalf.

What the coding agent found

In the demo, the coding agent analyzed the financial agent’s traces and evaluations and surfaced several problems. About 44% of interactions produced a bad summary. Fabricated answers were common. Tools were redundantly called, wasting time, resources, and money. There were numeric groundedness problems, silent tool failures where a tool reported success but returned no usable result, and hallucination when a tool failed instead of disclosing the failure. The implication is that the fixes span more than one layer: an evaluator to detect silent tool failures, a groundedness check on numbers, and prompt changes so the agent discloses failures rather than inventing answers.

Testing fixes with experiments

An earlier case shows why proposed fixes need testing. Stock queries about markets outside the United States, such as South Korea, were failing. Adding the country to the context was not enough, because the underlying tool was outdated and had to be moved to its stable version. The agent built a dataset from the failing spans and ran a baseline, which had a 0% pass rate as expected. A naive fix did not help, and a third iteration reached roughly 90%, which is better but not yet complete. The value of the workflow is that each proposed edit becomes a measurable hypothesis rather than an unverified change.

Evaluators, judges, and costs

In the demo, experiments ran on a local machine while evaluators ran on the platform, with an LLM-as-a-judge and a code-based evaluator side by side. The choice of judge model is fully customizable. Some teams use cheaper models, while others use top-tier models for evaluations because an agent is only as good as its evals, even if those models are not used in production. Local and open-source models can also be connected. Public benchmarks are useful for orientation but generic, and the right judge depends on the domain, since a finance agent and a legal agent may need different models. Evaluation cost is visible through trace history, and sampling keeps evaluations from running on every trace and call.

Skills versus evaluations

A well-written skill can make summaries better, and that should be done. But evaluations are what keep quality from degrading. A model regression, or a shift in how users phrase their questions, can cause results to drop months after a fix works. Evaluations monitor for that drift and catch regressions before they accumulate.

Keeping the agent focused on the defect

A recurring concern is that coding agents fixate on linting instead of the real problem. Giving the agent trace context helps it focus on the actual failure, such as calling the wrong tool, which linting would never catch. The speaker reported that this has been less of a problem with the skills in place.

Human review and tailored evaluations

Coding agents automate the workflow from observing issues through testing fixes, but they are not magic. Human review of proposed changes is still essential to confirm that edits improve the agent rather than merely passing evaluations or introducing degradation. The strongest recommendation is to build evaluations tailored to the application’s own failures, so that regressions can be detected as traffic, user behavior, and models change. The live repair in the demo was still running when the session closed.

• Tags:
• Claude Code
• Coding Agents
• LLM Tracing
• LLM as a Judge
• Agent Observability
• Agent Evaluation
• Arize
• Experiments
• Financial Agents
• Arize CLI