Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
A hands-on workshop from Langfuse on evaluating AI applications where mistakes are costly. Using a banking assistant as the running example, Annabell Schäfer and Doneyli De Jesus walk through the AI engineering loop: tracing with PII masking, building datasets, turning domain-expert comments into failure modes, mixing LLM-as-a-judge and code evaluators, and CI gates that block prompt changes when scores drop.
Why evaluating AI agents is hard
The workshop opens from a simple observation: AI agents can fail while every system metric stays green. The speakers use a customer support example. A user asks for a refund, the agent looks up the policy and replies that the refund has been issued, and the user says thanks. Latency is fine, nothing throws an error, and traditional observability shows nothing wrong. Days later the user returns to say the refund never arrived. The tool was never called, and the agent claimed an action it never took. Because these systems are non-deterministic, the authors argue, teams need a continuous way to mine production for failure signals and turn them into safeguards, rather than relying on confidence from a handful of checks.
Langfuse grew from that gap. Its founders, while in Y Combinator, noticed there was nothing like a unit test for AI applications. The platform is now open source and is used by more than 150,000 developers. Twenty-one of the Fortune 50 run it in their core AI stack, and it ingests more than 90 billion observations a month. The speakers note that the product started as a developer tool but now also serves domain experts and coding agents, which increasingly work with AI applications on behalf of engineers.
The AI engineering loop
The core of the talk is a loop with five practices. Tracing is the foundation: it records every step an agent takes, including generations, tool calls, inputs, outputs, cost, latency and tokens, in a form that both humans and coding agents can inspect. Monitoring turns those traces into aggregate signals. These include cost and latency, but also quality indicators such as compliance guardrail hits, or how many users write in all caps because they are frustrated. Monitoring is what surfaces the interesting traces worth examining.
Datasets are a representation of what the application should handle, including out-of-scope requests that should be declined. Each item pairs an input with an expected output, which can be a sample reply, expected behavior, or required keywords. Experiments vary one thing at a time, such as the prompt, model, context or tool description, so the effect can be interpreted. Evals then score the results, either with LLM-as-a-judge, where another model checks the output, or with code-based checks for exact matches or deterministic behavior. The speakers stress that this is a cycle, not a one-time process: production learnings flow back into datasets.
Teams enter the loop differently depending on stakes. A low-stakes internal chatbot might start with tracing and a few monitors, and add test cases over time. A customer-facing system in a regulated industry should not risk unsupervised behavior, so tracing comes first and ground truth follows. The speakers describe three ways to build that ground truth: expert curation, shadow runs where the agent runs silently alongside the manual process and its answers are compared, and fully human-in-the-loop operation where data is gradually sliced into cases confident enough to automate and cases that always need a person.
Demo: a banking assistant
Doneyli De Jesus demonstrates the loop on a banking assistant built for financial institutions that are moving from prototype to customer-facing agents. The motivation is liability: LLMs are non-deterministic, so banks need to reduce unpredictability, reproduce outcomes, and locate failure modes. The demo shows several features. The first is persona impersonation, which tests whether the agent respects guardrails for different customer types, from everyday customers to ultra-high-net-worth clients.
The second is PII masking. Sensitive data such as card numbers or social security numbers is redacted before it is persisted into the observability system, so institutions keep this data in their own systems of record. In the demo, a card number typed in plain text appears fully redacted in the stored trace. The third is automated scoring. Scores run on each new trace and check, for example, whether the answer is compliant with banking rules or whether a guardrail fired. When a customer asks whether savings can go into Bitcoin, the agent declines, an input guardrail flags the request as investment advice, and a score records that intent so the case can be filtered later. The demo also covers multimodal tracing: a recorded contact center call is transcribed, evaluated for violations, and linked back to the original audio.
Building datasets and running error analysis
De Jesus frames datasets as test cases, comparable to QA before a software release. Different datasets measure different qualities of the agent. One set red-teams the agent with prompt injection attempts. Another focuses on disputes that were never resolved. Datasets can be uploaded as CSV files, built through the UI, or created with the CLI, SDK or API. Production traces can also be promoted into datasets once someone has confirmed they are good examples.
The central practice is error analysis. Reading traces by hand is still important, because it reveals whether the agent takes the right steps, calls the right tools and produces the right output. The problem is scale, so the approach is to sample a representative slice of production traces, perhaps 10 to 20 percent, filtered by topic such as disputes or card information, and place them in an annotation queue for a domain expert. The key design principle is to keep the expert’s job trivial. Experts give simple yes/no or 1-to-10 scores and write short free-text comments, which can even be dictated. An LLM then clusters those raw comments into recurring failure modes, such as missing citations, out-of-scope answers, or informal register where a formal tone is required. Those failure modes become the quality dimensions that evaluators measure.
Choosing evaluators: mixing LLM judges and code
For each failure mode the team decides how to score it. LLM-as-a-judge evaluators handle nuanced judgments, but they are not perfectly accurate. Code evaluators, such as a regex or exact-match check, return the same result for the same input every time. The speaker’s main lesson is to blend both, so that deterministic checks anchor a score that would otherwise rest entirely on a probabilistic judge. The demo shows a simple code evaluator performing an exact match, which will always return the same result.
Experiments and automated gates in CI
An experiment runs a dataset against a configuration: a particular system prompt, model, set of tools or data access. Changing one of these and comparing results before promotion replaces what the speaker calls ‘running on vibes’. The workflow ties this to CI/CD. Prompts are versioned, either in Langfuse, which does not charge for this, or in Git. An automation triggers a GitHub CI job whenever a prompt is promoted to the production tag. The job runs the experiment and applies pass/fail gates with minimum thresholds per evaluator, such as the presence of citations, source recall, language match, correctness, and absence of unsolicited upsell. The demo shows a run where the candidate prompt version passed the gates. The speaker notes that the process could go further and open a pull request automatically, but the key value is the evidence trail behind each decision.
Dashboards for stakeholders
The demo closes with dashboards designed to justify the system to business leaders. Over the previous seven days, 1,400 interactions did not need a human agent, at a total cost under $7. The speaker compares that to a human contact center employee’s hourly rate, arguing that the directional business value is clear. A failure-mode chart ranks issues by frequency, showing out-of-scope requests, informal register and unresolved disputes as the top three, with some trending upward. From there, a saved filter lets engineers narrow thousands of traces to a targeted set. Thumbs-up and thumbs-down ratings serve as user feedback, and an 18 percent thumbs-down rate invites investigation. A security-risk breakdown shows 89 investment-advice flags in the same period, which helps teams review those cases.
Working with coding agents
The final section cautions against asking a coding agent to ‘fix my agent, make no mistakes.’ The speakers describe a two-axis picture of time invested and system quality. Fully automating everything produces a low-quality system that customers will not accept. Fully manual work is too slow, because instrumentation, failure-mode discovery and eval runs can be automated. The target is a balance in which coding agents handle instrumentation, failure finding, eval execution and running many experiments, while humans supply taste, domain expertise and the curated datasets that define what the application should do. The speakers call this the tasteful AI engineer.
To start, they recommend installing the Langfuse skill, a set of markdown files that teaches a coding agent how to work with Langfuse. It links to the documentation, integration pages and Langfuse Academy material on datasets, instrumentation and evaluators. In regulated domains, hard limits, such as never giving investment advice, should be encoded in the evaluation set, so that violations are at least detected when they happen.
Q&A highlights
On infrastructure, the speaker explains that Langfuse’s data plane depends on ClickHouse. The product began on Postgres, but at billions of events per day it moved to ClickHouse. Data can be exported to S3 or accessed through the API, CLI or MCP. Hosting is possible on Amazon through the managed cloud, or through a self-hosted deployment for teams that run their own infrastructure.
On LLM-as-a-judge reliability, the advice is to calibrate the judge against human judgments. A human labels a representative sample, the judge is tuned to agree with those labels, and agreement is rechecked over time as the data shifts. In production the recommendation is to accept some false positives at first, catching slightly more than needed rather than missing failures, and then tune the judge down over time.
Resources
The demo repository is public, though its data is fictional. The Langfuse Academy offers a longer write-up of the AI engineering loop, and a self-serve workshop walks through the steps. Doneyli De Jesus gives a talk on building deployment gates for AI agents on Wednesday. The Langfuse team is at booth 9, offering custom AI-generated stickers, and invites attendees to a community run with the founders.