Don't Approve the AI Model. Approve the App - Doneyli De Jesus, Langfuse
Doneyli De Jesus, AI engineering lead at Langfuse, describes how a top-five bank built deployment gates for AI agents in an on-premises, air-gapped environment. He covers golden data sets, multi-dimensional scoring, why a model that was 97% grounded still failed on numbers until it was given a calculator tool, and how domain experts set thresholds and review failures, cutting evaluation time from weeks to minutes.
Regulated institutions rarely make deployment decisions based on public leaderboards. Banks treat their data, workflows, and fine-tuned models as competitive advantages, and public benchmarks run on open datasets that don’t reliably predict how an application will perform on internal data. The bank in this case put it plainly: ‘we don’t certify models, we certify applications.’ The use case was a 10-K analyst assistant for investment bankers, who need earnings call transcripts analyzed in minutes. Summarizing a call is the easy part. The harder problems are injecting the firm’s institutional knowledge and context into the system, and then showing reliably that its outputs are correct enough to clear a release gate.
Infrastructure constraints: on-premises, gated, air-gapped
The bank’s requirements fell into three categories:
- On-premises: Banks are rethinking the data centers they moved away from during the cloud era and standing up GPU infrastructure to run models locally, because of how sensitive the data fed into these models is.
- Gated: Traffic cannot leave to the open internet. Even with zero-data-retention guarantees, the appearance that a model provider might train on bank data is unacceptable. Some banks have research divisions with PhDs training and fine-tuning their own models.
- Air-gapped: Only the people who need the data should see it, and it should never leave their infrastructure.
What ‘ready for production’ means
The bank needed to answer three questions: does the application work the way it should, does it meet regulatory compliance requirements, and will users actually adopt it? A poor experience erodes trust in AI-enabled tools as a whole, so usability matters as much as correctness. The goal was a pass/fail approval that is re-triggered whenever anything in the agent’s configuration changes, including the system prompt, the model, or the tools it can call.
Golden data sets and traces
The golden data set is the anchor of the process, much like test cases in a software QA pipeline, where nothing reaches production without passing. Many banks already have one, but they often lack a way to operationalize running it. Each run produces traces, which serve as the evidence of how the system behaved on each case. Scores computed from those traces produce pass/fail results across the dimensions that matter.
Grounded but bad at math
Early comparisons of several models, from small to large, surfaced a telling pattern. One small model scored 97% on groundedness, meaning it drew reliably on the bank’s internal sources. But its numerical accuracy was around 60%. LLMs are next-token predictors, not calculators, so financial figures and simple arithmetic broke down even when retrieval was correct.
The fix was architectural rather than a matter of swapping in a bigger model. The agent workflow includes planning, retrieval from official sources, and a calculation step. For that step, the team gave the model a deterministic calculate tool, since arithmetic should return the same answer every time. Numerical accuracy rose to about 95%, and the application cleared the bank’s 85% gate. Because the controls live in the application and its configuration rather than in the model, the quality of the application should hold even if the underlying model is replaced tomorrow.
Building the data set and scoring
A data set is a set of key pairs: inputs, expected outputs, and metadata used to evaluate quality, behavior, and responses. Scoring combines three approaches:
- Deterministic checks: Code-based assertions that return the same result for the same input.
- Path evaluation: Whether the agent took a correct route to its answer. A response can look complete while the process behind it skipped steps or took an inappropriate route.
- LLM-as-a-judge: Assessment of qualitative aspects of an answer.
As solutions grow more complex, a blend of deterministic and non-deterministic evaluators becomes necessary, because relying solely on LLM judges is not sufficient.
Multi-dimensional gates and domain experts
Applications should not be judged by an average score. In one full end-to-end run, which used a 150-question data set and a 1,500-question data set, the agent passed groundedness and completeness but scored zero on regulatory compliance. That dimension is binary, and a single violation can bring penalties and reputational damage, so it can block promotion to production regardless of the other scores. Each dimension needs its own threshold, and those thresholds vary by use case.
Setting those thresholds cannot be left to technical staff alone. Risk and domain experts need to define the bar the application must meet before the business feels comfortable deploying it internally or in front of customers.
Annotation queues and feedback loops
Failure modes are curated into annotation queues. For example, every compliance failure is routed to a queue where a domain expert can open the trace, examine what happened, and provide feedback that feeds back into improving the agent. Changes to a prompt, model, or configuration trigger a re-run of the gate evaluation. Expert-reviewed traces are also strong candidates for the golden data set, because they capture a validated standard for how a particular dimension should be judged.
Results and takeaways
This process reduced application evaluation, especially in risk and compliance work that often takes months, from weeks to minutes. The workflow is iterative, and there is no established template yet because the industry’s practices are still forming. Teams need to define the metrics that matter for their own context.
The central lessons are these. Build your own benchmark, because public results don’t reflect your data. Gate the application rather than the model, so that swapping models doesn’t mean rebuilding your controls. Bring domain experts in to set the thresholds your organization considers reasonable for promotion to production. In regulated industries, where data stays on-premises and public information plays a limited role, your own data, your own bar, and your own gates are what make AI deployable.