Building the AI Agent Is Easy. Proving It's Safe Isn't - Jonathan Osborne, NatWest
NatWest's digital assistant Cora is moving from scripted answers to AI agents serving millions of retail customers in a regulated banking environment. The core argument is that building the agent is straightforward, while proving it is safe requires a routing funnel, multi-dimensional quality scoring, human validation that exposes hidden gaps, calibrated LLM judges, continuous production evaluation, and shadow deployments with feature flags.
Building the agent is the easy part at NatWest. Proving it is safe is the hard part, and that is where most of the work went. NatWest is the UK’s second-largest retail bank, with about 19 million customers, and its digital assistant Cora handles roughly 100,000 conversations a day. Until now most of those answers were scripted. If a question had not been explicitly scripted, the customer was stuck and usually had to queue for a human agent. The plan for this year is to move about a third of Cora’s journeys to AI-generated answers grounded in RAG and knowledge bases, with agents that can take action for customers, such as making a payment or changing an address.
Why the risk is different
The model itself is not the main concern, since current models give good answers on average. The concern is the blast radius. With 19 million customers, a few wrong answers are not a bug ticket or an awkward headline. They involve real customers, real money, difficult questions from the regulator, and reputational damage.
A concrete case shows the difference. A customer asks whether they can overpay their mortgage without a charge. A scripted bot spots the keyword “mortgage”, shows a menu, and most likely the customer ends up waiting for a human. An AI agent can answer from the knowledge base that up to 10% of the balance can be overpaid each year with no early repayment charges. But customers digress constantly, and the same customer might then ask to close the account instead. Decision trees handle digression badly, and digressions tend to return. The key point is that an agent able to answer any question can also answer any question wrongly. If the 10% rule does not apply to this customer’s specific deal, the consequences are real.
Four new failure modes
Agents reason, decide and generate, which creates failure modes a scripted system does not have:
- They can be confidently wrong, hallucinating a source while sounding certain.
- They can be talked into things through prompt injection, since the input itself can be adversarial.
- They can be accurate and policy-compliant yet unhelpful, circling without ever resolving the customer’s problem.
- They can be nondeterministic, giving different outputs for the same input.
Because of this, traditional software assertions and years of accumulated regression tests stop being reliable. A scripted system’s failure surface is bounded, so every situation can be written down and tested. An agent’s surface cannot be enumerated, so testing alone cannot establish safety.
Shrinking the attack surface with a routing funnel
Before hardening the model, the team reduced what reaches it. Every conversation first passes a contextual filter that removes events like button clicks, about 20% of traffic. Deterministic pre-classification rules then route based on the utterance and metadata, with no ML or AI involved. Only what survives these gates reaches the models. That includes a gated relevancy and rephrasing step for detailed multi-context questions, an intent classifier that is a neural network over embeddings rather than a generative model, and finally policy rules that confirm the customer should reach that particular back end.
The design is driven by four banking requirements. Access control has to be enforced at each step. Every decision must be auditable and reproducible. The system must stay resilient if a model or the generative layer is unavailable, while still giving the customer an answer. Governance requires that the routing is versioned configuration that can change without a full redeployment, so the bank can respond quickly to market conditions. The guiding principle is that a model was not simply pointed at 19 million customers. The team first asked carefully what actually needed a model.
Quality is a vector, not a score
A single accuracy figure is misleading. A 94% accurate router that is 100% confident is more dangerous than an 85% accurate router that knows when it is uncertain. The team also needed to ask whether an answer was grounded in a real source, whether the customer’s problem was resolved, and whether the system knew when to hand off to a human. Each response is scored across three independent dimensions: safety (could it harm the customer), task (did it do its job) and experience (how did it feel to the customer). Each dimension can be monitored and alarmed separately. This distinguishes a hallucinating system from one that is correct but insufferable to talk to. These failures call for different fixes.
Human validation exposed the averages
The first step was human validation of a payment agent: 111 conversations across 37 payment journeys and four brands, with three independent scorers per conversation. The headline pass rate was 83%, which turned out to be nearly useless. Happy-path scenarios scored 97%, edge cases 76%, and topic changes like the mortgage digression only 43%. A single headline figure concealed a 54-point spread.
Agreement between humans was also uneven. On objective matters such as bias and fairness, scorers agreed 95% of the time. On subjective matters such as whether the bot understood what the user wanted, agreement fell to 64%. Overall human-to-human agreement was around 60%. This matters for automation, because an LLM judge cannot be expected to match a target that humans themselves do not agree on. Human validation also surfaced the two largest causes of failure, and both were engineering problems rather than model problems.
LLM judges that grade too kindly
To scale evaluation, the team tested an LLM judge on an internal complaints agent that drafts investigation summaries for complaint handlers. It has been live since February, and around 14,000 cases were scored by both humans and an LLM judge. On faithfulness, the judge’s scores clustered tightly near the top, while human scores spread much more widely. The average gap looked small, which might suggest the judge was performing acceptably, but correlation with human scores was almost zero. A judge that cannot separate good cases from bad ones is effectively worthless. LLMs grade generously by default, and that tendency must be actively countered.
The judge’s model also became part of the configuration. Changing only the underlying judge model, with identical inputs and outputs, moved a score from roughly 0.8 to around 0.2 relative to the threshold, which could cause breaking changes. The team’s mitigation was to run the judge on a different model family from the agent, so that the two systems have different blind spots. This brought the scores noticeably closer together.
Continuous evaluation in production
Live conversations, including agent and guardrail steps, are captured with open standards, OpenTelemetry and OpenInference. By default 5% of traffic is sampled at random, and all interesting conversations are also scored: handoffs, failures, and especially long conversations where the user is angry. Scores, traces and experiments all land in Comet, which is consumed as a signed partner app on AWS.
The environment is airgapped, with no internet access. The judge is reached over a private path through an internal GenAI gateway. A zero-tolerance policy keeps personally identifiable information off the network, so much of the evaluation runs within those constraints. Different components run at different cadences. Traces are captured in real time, offline scoring runs every four hours, and regression testing is built into CI/CD with a hard gate. Changes to prompts or context can have outsized effects, so every change is compared against a known good baseline before it can be merged. Continuous evaluation is part of the production infrastructure, not a notebook run once at launch.
Generative components and classical ML components fail differently and are monitored differently. For the generative agent, a span-level trace shows each node in the LangGraph workflow, so when something is slow or breaks, engineers or agents can see exactly where. The intent classifier is traditional ML and can drift as human language changes after training. It is monitored with input feature drift, F1 scores computed against human labels, and defined thresholds that trigger retraining. Input feature drift proved to be a useful early warning, declining before the F1 score does. Comet brings both the generative monitoring (Opik) and the ML model performance monitor into one platform, avoiding two separate observability stacks.
Rolling out safely
Deployment began with shadow mode. The new system classified live utterances while the scripted solution still answered customers. This gave the team real production input data at almost no risk. It also resolved a common deadlock in regulated industries, where production data is needed to approve a release, but release approval often requires seeing how the model handles that data. Shadow mode separates observing production data from answering customers, and it replaces one large release decision with a series of smaller, confident ones.
Rollout then proceeds brand by brand, with percentage-based increases within each brand. KPIs determine go or no-go decisions at each stage, some automated and some requiring manual approval. Feature flags make this possible. A global flag can send everything back to the scripted solution, and each back end can be isolated globally or along a dimension such as brand, all within seconds.
Human feedback as a control
In the complaints project, human feedback is a gate rather than a thumbs-up button. A complaint handler cannot rerun an investigation until they have provided feedback explaining why. The rerun reasons are valuable signals, and the process built a labeled dataset of around 14,000 cases, a volume the team rarely manages to collect. The project surfaced the judge calibration problem in production, which led to judge calibration and regression checks. The architecture is also being adopted by another team for its production monitoring.
The agent was the cheap part of this work. The reusable asset is the evaluation spine, which includes the routing, scoring, judge calibration, regression gates, and drift monitoring. The judge model itself must be treated as part of the production configuration, because an unnoticed change to it can break behavior without warning.