Shipping AI Into 5 Major Banks Without a 12-Month Compliance Wait — Al Chen, Cisco
Al Chen explains what it takes to get AI observability software through the security and compliance reviews of major banks. He covers build pipelines, agent-driven CVE remediation with verifiable exploit tests, ClickHouse backup and restore verification, PII and MNPI leakage through LLM judges, and the tradeoffs of zero leakage and data sovereignty, with an emphasis on showing evidence to shorten time to production.
Why banks are a different deployment problem
Al Chen opens with a comparison to his 10th-grade math teacher, Mr. Huff, who graded students on the work behind an answer, not just the answer. He argues that the compliance and risk teams he works with are the same kind of reviewer, only with a multi-million dollar compliance budget. The systems he is talking about are not internal prototypes or employee-productivity RAG assistants. They are production applications that millions of bank customers use every day in web and mobile banking. His team deploys both its software and its engineering processes into some of the largest banks in the world, with the goal of cutting time to production from roughly a year to a couple of weeks.
He is blunt that standard SaaS AI fails the bank review as soon as it is proposed, because outbound egress to the internet creates a path for sensitive data to leave. On-prem deployments into air-gapped environments are more complicated still. He organizes the talk around three pillars: the build pipeline (how banks pull images into their clusters), data resiliency (what happens when a region fails mid-request), and making sure no sensitive information leaks to internal employees or outside the bank’s VPC.
CVE remediation as a system, not a ticket
Compliance velocity and CVEs go together, and he argues that no serious platform should lack a plan for them. Looking up one CVE on NVD, CISA, or MITRE is not the real problem. A platform has many images, each with many CVEs, and must remediate across several banks at once. Those banks run different image versions, use different scanners (Sysdig, Wiz, Prisma), and set their own policies for which CVEs matter, what severity counts, and what remediation SLA applies. Some banks want a specific package version, while others ask for an older one because the newer version has too many CVEs. The team therefore needs a tooling layer that does the heavy lifting: it produces a table telling engineers what to fix and gives security teams explanations for what cannot be fixed.
Many of the unfixable CVEs sit in transitive dependencies of open-source projects. The team has to read release notes and explain to bank security teams why they depend on upstream maintainers. Remediation itself can mean bumping a package or rebuilding an image against a newer base image, while keeping the software working.
He also points out a gap in scanner output. Scanners confirm that a package has been bumped to a given version, but say nothing about whether the exploit is still reachable in the application code. To close that gap, his team builds Harbor tasks, which are controlled environments for agents. Each task includes instructions, a containerized copy of the application, and tests. A typical instruction says that a newer package version supposedly fixes a CVE, so the agent should write an exploit against the older version, confirm it works there, and then test whether it still works on the new version. Harbor can run these trials in parallel across Docker, Modal, or Daytona sandboxes and report results across many runs. His takeaway is that a version bump is not evidence. The team has to show that the exploit is gone.
Disaster recovery and the limits of backup logs
He calls disaster recovery mandatory infrastructure hygiene for banks. The database behind most AI observability tools, ClickHouse, has no managed cross-region failover or replication service. He contrasts this with Redis and Postgres, which do offer managed cross-region failover, and notes that ClickHouse’s own resiliency documentation says as much. Banks are therefore responsible for their own DR design.
The pattern he describes has a primary-region ClickHouse cluster, a job that backs up to a bucket, and another job that restores the bucket into a DR region, where multiple pods hold replicas. The procedure relies on clickhouse-backup, an open-source tool maintained by Altinity, which is itself a competitor to ClickHouse in managing this kind of failover. The tool logs whether backup and restore ran, which produces a clean status table. But that status says nothing about the data at that moment. Bad disks, bad nodes, bad migrations, and replication lag can all leave data stale without any job failing.
To verify correctness, the team compares row counts across tables and checks whether replicas within a region have converged. The takeaway is that the question is data correctness, not whether a job ticked a box.
He adds a warning for architects: replicating to a bucket is also a data residency decision, whether they intend one or not. A bucket does not care whether it sits in Oregon or Virginia, but residency rules care where individual records live. One approach is to replicate only IDs and metadata across regions, keeping the core trace data in the region required by jurisdictional law.
PII and MNPI leaking through LLM judges
The third pillar is the one he thinks banks most underestimate. Most banks believe they leak nothing to employees or outside users. Redaction usually happens at the SDK, gateway, or middleware layer, so the UI may show customer data already masked. The side door is the LLM judges. He asks the audience to read the system prompts for their judges and check whether the reasoning traces contain PII or MNPI.
His example is a simple judge that outputs a score and an explanation. One innocuous-looking line asks the judge to justify its reasoning so a human reviewer can verify the agent’s accuracy. That line can cause the judge to surface unredacted customer information in its reasoning. The UI looks clean, but the backend reasoning does not.
His mitigation has layers. The judge can still read raw trace data, but the reasoning is not persisted by default, while the score is persisted for evals. A check then scans the reasoning. If it is clean, the reasoning can appear in the UI. If it contains sensitive data, the reasoning is suppressed. Many banks want to see judge reasoning even when it touches sensitive data, so the upgrade is a second LLM judge whose input is already redacted. Its output can then be trusted to display. The cost is an extra LLM call per request. In his words, zero leakage and data sovereignty are not free.
From customer problems to engineering problems
He describes trust as a product of evidence. Teams that say “trust us” do not get through reviews. Teams that show the evidence and the work do. Many vendors quote deployment timelines of a few weeks without checking them against a bank’s infrastructure, security posture, and cluster specifics.
He lists the signals that shorten or lengthen timelines. Short timelines come from established processes for pulling images from a registry and GPUs that are already sourced for inference. Timelines stretch by months with formal security reviews, custom LLM gateways, or RBAC configuration. They stretch further when there is no approval process for pulling images into the bank’s registry and no plan for procuring GPUs. Most of these delays, he says, are organizational and human rather than technical.
When a bank customer reports a problem, his team takes ownership and splits it into engineering, workflow, and organizational work streams. The goal is to automate as much as possible. He also wants his team to avoid being just another vendor and to actually solve customer problems. The biggest lesson, he says, is how much is at stake in finance, from checking accounts to mortgages and credit cards, and why so much banking software is old. The line between legacy banking and innovation is moving, he suggests, because of the people in this audience.
He closes by returning to the lunch table after his 10th-grade math test, where students reconstructed problems and compared approaches. He presents conferences like this one in the same way and says he wants to learn how others solve their customers’ problems. He is willing to walk anyone through the broader production checklist behind these three pillars.