Voice AI Where One Wrong Question Breaks the Rules - Shiv Indap, Upstart & Brooke Hopkins, Coval
Shiv Indap of Upstart and Brooke Hopkins of Coval walk through automating loan verification calls in a heavily regulated lending environment. They cover the compliance traps that shape the agent's behavior, moving human review beyond a Google Sheet, and raising LLM judge agreement with human reviewers from 33% to 92%. The talk then lays out a continuous loop of live monitoring, human review queues, simulation-based testing of fixes, and self-healing agents, with humans signing off on every change.
Automating loan verification calls at Upstart
Upstart handles thousands of lending calls every day, and in a highly regulated industry a single misstep is not just a customer experience problem but a potential regulatory one. Shiv Indap, who leads voice AI at Upstart, describes the first agent the team launched: an automated verification call for applicants who are not instantly decisioned. About 90% of applicants receive an instant decision, while the remaining 10% call in to verify information. The agent asks three questions: whether the applicant’s income sources will continue for the next three months, how they plan to use the funds, and whether they have outstanding obligations such as alimony or wage garnishments. Before asking any of these, it confirms that the caller is the applicant, so the agent cannot be used to probe details of applications the caller does not own.
Where compliance gets tricky
The use case looks simple, but compliance rules shape nearly every branch of the conversation. If an applicant says the funds are for medical purposes, the agent cannot ask for the amount, and asking would create a serious problem. If an applicant says the funds are for travel to a graduation, the agent has to clarify whose graduation it is, because that determines the true use of funds. Applicants also often ask about their status or when they will receive the money. The agent deliberately sidesteps these questions politely, staying strictly within the single use case that was approved.
From a Google Sheet to structured human review
Every call was initially reviewed by a human, and those reviews were tracked in a Google Sheet. Within a few weeks the sheet grew from five columns to fifteen as operators added fields at will, and the process became unmanageable. The team needed structured data both to improve the agent and to track its performance, which led them to build a formal human review process with Coval.
LLM judges, metrics, and closing the agreement gap
Once a call completes, an LLM judge scores it automatically. For compliance, the judge answers a yes or no question about whether requirements were met and provides an explanation. The team tracked many metrics at first, including call time, accuracy, clarity, completeness, policy alignment, correct escalation, and CSAT, then narrowed the list to those that mattered most. Shiv’s advice is not to become too attached to metrics: start with the ones that seem logical, and discard any that stop reflecting what drives the use case.
Human reviewers audit the judge’s output, and the agreement between the judge and reviewers on compliance started at only 33%. Closing the gap happened in stages. First, a large amount of tribal knowledge that had never been coded into the agent was written down, which raised agreement. Next, the team discovered that human reviewers disagreed with one another for the same reason, since the same unwritten knowledge governed their decisions, so they worked to encode that too. The largest later jump came when they examined more use cases and realized the judge itself needed improvement. Agreement is now at 92%.
Continuous improvement instead of a one-time release gate
Brooke Hopkins, founder of Coval and former head of evaluation infrastructure at Waymo, where her team built the developer tools for launching and running simulations, argues that enterprises often treat release as a single gate. They launch once, and then it becomes very hard to change a model, a prompt, or anything underneath without breaking the system. The alternative is a continuous loop of monitoring, defining metrics, flagging calls, and improving the agent. She frames evals as the product definition: deciding what the agent should and should not do is where teams should spend most of their time.
Evals fall into three broad categories: whether the call reached its end, whether the agent followed the required steps in order, and how good the conversation and experience were. Human reviewers should label what they care about, and those labels should become automated evals. The reverse approach, starting from a proxy metric such as word error rate that doesn’t drive any action, is a common mistake. Metric types include LLM judges, heuristics, and audio processing.
Monitoring live calls and routing failures
Metrics run against live calls to detect known failure patterns at scale, such as a missed verification of loan details or incorrect wording. Clustering and anomaly detection surface unknown unknowns, the failures no one anticipated. Because no system is perfectly reliable at scale, even at Google-level reliability, failures will happen, so they are routed into review queues. Human reviewers then focus on the calls that matter most instead of reviewing every call. Each failure category goes to the team that can act on it: off-script or wrong answers go to the agent team, compliance violations go to a compliance team that can file a process or take tactical action, and other issues go to engineering. The point is to build an operational practice where every incoming failure leads to a concrete action.
Re-simulating fixes before they reach production
Once real failures are identified, the team proposes a fix and re-simulates it to confirm that the issue is actually resolved. Coval’s simulations run voice-to-voice, calling the agent with varied background noise, accents, voice quality, and interruptions. Brooke describes this as shifting left, a DevOps term: catch as many failures as possible in simulation rather than in production, where testing a fix means risking impact on real users.
Self-healing agents with humans in the loop
The loop can extend to self-healing. Agents can review evaluation results and identify fixes, while RL pipelines or fine-tuned LLMs help the system improve over time automatically. Every change is still gated by human review. Each time a reviewer labels a metric as incorrect or confirms a true failure, that feedback either aligns the metric with human judgment or provides data that can drive automatic improvement of the agent.
Keeping humans in control
Evals do more than score calls. They determine which calls need human attention, which matters because reviewers cannot look at thousands of calls each day. Reviewer annotations improve the judge, and the data collected improves the agent as well. Changes are always placed behind a feature flag and must be signed off. Engineers typically approve them, and when a compliance change is involved, the compliance team approves the agent’s behavior. In fintech, this approach became a blueprint for launching agents safely while meeting regulatory requirements and improving performance at the same time. Brooke adds that in regulated industries, robust evals are what make it possible to deploy autonomous systems with precision and confidence, and they define a high-quality product.