How to Build Voice Agents That Don't Fall Apart When You Interrupt - Jesse Hall, LiveKit
Jesse Hall of LiveKit explains how to build voice agents for financial services that feel human rather than robotic. The talk covers why model leaderboards mislead for real-time conversations, why time to first audio and time to first sentence matter more than first byte or first token, and how async tool calls keep a conversation flowing during slow back-end work. It also covers audio-aware turn detection, the roughly 1.5-second latency budget, and benchmarking your own stack with simulated callers.
Real-time voice agents are judged by a different standard than background agents. A coding or research agent can take ten minutes without anyone minding, but a caller on the phone notices every pause, interruption, and dropped detail. In financial services the stakes are higher still: if a caller changes a payment amount or date midway through and the agent fails to keep up, errors flow downhill and trust collapses. The goal is an agent that feels human, and latency is only one part of that.
Why leaderboards don’t pick your stack
Model leaderboards score components in isolation, but what you ship is a conversation. If speech recognition hears “the 30th” when the caller said “the 3rd,” every downstream component plans around the wrong date, and the LLM has no way to know the input was wrong. Leaderboards also tend to rely on clean audio, while real callers are on speakerphone, in traffic, with kids in the back seat. The useful question is not which model tops a chart but which stack performs best for this use case, and that can only be answered by testing the stack end to end.
Measure what the caller actually hears
Time to first byte looks impressive on a provider dashboard, but a byte is not audio. Popular TTS providers reporting around 210 ms to first byte have been seen shipping up to three-quarters of a second of silence before audible speech. The metric that matters is time to first audio: how long until the caller hears something. The same logic applies to language models. Time to first token is a text measurement, and a TTS model needs a complete sentence before it can start speaking, so the relevant figure is time to first sentence. The model that emits tokens fastest is not necessarily the one that delivers its first full sentence soonest.
Async tool calls keep the conversation moving
Back-end work such as looking up a transaction, checking payment status, or submitting a dispute can take anywhere from a couple of seconds to forty or more. A human in that situation does not go silent. They say “give me a second, I’m looking that up” or move on to another question. Agents can do the same. The key is making the tool call asynchronous. Awaiting it hands the microphone back to the agent, so the conversation and the tool no longer share a single thread. If a tool returns an error, the agent can verify and correct itself, and in the worst case hand the caller to a human.
In a hotel booking demo built on this pattern, the agent collected dates, guest count, room type, view preference, name, email, phone number, and a credit card number. It saved each piece of information as it arrived without waiting for the back end to confirm, and the caller could interrupt mid-sentence without the conversation breaking down.
Turn detection that listens to the audio
Knowing when to speak and when to keep listening matters as much as speed. Most turn detectors rely on the words and a pause length. Consider a caller saying “I’d like to pay 500 and $55.23.” A transcript-only detector sees “500,” hears a short pause, and cuts in. An audio-aware detector hears the inflection that signals the caller isn’t finished and waits. LiveKit’s detector also weighs rhythm, pitch, and intonation. In the presenters’ testing on their hotel agent, the leading alternative interrupted callers 9.9% of the time, compared with 4.5% for LiveKit’s detector. These evaluations are open source, so teams can run them against their own data.
The latency budget
Latency is the gap between the caller’s last syllable and the agent’s first audible response. Inside that window the turn detector decides the caller is done, the LLM begins preparing a response from a provisional transcript, and TTS generates audio as soon as it can. Every model, function, and handoff competes for the same fixed window. Staying around 1.5 seconds keeps the call intact, though it starts to feel slightly off. Getting down to 500 to 600 milliseconds is where the exchange truly feels human. So the question is not how fast the stack can possibly go, but how human-like an agent can be while fitting inside the budget.
Benchmarking with simulated callers
Because every agent is different, the only reliable benchmark is your own. Start with a complete working agent that has a real back end, then run simulated callers against it. These callers pursue goals, change their minds, interrupt, mumble, and talk over background noise. Text simulations test business logic. Audio simulations also exercise speech recognition and turn-taking across the full voice pipeline.
Each scenario defines what the caller wants, which actions the agent should take, and what the outcome should be. Scoring checks outcomes such as whether the correct reservation reached the database or the correct authorized amount was charged in a financial workflow. Completion alone is not enough, because agents can game it. Evaluation also examines the path: did the agent take ten steps when two were needed, and did it hallucinate a function call?
To compare stacks, keep the agent and scenarios fixed, swap the models, and rerun. Every model receives byte-identical input. Repeating this loop produces a personalized benchmark, and the model that answers fastest for your use case may not be the one at the top of a leaderboard. On the hotel agent, Gemma 4 31B served through LiveKit inference ranked highest, while the same model on another provider was slower, reflecting inference optimized for voice workloads. LiveKit is model-agnostic, so any model can be plugged in.
Pre-launch checks for financial voice agents
A practical pre-launch checklist sets the latency budget from the caller’s perspective and measures time to first audio and time to first sentence. It confirms critical dates, adds explicit checks for anything involving money, and verifies identity before any account is accessed. It also requires the agent to keep listening while slow tools run. Full conversations should be run repeatedly through simulation, testing the final state, privacy safeguards, unnecessary tool calls, and turn-taking. Each failure found should become a regression scenario, so the test suite grows alongside the agent.