How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen
Shawn, CTO of PolyAI, explains why voice agents are harder than text agents: voice adds time, and good conversation depends on adapting to the person on the line. He describes Dialog-RSN-1, an audio-native model that predicts turn-taking, replies in text with citations, and writes the transcript last for auditability, along with training on real noisy calls, latency and what agents should do while thinking, voice personality, benchmarking, harness ownership, and the shifting bottleneck from producing content to validating it.
Why voice is harder than text
Voice agents look like a solved problem from the outside, and Shawn argues the opposite. Most recent AI progress has landed in text and digital environments, where a hard physics question can be researched at leisure. Voice adds a dimension text lacks: time. A reply that is correct but arrives late, or that talks over the caller, fails the conversation even when the content is right. The intelligence required is less about reasoning to the best answer and more about adapting to the person on the line: their pace, confusion, and patience. That is why he compares voice to robotics. Machines struggle in the physical world because the data is sparse, and conversation is similarly rich in signals that machines have rarely observed at scale, even though humans encounter them constantly.
What enterprises actually want
Enterprises want a branded voice that says their name exactly as they intend. They also want an agent that is ‘as powerful as a human being’ yet ‘as controllable as a robot’, a tension that makes adoption slow. The bar is set by the best human on the team, so agents usually earn trust by first proving efficiency in a narrow area and only later being seen as better than expected.
Why PolyAI built its own model
PolyAI had built its own speech recognition and language models before deciding, about a year before recording, that cascaded pipelines would give way to end-to-end models. It stopped investing in its ASR accordingly. Speech-to-speech demos look natural, but in contact centres they break: interruptions are too aggressive, and a confused elderly caller talking to a bot for the first time gets spoken over. Frontier labs, he suggests, are not optimizing for enterprise turn-taking, governance, and compliance. Consumer voice agents are already workable; enterprise ones need considerably more.
The audio-native architecture
Dialog-RSN-1 is a language model that perceives audio directly. Instead of chaining ASR, turn-taking, and an LLM through separately tuned parameters, audio chunks stream into the model, which first predicts a turn-taking token: has the speaker just started, are they still speaking, or have they finished? This token is cheap to generate, and because the audio is already encoded by the time the turn ends, the model does not re-encode the utterance before responding.
The response itself is text, streamed like a typical chat API, which keeps guardrails straightforward, since speech output is much harder to constrain. After the response, the model emits citations pointing to the knowledge base facts it used. The transcript comes last by design: it is needed for auditing and debugging, but no caller waits for it.
He contrasts this with waveform-to-waveform systems such as GPT Realtime, which sound more natural and break the uncanny valley but are harder for enterprises to constrain. Retrieval changes too. With no transcript to embed, the model generates a search tool call instead, which is essentially agentic search. The innovation, he stresses, lies less in the base model than in the input-output contract: how data is prepared and fed in, how output is structured, and how training pairs shape behaviour. Because strong open-source models appear every month, the team keeps training scripts ready to adapt whichever base performs best, and treats its data as the moat.
Training data, privacy and noise
The training data comes from live deployments with banking, utility, logistics, restaurant, hotel, and retail outbound clients. Using it required enterprise agreements covering GDPR and other privacy obligations, and many customers agreed because they could see the model improving for them. PII is redacted before training and replaced with synthetic PII, so the model learns to recognise names and addresses without memorising customer data. The aim is to learn the mechanics of conversation rather than the content of any individual call.
On the input side, the encoder works on mel spectrograms, processed segment by segment, which makes the task feel closer to image recognition. Shawn recalls Victor Zue’s early claim that speech could be read from spectrograms, and notes that a violin shows up as horizontal harmonic lines while percussion appears as vertical bars.
Noise is deliberate. Real calls already carry plenty of it, but the team layers generative TTS and audio models on top, so a synthetic voice pronouncing his name might sit over a crying baby. The surprising lesson was that the previous pipeline’s denoising and sanitisation made the new model worse. It had never encountered the messy conditions it would face in production, and over-cleaned audio proved to be a poor teacher.
Cocktail parties, crosstalk and turn-taking
Crosstalk has long been a problem, as the cocktail party effect shows. Humans attend to one speaker and filter out the rest, while ASR historically could not. Speaker diarisation, which labels speakers one, two, and three, is a cascaded-pipeline solution that feels unnatural. People in a crowd do not label voices; they decide which one to attend to. Because the LLM perceives audio natively, it can be prompted to ignore background speakers or to engage with them, and the training data and instructions determine which behaviour emerges.
He sees one-on-one turn-taking as the current frontier. In roughly ten years, he imagines meeting rooms with microphones supporting multi-party agents that know who is speaking and what is private. He believes the technology will get there, but consumer adoption is a separate curve, since voice is personal and people must surrender control to an agent they trust.
Latency and what the agent does while it thinks
Model processing is rarely the bottleneck. Deciding when the user has finished is. In cascaded systems, a tunable silence threshold is a crude compromise: too short and older callers get interrupted, too long and impatient callers get frustrated. Because the model hears prosody and semantics, it can tell that a mid-sentence pause is not the end of a thought. Hosting models alongside the rest of the platform also removes network hops, which dominate tail latency at P95.
The time saved is being reinvested in adaptive reasoning. The model is trained to be latency-budgeted, learning when extra thinking is worth it and when it should answer within about a second. Human interviewees often open with filler while they think, and voice agents face the same expectation. Tactics include varied filler phrases, typing sounds, hold music, and constant feedback, because even a few seconds of silence makes callers wonder whether the system has died. Trust in contact centres is fragile, since callers have been burned by automated systems before, and slow enterprise backends make all of this harder.
Voice, personality and the uncanny valley
Most enterprises remain conservative about anthropomorphism, in line with Apple and Google, which give assistants a voice without pushing personality too far. A South American casino wanted a regional southern American voice that opened with ‘howdy’, but did not launch it because it felt like too much. Generic voices perform worse: callers recognise them as robots immediately, associate them with older IVR systems, and hang up or try to bypass the agent.
The best-performing voices carry a light regional accent. A Newcastle voice signals to UK callers that a human is on the line. A surprising finding from early deployments was that English voices sounded unnatural, while a French voice forced to speak English, keeping its French accent, sounded the most real. The imperfection is what made it convincing.
Benchmarks
Public benchmarks suffer from the same problems as ASR leaderboards: vendors choose their own tests. Voice agents are harder to evaluate because the criteria span latency, response quality, reasoning difficulty, accuracy in capturing names and personal details, transcription, instruction following, citations, and tool calls. Most public voice benchmarks are synthetic, since real voice data is private and difficult to release.
PolyAI’s internal benchmark is built from real customer conversations and includes latency, because a voice agent that makes a caller wait sixty seconds is not a working agent. The team plans to release it to the research community within a month or two. Subjective qualities such as tone and cultural fit resist objective measurement. In production the signals are behavioural: whether callers want to keep engaging, and whether they finish the task. Measuring those requires evaluating the whole harness, not the model alone.
Harness engineering and owning the intelligence
Enterprises want to own their AI, but they cannot realistically tune models or afford the GPUs, so they give up on owning models and try to own the harness. The result is many teams reinventing the same wheel. Shawn thinks not every harness needs to be owned. Voice is a harder product problem than general reasoning, and enterprises value the reasoning more than the adaptation.
His suggested architecture makes the voice agent the customer-facing front door, owning the brand and the adaptation, while delegating heavy tasks to a back-end reasoning system the enterprise controls. The key decision is where to draw the boundary: what to keep and what to hand off.
Well-specified problems and auditable agents
Customer service is more well-specified than much software work. Success criteria are concrete: was the problem solved, did the customer feel well treated, and how quickly was it resolved? Unlike human chitchat, which has no gold standard, business conversations have a clear objective, which makes automatic optimisation with coding agents practical.
Enterprises are less ready for autonomous organisations of agents exchanging messages and merging code. A large utility customer worried that agent-generated pull requests would flood review cycles that already take two weeks, and it wanted everything auditable. PolyAI responds with visualisations of the agent’s flow, a bounded toolset so the agent cannot act outside it, and a citation for every response. No-code flow builders are shifting from being the primary interface to being an audit surface, though that visual layer remains essential.
Weight adaptation versus harness customisation
Adapting skills and code is brittle. They do not transfer easily between agents across an organisation, whereas adapted weights can be shared. Shawn agrees that some behaviour is inherent and cannot be fully fixed by harness rules, which tend to accumulate red tape. Fine-tuning carries its own danger, however: without deep expertise, a team can end up with a suboptimal model. For most enterprises, harness work should come first, with model tuning as a second resort.
Cognitive debt and the shift from production to validation
Shawn describes controlling coding agents from his phone while out walking. He found that agents sometimes spawned new agents rather than using established ones, and that visual context still matters. The bottleneck has shifted from producing content to validating and auditing it. Inside his company, colleagues send walls of text over Slack and email, which recipients then hand to an AI for summarising. He argues that people should only send AI output after reading and understanding it themselves.
Voice avoids much of this. Production voice agents average about 50 output tokens per turn, so the content is small. The cost lies in the reasoning traces of larger agents. Voice has its own limitation, its ping-pong rhythm offers little bandwidth, so a voice interface for coding would need harnessing to deliver high-level bullet points rather than a stream of detail.
Agents at work
He runs several agents at once across email triage, video editing, and coding, a breadth-first workflow that the well-specified nature of these domains makes possible. His daily tools are Claude, Slack, and Chrome, and he opens Chrome so agents can perform computer use after he has logged in. Voice input through Wispr Flow works well for him despite accent-related transcription errors, since agents reason around mistakes that would confuse a human recipient. Using voice in an office still draws stares, much like the reaction to Meta glasses, so privacy and social norms slow voice adoption.
He builds tools that make applications agent-enabled: an abstract model of the domain, plus an agentic CLI with a skill prompt, for applications such as DaVinci Resolve or Blender, where a bare agent would go wrong. The gap is that his mother could not do this. The goal is for non-technical people to demonstrate a task, describe their reasoning, and then have an agent interface built and corrected through use. He has also built internal wikis tracking product features, sales pitches, and competitor research, and has dropped default connectors in favour of direct access to sources.
The next decade and slop
Shawn expects voice agents to become more end-to-end, with turn-taking folded into the full pipeline, since it is the most annoying part to engineer. Voice will branch. Consumer voice will serve entertainment and daily tasks, while enterprise voice will serve regulated, auditable contact centres, support, meeting notes, and coordination. Hardware devices and a second wave of IoT could carry voice naturally.
The technology should eventually become invisible, taught much as one teaches a child. Sharing organisational context is still the hard part. Culture values alone are not enough, because agents latch onto single values without the reasons behind them, and much of what makes an organisation work has never been written into markdown files.
On slop, Shawn offers a definition: generation without competence. Output backed by sufficient context is not slop to its author, but the recipient may lack the context that gives it meaning, so it reads as slop. Whether humans should read everything or delegate summarising to personal agents is an open debate. His team leaned towards personal summarisation skills, though he admits this risks running in circles, with humans relying on agents to pick out useful information from messages written by agents.