Turn taking

How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen

How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen

Shawn, CTO of PolyAI, explains why voice agents are harder than text agents: voice adds time, and good conversation depends on adapting to the person on the line. He describes Dialog-RSN-1, an audio-native model that predicts turn-taking, replies in text with citations, and writes the transcript last for auditability, along with training on real noisy calls, latency and what agents should do while thinking, voice personality, benchmarking, harness ownership, and the shifting bottleneck from producing content to validating it.

⏭️ Forward Deployed: Voice AI on what works in 2026

⏭️ Forward Deployed: Voice AI on what works in 2026

This podcast episode delves into the pragmatic realities of building and deploying voice agents in enterprise. Featuring leaders from Decagon, Vapi, Retell, Daily, and Smallest AI, the discussion uncovers why the current state-of-the-art relies on cascaded pipelines (STT -> LLM -> TTS) instead of direct voice-to-voice models. Key challenges explored include managing latency versus intelligence, ensuring system reliability through fallback models, the complexities of turn-taking, and architectural strategies to overcome LLM context limitations and optimize costs. The experts also touch upon the differences between inbound and outbound use cases, multilingual considerations, and the potential future of hybrid voice agent architectures.