Full Workshop: Realtime Voice AI — Mark Backman, Daily

An in-depth look at building real-time, production-grade voice AI agents using the open-source Pipecat framework. This summary covers the core concepts of voice AI pipelines, the shift to speech-to-speech models like Gemini Live, and advanced techniques for managing latency, context, and turn-taking.

August 3, 2025 4 min read

Full Workshop: Realtime Voice AI — Mark Backman, Daily — source: https://www.youtube.com/watch?v=nxuTVd7v7dg

Building high-quality voice AI is challenging because human conversation is incredibly nuanced. Users have high expectations: agents must be intelligent, conversational, fast, and natural-sounding. The benchmark for end-to-end latency in a conversation is around 800 milliseconds, a difficult target to achieve.

Pipecat is an open-source Python framework from Daily designed to tackle these challenges. It enables developers to build voice and multimodal AI agents by providing robust orchestration and a modular architecture.

The Pipecat Pipeline Architecture

The core of Pipecat is the concept of a multimedia pipeline, a series of processors that stream data. A traditional, or “cascaded,” voice agent pipeline consists of several distinct services working in sequence:

Transport Input: Receives raw audio from a source, such as a WebRTC call.
Speech-to-Text (STT): Transcribes the user’s raw audio into text.
Language Model (LLM): Receives the transcribed text, processes the request, and generates a text response.
Text-to-Speech (TTS): Converts the LLM’s text response back into audible speech.
Transport Output: Streams the generated audio back to the user.

Pipecat’s primary strengths are orchestration and modularity. It manages the flow of data through this pipeline and allows developers to “pluck and play” different services without rewriting their application logic. You can easily swap STT providers (e.g., Deepgram), LLMs (e.g., OpenAI, Anthropic, Google), or TTS services (e.g., ElevenLabs, Cartesia) to find the best combination for your use case.

The framework also supports more complex structures like parallel pipelines, which can be used for tasks like implementing a failover strategy (dynamically switching from vendor A to vendor B) or simultaneously processing different media streams like audio and video.

The Shift to Speech-to-Speech Models

A major evolution in voice AI is the emergence of native speech-to-speech (S2S) models like Gemini Live. These models simplify the pipeline significantly by taking raw audio as input and producing raw audio as output, effectively collapsing the STT, LLM, and TTS steps into a single, highly optimized component.

While this simplifies the core logic, the surrounding orchestration for managing the conversation, handling events, and integrating other tools remains crucial. Pipecat provides this orchestration layer. A pipeline using an S2S model might look like this:

# A simplified Pipecat pipeline with a speech-to-speech model
pipeline = Pipeline([
    transport.input,
    llm,  # e.g., GeminiMultimodalLiveLLMService
    transport.output
])

Advanced Concepts and Real-World Challenges

The workshop detailed several critical considerations for building production-ready voice agents:

Voice Activity Detection (VAD): A fast and accurate VAD is essential for enabling interruptions (barge-in). While services like Gemini have built-in VAD, it’s often better to run a highly optimized, local VAD (like the recommended Silero) to detect the start of user speech with minimal latency. This triggers an interruption, creating a more natural conversational flow.
Turn-Taking: A major unsolved problem is knowing when a user has truly finished their thought, rather than just pausing. Simple VAD timeouts can lead to the agent interrupting prematurely. The future is semantic end-of-turn detection, using models that analyze audio cues (intonation, filler words) and conversational context. Pipecat is experimenting with a model called smart-end-of-turn that classifies turns as “complete” or “incomplete” to dynamically adjust the VAD timeout, giving the user more time to think.
Function Calling & Latency: While powerful, function calling with LLMs introduces significant latency in real-time conversations. Unlike conversational text, which can be streamed token-by-token, the agent must wait for the entire JSON object defining the function call to be generated before it can act. This can add 1.5-2 seconds of dead air. For some use cases, it can be faster (though potentially less accurate) to embed data directly into the system prompt rather than using a RAG or tool-based approach.
Transport & Connectivity: For client-server applications, WebRTC is strongly recommended over WebSockets due to its superior audio quality and error correction. For server-to-server connections, such as integrating with telephony providers (e.g., Twilio), WebSockets are a perfectly viable option.
Handling Noisy Environments: Real-world environments are noisy. Pipecat can integrate third-party noise cancellation services like Krisp as a processor in the pipeline. This cleans the audio before it reaches the VAD or STT, dramatically improving transcription accuracy.

Getting Started

The workshop demonstrated building a bot from scratch using Python, asyncio, and the Pipecat framework. The core components involved setting up a transport, an LLM service, and defining the pipeline that connects them. Event handlers, such as on_client_connect, are used to manage the conversational state, for example, by injecting an initial prompt to kick off the dialogue. The “Word Wrangler” demo, a game where the AI guesses a word described by the user, was shown as an example of a more sophisticated application built with Gemini Live and Pipecat.