Build Hour: Voice Agents

An agent is an application composed of an AI model, instructions to steer its behavior, and connections to tools that augment its capabilities. The agent operates in a dynamic execution environment, allowing it to determine when it has met its objective and stop executing.

Voice agents are at an inflection point due to rapid improvements in voice models and integration tools, alongside growing user adoption. Their compelling nature stems from three key attributes:

1. Flexibility: Modern agents handle a wider range of intents and ambiguous situations compared to older, more deterministic systems.
2. Accessibility: Voice offers a more natural and convenient interface than text, especially in mobile or hands-free contexts.
3. Personalization: Voice agents are highly expressive and can perceive nuances like tone and cadence, which are lost in transcription.

Architectural Approaches for Voice Agents

There are two primary architectures for building voice applications:

• Chained Approach (STT → LLM → TTS): This traditional pipeline uses a Speech-to-Text (STT) model to create a transcript, which is then processed by a text-based LLM (like GPT-4.1) to generate a response. Finally, a Text-to-Speech (TTS) model converts this text response back into audio.

  • Pros: This approach is modular, allowing developers to plug and play different models for each stage. It also enables the reuse of existing text-based agent pipelines.
  • Cons: The transcription step is inherently lossy, discarding crucial vocal cues like emotion and tone, which Prashant Mital describes as “what gets lost in translation”.

• End-to-End Speech-to-Speech Models: This newer, more advanced approach uses models that understand audio natively, reason over it, and directly produce audio output tokens. These models power features like ChatGPT’s Advanced Voice Mode and the Realtime API.

  • Pros: They are significantly faster and more “emotionally intelligent” because they bypass the lossy transcription step.
  • Cons: They may have more limited reasoning abilities compared to large text-based models. However, this can be mitigated by delegating complex tasks to smarter models.

Building with OpenAI’s Realtime API & Agents SDK

Recent releases have significantly simplified the development of real-time voice agents:

• TypeScript Agents SDK: A new SDK with feature parity to the Python version, but with first-class, built-in support for the Realtime API.
• Platform Traces for Realtime Models: All input and output audio from agents built with the SDK is automatically logged to the OpenAI platform dashboard. This is a “huge unlock” for debugging speech-to-speech models, as it allows developers to listen to the actual audio conversation that led to a specific completion.
• Improved Realtime Model: The June 3rd model snapshot offers significant improvements in instruction following and tool-calling accuracy, along with a speed parameter for granular control over the AI’s pace of speech.

The SDK abstracts away complexities like using WebRTC in the browser or WebSockets on the server. A developer can turn any agent into a real-time voice agent with a single line of code change, switching from a standard constructor to the RealtimeAgent constructor.

Multi-Agent Systems and Handoffs

A key primitive in the Agents SDK is handoffs, which allow one agent to delegate control to another within a conversation. This enables the creation of sophisticated, multi-agent networks where different agents are specialized for specific domains or tasks.

For example, a Greeter agent can hand off to a Math Tutor agent once it determines the user needs help with a math problem. This is achieved via a tool call under the hood.

This pattern was central to the live demo, which evolved from a single agent to a multi-agent system:

1. Workspace Manager Agent: A foundational agent with tools to create and manage tabs in a web-based workspace.
2. Designer Agent: A specialized agent, prompted to be an expert in interior design. By separating this from the workspace manager, the agent’s focus is narrowed, improving the quality and relevance of its design-related outputs. As Brian Fioca noted, this allows the agent to “focus on its design task by narrowing down its latent space and its autocomplete potential.”
3. Estimator Agent: A third agent designed to handle budgeting and scheduling, allowing the designer agent to remain focused on creative tasks. This demonstrates the principle of telling an agent what not to do by providing a clear handoff path for out-of-scope tasks.

Delegation: Combining Real-time Speed with Advanced Reasoning

A powerful pattern demonstrated was the combination of a fast, real-time speech-to-speech model with a slower, more powerful text model like GPT-4.1.

The Designer agent (real-time) was given a tool called make_workspace_changes. When this tool was called, it didn’t execute simple UI commands directly. Instead, it made a server-side call to a separate agent running on GPT-4.1. This “smarter” agent could perform more complex logic, like searching the web or making multiple, intricate changes to the workspace, before returning the result. This frees up the conversational, real-time agent to focus on interaction, while offloading heavy cognitive tasks.

Productionizing Voice Agents: Evals, Guardrails & Debugging

• Evaluations: For deterministic workflows (e.g., “create three specific tabs”), developers can write simple integration tests that assert tool calls were made correctly. For more complex, conversational flows, the recommended approach is to create model-graded evaluations. This involves running mock conversations and using a model to grade whether the agent successfully followed the desired workflow.
• Debugging with Traces: The platform’s Traces tab is the starting point for debugging. It provides a complete record of a conversation, including the user’s audio, the agent’s audio response, and all tool calls made. Soon, developers will be able to convert these traces directly into evaluations.
• Guardrails: The SDK supports output guardrails to ensure agents stay on-topic and behave safely. The guardrail operates on the transcript of the agent’s response as it’s being generated. Because the transcript is available faster than the audio playback, the system can interrupt the agent mid-sentence if it violates a constraint (e.g., starts talking about a non-approved topic). The agent is then notified of the interruption and can correct its course.

• Tags:
• Feature
• Speech-to-Speech
• OpenAI
• Multi-Agent Systems
• Evals
• Voice Agents
• Agents SDK
• Realtime API
• Guardrails