Real-Time Voice Agents in Production

At Elyos AI, the focus is on building AI agents that can perform tasks with human-level quality and transparency, particularly for home services like plumbing and electrical emergencies. This requires more than a simple Speech-to-Text (STT) -> LLM -> Text-to-Speech (TTS) pipeline; it’s a significant infrastructure and orchestration challenge. The key to success lies in focusing on four pillars: latency, consistency, context, and recovery.

The Challenge of Real-Time Voice Agents

A standard cascaded architecture (STT, LLM, TTS) faces inherent latency issues. Even in a best-case scenario, the P90 latency can range from 500 to 1200 milliseconds, which is too slow for natural conversation.

• STT: Generally consistent, with latency between 100-300ms from providers like Deepgram, Speechmatics, and AssemblyAI.
• LLM (Time to First Token): Highly variable depending on the provider and model. For instance, GPT-4.1 is often faster than newer versions.
• TTS: Can range from 200ms to over 600ms, with significant performance spikes that make conversations feel “clunky.”

Beyond latency, real-world interactions are noisy, and conversations are not linear. Users often change their intent mid-conversation, requiring the agent to adapt quickly.

The Four Pillars of Production-Ready Voice Agents

Success in building reliable voice agents hinges on mastering orchestration across four key areas.

1. Latency

Optimizing for speed is critical. This involves:

• Infrastructure Readiness: Using warm starts for workers and regional clusters to co-locate pipelines close to customers and telephony providers (e.g., Twilio).
• Architectural Efficiency: Keeping tools and business logic close to the orchestration layer to minimize network latency, even in microservices architectures.
• Dynamic Provider Routing: Actively monitoring and routing LLM calls between endpoints (e.g., OpenAI vs. Azure) to mitigate performance spikes based on time-of-day traffic.
• Observability: Investing early in detailed monitoring to quickly identify bottlenecks. Providers are much more helpful when you can supply specific request IDs and performance data.

2. Consistency

While LLMs are stochastic, production agents require deterministic behavior.

• Constrain the Model: Define expected outcomes and minimize the number of distinct flows per journey. Be concise and avoid trying to handle too many tasks in a single runtime.
• Avoid Unnecessary RAG: Retrieval-Augmented Generation (RAG) introduces its own accuracy risks. If you can fetch the correct information deterministically or inject it “just-in-time” based on intent, it is often more reliable than performing a complex retrieval that might return the wrong context.
• Benchmark Against Humans: A head-to-head comparison between AI and human agents can yield surprising results and provide a clear benchmark for performance.

3. Context Engineering

Managing the conversational context is crucial for coherence and accuracy.

• Just-in-Time Context: Start with a minimal prompt and dynamically inject necessary context based on the user’s intent. Once a piece of context has served its purpose (e.g., a tool call result), summarize its outcome and remove the verbose details from the prompt to prevent confusing the model later.
• Smart Sub-Agents: Use sub-agents carefully, as the main agent can lose awareness of their internal state. Ensure there is a mechanism to maintain control of the overall context.
• Track Sentiment and Tone: For voice interactions, monitoring the user’s emotional state provides valuable signals about how the conversation is progressing and whether the agent needs to adjust its approach.
• Clean Handoffs: Clearly define the expected inputs and outputs between different components or sub-agents to ensure smooth transitions.

4. Recovery

Things will inevitably go wrong; a robust recovery strategy is non-negotiable.

• Use State Machines: Model all workflows as state machines in your backend. This allows for clear error handling and guides the LLM through a recovery path when an issue occurs. LLMs are excellent at interpreting clear error messages.
• Post-Runtime Reconciliation: After a conversation, use one or more models to verify that the outcome was correct and, if not, to trigger reconciliation steps.
• Path of Uncertainty: Don’t annoy users by repeatedly asking for the same information if input is unclear (e.g., STT struggling with postcodes or emails). Have a predefined path to handle uncertainty gracefully, which may involve escalating.
• Human Escalation: Always provide an escalation path to a human. This can be a direct transfer or a “human-in-the-loop” process where a person can approve a step or provide instructions to the agent.
• Parallel Judge Model: Run a faster, secondary LLM in parallel to the main conversation. This “judge” model’s role is to monitor the conversation’s flow and ensure it stays on track, intervening with context corrections if it derails.

Key Metrics to Monitor

• Performance: Time to First Token (TTFT).
• Quality: Groundedness (ensuring no hallucinations), conversation quality (interruptions, repetitions), and task completion.
• User Experience: General sentiment of the caller and how they felt their request was handled.
• Improvement: A systematic focus on the most common failure scenarios to drive rapid improvements.

• Tags:
• AI Agents
• Context Engineering
• Feature
• Real-Time AI
• Voice AI
• LLM Orchestration
• State Management