⏭️ Forward Deployed: Voice AI on what works in 2026

This podcast episode delves into the pragmatic realities of building and deploying voice agents in enterprise. Featuring leaders from Decagon, Vapi, Retell, Daily, and Smallest AI, the discussion uncovers why the current state-of-the-art relies on cascaded pipelines (STT -> LLM -> TTS) instead of direct voice-to-voice models. Key challenges explored include managing latency versus intelligence, ensuring system reliability through fallback models, the complexities of turn-taking, and architectural strategies to overcome LLM context limitations and optimize costs. The experts also touch upon the differences between inbound and outbound use cases, multilingual considerations, and the potential future of hybrid voice agent architectures.

August 25, 2026 5 min read

⏭️ Forward Deployed: Voice AI on what works in 2026 — source: https://www.youtube.com/watch?v=MwNvowwcZOo

The episode introduces a new “Forward Deployed Engineering” (FDE) focused podcast on Latent Space, hosted by Basil Chatha. Chatha, previously a product manager and consultant building AI agents, created the podcast to provide practical insights from industry leaders, distinguishing real-world applications from marketing hype. The FDE track at AIE, which he hosted, was well-received, focusing on deep dives into agentic engineering, voice agents, computer vision agents, and enterprise AI.

This particular episode focuses on the “hot” but challenging domain of voice agents, featuring insights from Basia Sudol (Decagon), Varun Singh (Daily), Steven Diaz (Vapi), Tyler D’Silva (Retell AI), and Sudarshan Kamath (Smallest AI).

Current State of Voice Agent Architecture

The current state-of-the-art for voice agent deployment relies on a cascaded pipeline, described as a three-step process:

Speech-to-Text (STT): Converts user’s speech into text.
Large Language Model (LLM): Processes the text, generates a response, potentially performing tool calls.
Text-to-Speech (TTS): Converts the LLM’s text response back into speech for the user.

Why not direct Voice-to-Voice (V2V) models? While V2V demos can appear snappy and natural, they currently lack the reliability and intelligence for real-world enterprise use. Cascaded models, despite their complexity, offer:

Enhanced Control: Allows for rigid guardrails, supervisor models to detect prompt injection or social engineering, and precise intent selection.
Context Optimization: Enables pre-checking conditions to select relevant prompts based on customer type, compiling a refined system prompt for the LLM.
Grounding in Truth: Facilitates verification of factual accuracy (e.g., correct dates).
Parallelization: Engineers continuously optimize the cascaded pipeline by parallelizing steps to shave off milliseconds and achieve a snappy feel despite multiple underlying processes.

Key Challenges in Voice Agent Deployment

Latency vs. Intelligence Trade-off: More intelligent and nuanced responses from LLMs typically incur higher latency. Balancing this trade-off is crucial and depends heavily on the specific use case. What feels “good” for latency is constantly evolving.
- Mitigation: The use of contextual fillers (e.g., “hmm,” “just a sec”) can bridge latency gaps, making the experience feel more natural while the system processes complex requests.
Reliability: LLM providers can experience outages (e.g., Opus going down). A robust voice agent architecture requires waterfall models or fallback mechanisms to ensure continuous operation.
Turn-taking: Distinguishing between a user pausing to think and the end of a conversational turn is a non-trivial problem. Solutions involve Voice Activity Detection (VAD) and sophisticated turn models.
LLM Context Window Limitations: LLMs tend to “forget” information in the middle of long prompts, primarily retaining the beginning and end (e.g., first and last 4%). This can lead to hallucinations if critical context is lost.
- Mitigation: Compaction models dynamically summarize or reduce context to prevent overload. This challenge has been a long-standing issue in voice AI, predating similar problems in coding agents.
Prompt Design and Cost:
- Giant System Prompts: While convenient, large system prompts can be expensive, especially when calls are short-lived or abandoned early, as costs are incurred for all tokens processed.
- Workflow-based Prompts: Depending on the use case (e.g., inbound with predictable flows), a node or graph builder approach for prompts can be more efficient. For erratic user behavior (e.g., in outbound calls), a central, comprehensive prompt might be necessary.
- Situational Prompting: Dynamically including or excluding parts of a prompt based on real-time relevance can optimize cost and performance, though models are improving in handling large contexts.
Inbound vs. Outbound Use Cases:
- Outbound: Often simpler (e.g., debt collection). Bots can be more constrained and drop calls if users go “off the rails,” reducing complexity. Users may also hang up upon realizing it’s a bot.
- Inbound: Significantly more complex due to unpredictable user intent and vast potential knowledge bases (e.g., an Amazon customer support agent needing access to billions of products and policies).
Multilingual and Accent Support:
- Cascaded Advantage: Allows swapping out specific components (e.g., using custom TTS engines for challenging languages like Japanese or Arabic) that perform better than generic models.
- Smallest AI’s V2V: Initially focusing on English for intelligence, while cascaded solutions see demand from diverse regions like India and Latin America. Noise cancellation remains a key challenge for accents.

The Future: Hybrid and Specialized Models

While cascaded architectures dominate, Speech-to-Speech (V2V) models are evolving. Smallest AI’s “Hydra” V2V model aims for more natural, asynchronous human-like interaction, where the system thinks and processes in parallel, similar to how humans listen and formulate responses.

Trade-offs: V2V models might initially lag behind specialized STT in accuracy, but research is addressing interpretability and multimodal input/output (speech and text).
Hybrid Approach: The mid-term future will likely involve a hybrid architecture where V2V handles the active conversational loop for naturalness, while cascaded elements manage complex lookups or tool calls.
Specialized Models: Training smaller, fine-tuned models for common, high-volume use cases (e.g., 50% of customer queries) can offer lower latency, reduced cost, and higher reliability than relying solely on large frontier LLMs, especially given the variability of public API services. Companies like Smallest AI offer specialized SLMs (e.g., Electron) for this purpose.

The discussion emphasizes the need for continuous evaluation and benchmarking (e.g., turn-based STT, LLM, TTS benchmarks from Daily) to test new models and workflows, especially for forward-deployed engineers dealing with real customer interactions.