The AI Builder Podcast
How to Build Production Voice AI Agents with LiveKit's Ben Cherry
Production voice AI depends on far more than a strong model: turn detection, interruption handling, memory, guardrails, escalation — and the latency-versus-reasoning trade-off.
What you’ll learn
- The central trade-off in voice AI is latency versus reasoning: a fast model runs the live conversation, a larger model handles hard tasks behind the scenes.
- Turn detection and adaptive interruption handling are what make a voice agent feel like a conversation instead of a queue.
- Production needs the full apparatus: LLM-as-a-judge evals, in-session and cross-session memory, guardrails, human escalation.
- Tracing, transcripts and recordings are how you debug a conversation after it happened.
- Build on proven real-time infrastructure and spend your engineering effort on domain logic, reliability and customer experience.
About this episode
A strong language model gets you a voice-agent demo. Production is everything else: real-time model selection, turn detection, adaptive interruption handling, tool calling versus RAG, evaluations, memory, guardrails, escalation, and the tracing that lets you debug a conversation after it happened.
Ben Cherry, Head of Product at LiveKit, joins Moe Ali to walk through each layer. The organizing idea is the latency-versus-reasoning trade-off: use a fast model for the live conversation, then delegate harder tasks to a larger model behind the scenes. Memory splits the same way — in-session for the call, cross-session for the relationship — and LLM-as-a-judge evaluations keep quality measurable.
His advice for teams is to rely on proven real-time infrastructure rather than rebuilding it, and to spend limited engineering effort on domain logic, reliability and customer experience — the parts that actually separate a product.
Questions this episode answers
What separates a voice AI demo from a production system?
Everything around the model: turn detection, adaptive interruption handling, evaluations, memory, guardrails, human escalation, and the tracing, transcripts and recordings you need to debug real conversations.
How do voice agents balance latency and reasoning?
Split the work: a fast model runs the live conversation so responses feel immediate, and harder tasks are delegated to a larger model behind the scenes.
Should teams build their own real-time voice infrastructure?
Cherry's argument is no — rely on proven real-time infrastructure, and spend your limited engineering effort on domain logic, reliability and customer experience instead.
