AWS Experts Detail Turn-Taking in Voice Agents

Achieving a natural conversational flow in voice agents hinges significantly on effective turn-taking, a challenge that goes beyond the capabilities of the Large Language Model (LLM) itself. Chintan Agrawal and Daniel Wirjo, Solutions Architects on the AWS APJ startup team, delved into the intricacies of this audio engineering problem in a recent presentation. They highlighted that even with a perfect LLM, poor turn-taking can break the user experience, emphasizing the critical role of the audio pipeline.

StartupHub data

Companies working on this

Profiles of the companies named in this story, with funding and a one-liner from our database.

Cartesia

$191M

Ultra-realistic voice AI platform powered by State Space Models for developers.

Deepgram

$1.3B

Deepgram is an enterprise voice AI platform providing APIs for speech-to-text, text-to-speech, and voice agents, used by over 200,000 developers to build voice-first products.

Understanding the 200ms Constraint

Humans typically switch conversational turns within 200 milliseconds, a benchmark that voice agents must strive to meet. Delays exceeding 800 milliseconds begin to feel unnatural, and at 1.5 seconds, users may abandon the interaction. While chat agents might tolerate a 5-second response time, voice agents lack this luxury. Agrawal and Wirjo pointed to a Salesforce result where the best-measured response time was 755 milliseconds, nearly four times slower than human conversation speed.

The Voice Agent Pipeline and Turn-Taking

The standard voice agent pipeline includes Speech-to-Text (STT), LLM, and Text-to-Speech (TTS). However, a crucial, often overlooked component is Voice Activity Detection (VAD), which dictates turn-taking. VAD sits at the front of the pipeline, determining when a user has finished speaking. Alongside VAD, 'smart turn detection' analyzes VAD signals and audio features to decide whether the agent should respond or wait. An 'interruption handler' is also vital, designed to flush the pipeline and cancel LLM generations within milliseconds when a user interrupts.

Three Levels of Solving Turn-Taking

The presentation outlined three distinct approaches to tackle turn-taking:

• Level 1: Silero VAD (Take Control) This is the simplest, fully owned component, often seen in production systems. Silero VAD is a small, 300,000-parameter model that converts audio into spectral features, using convolutional layers and an LSTM for temporal context. Its key tunable parameter is min_silence_ms, which balances cutting off users prematurely against potential dead air. The optimal setting depends heavily on the domain, with sales agents benefiting from lower thresholds (e.g., 200ms) and domains requiring user contemplation needing higher values (e.g., 1000-1200ms).
• Level 2: Built-in Turn Detection This approach leverages the STT service provider's built-in turn detection capabilities. Services like Cartesia (via its STT WebSocket) and Deepgram (via its 'endpointing' feature) process both transcription and turn detection concurrently, using full audio signals and linguistic context. While often smarter and faster (e.g., Deepgram Nova 3's P50 latency of 250ms), the trade-off is a lack of transparency when misfires occur, as the decision-making is internal to the vendor's server.
• Level 3: Silero VAD + Smart Turn This method combines Silero VAD for basic speech detection with a local 'Smart Turn' model that runs during silences. Smart Turn analyzes prosody and intonation to distinguish between a completed thought and a simple pause. While recall might be around 58.9% and precision 68.4% for Smart Turn v3.2, the VAD timer acts as a safety net, ensuring a response even if Smart Turn isn't confident. This level offers full control, transparency, tunability, and portability, with Smart Turn being BSD-2 licensed and readily available.

Addressing Interruptions and Latency

Beyond detecting when a user has finished speaking, voice agents must also handle interruptions effectively. At Level 3, developers can implement logic to differentiate between a genuine interruption requiring the agent to stop and minor interjections like 'yeah' that can be ignored. The presentation also broke down the latency budget in a typical voice-to-voice interaction, highlighting that LLM Time To First Token (TTFT) is often the dominant bottleneck, typically ranging from 500-650ms for cloud APIs. While co-locating models can reduce this to around 500ms, most developers calling APIs experience 800-1300ms total latency. The key takeaway is that optimizing STT and LLM performance is crucial for reducing overall latency.

Production Challenges and Future Improvements

In production, issues like false interruptions can significantly increase escalation rates, leading users to request human support. While current best-measured cascaded pipelines still fall short of human interaction speeds, turn-taking mechanisms play a vital role in improving the perceived experience. The field is rapidly evolving, with expected improvements in models like Smart Turn over the next few years. For infrastructure, options like Picovoice Cloud for managed hosting or AWS guidance reference architectures are available for different deployment needs.