Tools · Voice · 09

Find where latency is spent before the agent starts speaking.

Break down the interval from the acoustic end of the user's turn to the first audio that reaches the listener. Change the architecture, enter measurements for each stage, and check whether the target is feasible before blaming the model. Barge-in is budgeted separately because stopping audio already in playback is a different critical path.

Responseend of speech → first agent audio

Interruptionspeech start → agent audio stopped

Comparisoncascade, half-cascade and speech-to-speech

Architecture and target

Architecture Presets are editable teaching scenarios, not provider measurements.

Your first-audio target (ms)

Path to first audio

Audio ingress (ms)Last user audio until your processing edge receives it.

Turn-end detection (ms)Residual wait after the acoustic end has arrived.

Residual STT finalization (ms)Use 0 when this critical path has no external STT stage.

Model → first usable output (ms)

TTS → first audio (ms)TTFB/first audible chunk, not total synthesis duration.

Audio egress (ms)

Playback buffer (ms)

Barge-in

New-speech detection (ms)

Control/cancel (ms)

Clear output buffer (ms)

Your interruption target (ms)

First agent audio———

Remaining model budget——

Barge-in until audio stops———

—. Each number is the residual contribution to the critical path after the acoustic end of the turn; if stages overlap in your system, do not count the same wall-clock time twice.

Audio ingress

—

Turn end

—

Residual STT

—

Model first output

—

TTS first audio

—

Audio egress

—

Playback buffer

—

Largest contribution—

Interruption path

—

Method

Choose a clear measurement boundary and add only the critical path.

Measurement boundary. This tool defines response latency as the time from the acoustic end of the user's turn at the capture edge to the first agent audio at the listening edge. If your telemetry uses another boundary, adjust components so every stage uses the same reference.

first_audio = ingress + turn_detection + residual_STT + model_first_output + TTS_first_audio + egress + playback_buffer

Architecture. A full cascade exposes external STT and TTS. In the half-cascade preset, audio understanding lives inside the model while TTS remains external. In speech-to-speech, external STT and TTS are set to zero. These are teaching presets, not a claim that one architecture is always faster.

Turn end. Deepgram documents VAD endpointing with a configurable silence duration. OpenAI Realtime distinguishes server_vad from semantic_vad; semantic turn detection can wait longer when it estimates the user has not finished. Reducing this stage without tracking false cuts can make the interaction worse.

TTS. ElevenLabs separates model inference time from end-to-end latency and recommends streaming/WebSockets to reduce time to first byte/audio. Text buffering can delay the start of synthesis. That is why this tool asks for “TTS first audio”, not total synthesis duration.

Barge-in. Interruption is not response latency in reverse. It includes receiving new speech, detecting it, cancelling generation/playback and clearing audio already queued. In bidirectional Media Streams, Twilio documents clear to empty the audio buffer and mark to track audio that finished or was cleared.

barge_in_stop = ingress + speech_start_detection + cancel/control + output_buffer_clear

These are not benchmarks. Initial values are intentionally round so the scenario is manipulable. Replace them with percentiles from your traces. A mean can hide long tails; production gates should inspect distributions by region, provider, language, turn type and architecture.

Research on human turn-taking shows a broad tendency to minimize silence and overlap, but it does not define a universal SLA for voice agents. The 750/800/900 ms preset targets are editable hypotheses, not scientific recommendations.

Sources: OpenAI Realtime API, Deepgram Endpointing, ElevenLabs Latency Optimization, Twilio Media Streams, and Stivers et al. (PNAS, 2009).

01Three architectures for voice agentsCompare where STT, reasoning and TTS live before assigning latency to each stage.Technical note 02Reactive and proactive voice agentsSeparate conversational events from internal execution and playback control.Technical note 03Latency, streaming and human interactionSee why time to first output can matter more than total completion time.Chapter