01 · Foundation
Why Voice, Why Now
Voice was the obvious interface for AI agents and, for a decade, the worst one — clunky wake words, five-second pauses, robotic replies. What changed in 2025–2026 is that speech-to-speech models made agents conversational: they listen and talk in the same breath, carry tone and hesitation, and answer fast enough that turn-taking feels human.
Sub-second replies became table stakes. Humans leave roughly 200–500 ms of silence between conversational turns; an agent that takes three seconds is not slow, it is broken — callers talk over it, repeat themselves, or hang up. Every architecture decision in this guide is downstream of one constraint: the reply must start before the pause gets awkward.
Working definition
A realtime voice agent is a full-duplex loop: it captures audio continuously, decides when the user has finished a thought, thinks, and streams speech back — while staying interruptible at every moment. The hard part is not any single stage. It is doing all of them concurrently under a latency budget.
What Changed
🗣️
Native speech models
Models that consume and emit audio tokens directly — no transcription hop, prosody preserved end to end.
⚡
Streaming everything
ASR emits partials, LLMs stream tokens, TTS streams audio. Every stage overlaps with the next.
📞
Real deployments
Support lines, drive-through ordering, scheduling, in-car assistants — voice agents took production traffic in 2025.
01 · Foundation
Two Architectures
Every realtime voice system is one of two shapes. The cascaded pipeline chains specialist components — voice activity detection, speech recognition, a text LLM, speech synthesis — each excellent alone, each adding a handoff. The native speech-to-speech model collapses the chain into one model that hears audio and speaks audio.
The Tradeoff
Cascaded wins
Swap ASR / LLM / TTS independently
Mature tool calling + guardrails
Text transcript is source of truth
Cheaper per minute at scale
Speech-to-speech wins
200–300 ms replies, no stage sum
Tone and emotion survive the trip
No ASR errors cascading into the LLM
Interruptions feel native, not bolted on
Choosing in 2026
Latency-critical, emotionally expressive, consumer-facing → native s2s. Heavy tool use, compliance transcripts, tight control over what the agent may say → cascade. Many production stacks run s2s for the conversation and drop to a cascaded text path for regulated moments.
01 · Foundation
The Latency Budget
Treat the reply like a performance budget: every stage spends milliseconds from a fixed allowance, and the sum decides whether the conversation feels alive. The comfort window is roughly 500–800 msfrom the user finishing a sentence to the agent's voice starting — inside it you feel heard, beyond it you start wondering if the call dropped.
Where the Budget Goes
The two big line items are LLM time-to-first-token and endpointing delay — the VAD waiting to be sure you have actually finished speaking. Everything downstream of the first token can overlap: stream tokens into TTS as they arrive and start playback on the first audio chunk, sentence by sentence.
measure time-to-first-audiostream every stageoverlap LLM + TTSp95, not average
The metric that matters
Time-to-first-audio, measured at the caller's ear, at p95. Mean latency hides the 1-in-20 spike that makes a caller say "hello?" — and one "hello?" per call is enough to feel broken.
02 · The Pipeline
Capture & VAD
The pipeline starts at the microphone: getUserMedia for the stream, an AudioWorkletto pull PCM frames off the audio thread, and a socket shipping 20 ms chunks to the server. Send audio continuously, not just when you think someone is talking — barge-in detection needs every frame.
capture.ts
// Capture 16 kHz PCM in an AudioWorklet, ship 20 ms frames
const stream = await navigator.mediaDevices.getUserMedia({
audio: {
echoCancellation: true, // non-negotiable on speakerphone
noiseSuppression: true,
autoGainControl: true,
sampleRate: 16_000,
},
});
const ctx = new AudioContext({ sampleRate: 16_000 });
await ctx.audioWorklet.addModule('/worklets/pcm-frames.js');
const src = ctx.createMediaStreamSource(stream);
const node = new AudioWorkletNode(ctx, 'pcm-frames', {
processorOptions: { frameMs: 20 }, // 320 samples per frame
});
src.connect(node);
const ws = new WebSocket('wss://voice.example.dev/session');
ws.binaryType = 'arraybuffer';
node.port.onmessage = ({ data }: MessageEvent<Int16Array>) => {
// Send raw frames continuously; the server runs VAD and
// endpointing so barge-in detection sees every frame.
if (ws.readyState === WebSocket.OPEN) ws.send(data.buffer);
};
VAD and Endpointing Are Different Jobs
Voice activity detectionanswers "is speech happening right now?" frame by frame — a small neural model like Silero runs in ~1 ms. Endpointinganswers the harder question: "has the user finished their thought?" Pure silence-based endpointing fails constantly — people pause mid-sentence to think, and rushing in at every pause is the fastest way to feel rude.
🎚️
Silence threshold
Classic: end the turn after N ms of silence. Simple, but slow talkers get cut off and fast replies feel laggy.
🧠
Semantic endpointing
A small model reads the partial transcript and judges whether the utterance is complete. Shorter waits on clear sentences.
🔀
Adaptive hybrid
Trailing question intonation → respond fast. Mid-list pause → wait longer. The 2026 default in serious stacks.
Tuning rule
Bias the endpointer toward waiting. Interrupting a user mid-thought is far more damaging than an extra 200 ms of silence — one feels rude, the other merely unhurried.
02 · The Pipeline
Streaming ASR & TTS
Nothing in the pipeline waits for anything else to finish. ASR emits partial transcripts that stabilize word by word; the LLM starts on the final transcript and streams tokens; TTS consumes those tokens sentence by sentence and streams audio. The user hears the first word while the model is still writing the last.
events.ts
// One socket, many event types. Partials arrive every 100-300 ms.
type ServerEvent =
| { type: 'asr.partial'; text: string } // unstable, will change
| { type: 'asr.final'; text: string } // endpoint reached
| { type: 'tts.audio'; chunk: string } // base64 PCM, play now
| { type: 'tts.done' };
ws.onmessage = (msg) => {
const ev: ServerEvent = JSON.parse(msg.data);
switch (ev.type) {
case 'asr.partial':
ui.showCaption(ev.text); // live caption; never act on it
break;
case 'asr.final':
ui.commitCaption(ev.text);
agent.onUserTurn(ev.text); // NOW the LLM turn starts
break;
case 'tts.audio':
player.enqueue(decodePcm(ev.chunk));
metrics.markFirstAudio(); // time-to-first-audio, the KPI
break;
case 'tts.done':
player.markTurnEnd();
break;
}
};
Rules of the Stream
Partials are for display and prediction only — they rewrite themselves as context accumulates, so acting on one means acting on a sentence the user never said. Commit to the LLM only on asr.final. On the output side, chunk LLM text into natural clauses before synthesis: TTS quality collapses when you feed it three words at a time, and latency collapses when you wait for a paragraph.
partials = captions onlycommit on finalsynthesize by clauseplay on first chunk
Speculative execution
Advanced stacks start LLM prefill on a confident partial and throw the work away if the final transcript differs. When the guess holds — most of the time — you erase the entire endpointing wait from the perceived latency.
02 · The Pipeline
Turn-Taking & Interruptions
Humans interrupt each other constantly and it works fine. A voice agent must grant the same right: when the user talks over it — barge-in — the agent has to shut up within a couple hundred milliseconds, and its memory of the conversation has to match what was actually heard, not what it planned to say.
The Two Halves of Barge-In
The first half is fast plumbing: cancel synthesis and flush the playback queue the moment VAD fires. The second half is the one teams forget — context truncation. The model generated a full answer, but the user heard 40% of it. If the transcript keeps the full answer, the model will later refer to things it never actually said.
barge-in.ts
// Barge-in: the user starts talking while the agent is talking
function onVadSpeechStart(session: Session) {
if (!session.agentSpeaking) return; // normal turn, nothing to cancel
ws.send(JSON.stringify({ type: 'tts.cancel' })); // stop synthesis
const playedMs = player.flush(); // drop queued audio
// The history must match what the user actually HEARD, not what
// the model generated. Truncate the assistant turn to played audio.
session.history.truncateAssistantAudio(playedMs);
session.agentSpeaking = false; // floor passes to the user
}
// Without truncation the model believes it said things the user
// never heard, and the conversation quietly diverges from reality.
Not everything is a barge-in
"Mm-hm", "right", a cough — backchannels are listening signals, not interruptions. Killing TTS on every sound makes the agent feel skittish. Gate the cancel on sustained speech (~250 ms) or on the semantic class of the partial transcript.
03 · In Practice
Realtime Stack 2026
The 2026 landscape splits into three layers: native s2s model APIs, orchestration frameworks that run the pipeline and the transport, and component vendors for teams assembling their own cascade. Most production systems mix layers — a framework for plumbing, a model API for the brain.
The Short List
OpenAI Realtime API
Native s2s over WebRTC or WebSocket. Speech in, speech out, tool calls mid-stream. The reference point everything else is measured against.
Gemini Live
Google's bidirectional streaming API — voice plus video input, s2s output, tight tool integration. Strong multimodal angle.
ElevenLabs
The TTS quality bar, now a full conversational platform — agents, voices, and telephony under one roof.
LiveKit Agents
Open-source WebRTC infrastructure plus a Python/Node agent framework. Run any model behind production-grade transport.
Pipecat
Open-source Python pipeline framework from Daily — composable frame processors for VAD, ASR, LLM, TTS. Maximum control over the cascade.
Component vendors
Deepgram and AssemblyAI for streaming ASR, Cartesia and Rime for low-latency TTS — the parts bin for custom cascades.
Selection heuristic
Fastest path to a working agent → a native s2s API. Custom models or on-prem constraints → LiveKit Agents or Pipecat with your own cascade. Voice quality as the product → ElevenLabs. Phone-heavy workloads → whichever framework has first-class SIP.
03 · In Practice
Choosing a Transport
Audio has to move between the caller and your agent, and the wire you pick shapes everything downstream. Three realistic options: WebRTC for browsers and apps on real-world networks, WebSocket for server-to-server plumbing, and SIP/PSTN when the client is a phone number.
WebRTC vs WebSocket
WebRTC wins
UDP: late packets dropped, not queued
Jitter buffers + Opus FEC built in
Echo cancellation in the stack
Survives wifi → LTE handoffs
WebSocket wins
Trivial server-to-server setup
TCP is fine inside a datacenter
Easy to proxy, log, and record
No SDP / ICE / TURN to operate
The classic mistake
Shipping WebSocket audio from end-user devices because it worked on office wifi. On a real mobile network, TCP head-of-line blocking turns one lost packet into a growing audio backlog — the conversation drifts seconds behind. Client-to-server audio over the public internet wants WebRTC.
SIP / PSTN — When the Client Is a Phone
Phone calls remain the largest deployed voice surface: support lines, reservations, outbound reminders. A SIP trunk (Twilio, Telnyx, or a framework's built-in telephony) bridges the PSTN into your pipeline. Budget for its constraints — 8 kHz narrowband audio that degrades ASR, carrier-side latency you cannot optimize away, and DTMF keypresses as a parallel input channel.
browser/app → webrtcserver ↔ server → websocketphone number → sip/pstn
04 · Execution
Failure Modes
Voice agents fail in ways text agents cannot, and the failures are audible — a user forgives a clunky UI far more readily than being interrupted, talked over, or trapped in a loop with a machine hearing its own voice. Four modes account for most production incidents.
Echo loops
The agent's own TTS leaks from speaker to microphone, ASR transcribes it, and the agent starts responding to itself — sometimes indefinitely. Defense in depth: acoustic echo cancellation at capture, half-duplex gating of ASR while the agent speaks, and a similarity check that discards transcripts matching what the agent just said.
Aggressive VAD cutting users off
An endpointer tuned for snappy demos fires on every thinking pause. The user says "I want to change my flight to… um…" and the agent barrels in. Users stop trusting the agent to listen and start talking in defensive, compressed bursts — which degrades ASR further. Tune on recordings of real, hesitant speakers, never on your own crisp test phrases.
Latency spikes mid-conversation
The first three turns land in 600 ms; turn four takes four seconds because the context grew, a tool call blocked, or the LLM provider hiccuped. The caller says "hello? are you there?" — and that utterance lands as new input on top of the late reply, derailing the turn state. Track p95 per turn, cap context growth, and speak a filler whenever the budget is about to blow.
Noise-induced hallucinated transcripts
ASR models are trained to produce fluent text, so silence, music, or a passing truck can transcribe as confident phrases — famously, whole sentences from nothing. The agent then responds to words never spoken. Gate ASR behind VAD so noise never reaches it, respect confidence scores, and confirm before acting on anything consequential heard through noise.
Common thread
Every one of these is invisible in a quiet office on good wifi. Test with speakerphones, background chatter, hesitant speakers, and throttled networks — the conditions of an actual phone call.
04 · Execution
Ship Checklist
The distance between a voice demo and a voice product is measured in edge cases: noisy rooms, slow talkers, flaky networks, long tool calls. Green across the board before a real caller hears your agent.
pre-launch · 0/10 complete
Echo cancellation verified on speakerphone and earbuds
captureVAD endpointing tuned — slow speakers not cut off
vadTime-to-first-audio p95 under 800 ms
latencyBarge-in cancels TTS and truncates context to audio heard
turnsFiller phrases cover any tool call slower than 800 ms
toolsASR tested against noise: car, cafe, street, bad mic
asrFallback path when ASR confidence collapses (text or human)
safetyLoad-tested over real jitter and packet loss, not localhost
transportEvery session records audio + transcript for replay
opsSilence timeout and graceful goodbye implemented
uxWhere to start
If you adopt only three: measure time-to-first-audio at p95, implement real barge-in with context truncation, and record every session. Those three surface almost every other problem on this list.