Latency budgeting
On the same clock, total waiting is 300 plus 480. It is not the model’s TTFT.
Links containing ?t= open the video at a specific second.
The ideas to retain
A latency metric needs two explicit boundaries
Saying "we have 400 ms of latency" is incomplete. You need to know which event starts those 400 ms and which event ends them.
The clock is part of the definition
Subtracting timestamps from different machines is valid only if you know how their clocks relate. In a distributed system, clock offset or drift can corrupt a small duration.
The critical path matters more than the sum of full stage times
In a classic STT → LLM → TTS pipeline, this expression is tempting:
Jump directly to a section
Read the reviewed transcript
This video has no narration. This transcript reproduces its on-screen text; it does not invent a spoken track.
Two events define latency
In this trace, speech stops at zero and the turn commits after 300 milliseconds.
Playout starts at 780: another 480 milliseconds elapse after the commit.
On the same clock, total waiting is 300 plus 480. It is not the model’s TTFT.
Follow the critical path
STT already runs while the user speaks. Its full duration is not added again afterwards.
The first text starts TTS while the model continues generating the rest.
Here first playout arrives at 700 milliseconds. The full processing bars overlap.
Different clocks. Misleading differences.
The client reads 1000 and the server 1090. Subtraction gives 90, but mixes clocks.
In this example, a known offset of 80 leaves a represented interval of 10.
Without reconciling clocks, that subtraction does not measure latency. Matching IDs is not enough.
Generated does not mean played
A1 is generated at 600 milliseconds. It still needs transport and a playback queue.
It arrives at 650, but playout starts at 700 in this trace.
Name the observed boundary. A server acknowledgement does not prove physical hearing.
The average hides the tail
These twenty synthetic turns include eighteen at 400 milliseconds and two at 1600.
The mean is 520. Nearest-rank P95 is 1600.
Segment by channel and conditions. Slow turns can have a different failure mechanism.
Change one layer. Measure the effect.
The illustrative budget totals 700 milliseconds: 300 for the turn and 400 for the rest.
Reducing model time from 180 to 90 brings the total to 610, not half.
Check quality and premature endings too. Less waiting cannot compensate for a misunderstood request.
