Prefill, decode, and the latency budget
Prefill processes context in parallel; decode generates tokens sequentially. TTFT, TPOT, and throughput measure different phases of the same service.
Links containing ?t= open the video at a specific second.
Video summary
The ideas to retain
The request changes shape after prefill
Consider a request with Lin input tokens and Lout generated tokens.
TTFT is a client-visible boundary, not one operation
A production path to first output can be decomposed as:
For reasoning models, distinguish TTFT from TTFO
The meaning of “first token” can itself change with model behavior. In current AIPerf, TTFT ends at the first token of any type, including a reasoning token, while TTFO, time to first…
Key moments
Jump directly to a section
Next step
