--- title: Prefill, decode, and the latency budget seo_title: Prefill, decode, and the latency budget — video description: Prefill processes context in parallel; decode generates tokens sequentially. TTFT, TPOT, and throughput measure different phases of the same service. keywords: LLM inference, prefill, decode, TTFT, TPOT, ITL, throughput, latency, KV cache, serving date: '2026-09-12T00:00:00+00:00' robots: index,follow,max-snippet:-1,max-image-preview:large,max-video-preview:-1 hide: - toc - navigation video_watch_page: true ---
All videosOther topics0:36

Prefill, decode, and the latency budget

Prefill processes context in parallel; decode generates tokens sequentially. TTFT, TPOT, and throughput measure different phases of the same service.

Links containing ?t= open the video at a specific second.

Video summary

The ideas to retain

01

The request changes shape after prefill

Consider a request with Lin input tokens and Lout generated tokens.

02

TTFT is a client-visible boundary, not one operation

A production path to first output can be decomposed as:

03

For reasoning models, distinguish TTFT from TTFO

The meaning of “first token” can itself change with model behavior. In current AIPerf, TTFT ends at the first token of any type, including a reasoning token, while TTFO, time to first…

Key moments

Jump directly to a section

  1. Prefill builds the state required for decoding
  2. Decode advances one token at a time
  3. TTFT, TPOT, and throughput form the budget
Next step