State and capacity
For 4096 tokens, KV payload is 512 MiB. Weights and buffers are excluded.
Links containing ?t= open the video at a specific second.
The ideas to retain
What the KV cache actually stores
In causal attention, once the model has processed a token, that token’s K and V projections do not need to be recomputed from scratch at every later decoding step. The runtime can retain…
A useful memory formula — with explicit limits
For an autoregressive full-attention transformer where each cacheable layer stores K and V in the standard layout, an estimate for one sequence is:
Illustrative arithmetic, not a benchmark
Assume a fictional architecture with:
Jump directly to a section
Read the reviewed transcript
This video has no narration. This transcript reproduces its on-screen text; it does not invent a spoken track.
Context occupies memory
Each token stores keys and values per layer and KV head. Precision also matters.
With 32 layers, 8 heads, dimension 128 and two bytes, that is 128 KiB per token.
For 4096 tokens, KV payload is 512 MiB. Weights and buffers are excluded.
Not all GPU memory is KV
Start with sixteen GiB in a simplified example.
Ten GiB hold weights, while two hold buffers and other reservations.
Four remain for KV. Request capacity depends on live state, not just FLOPs.
Separate logical and physical
The sequence needs six positions. Reserving a maximum of sixteen leaves ten unused.
With four-position blocks, a table maps those six positions to two physical blocks.
Two positions remain unused in the final block. Paging reduces waste; it does not eliminate it.
Completion frees a slot
A and B share a decode iteration. A finishes before B.
The scheduler frees A’s KV and admits C while B continues generating.
Continuous batching changes membership at each step. It is not the same as memory paging.
Moving state also costs
A block evicted from the GPU remains in CPU memory.
Reusing it requires crossing the interconnect and restoring residency.
More total capacity can add critical-path movement. Measure transfers and waiting, not just occupancy.
Cancellation releases state
A request reserves three blocks and keeps growing.
Cancellation stops the work and returns its blocks to the free pool.
Accounting must also close on errors. Orphaned blocks reduce capacity for later requests.
