KV cache, memory hierarchy, and continuous batching
The KV cache trades memory for less repeated compute; paging and continuous batching coordinate capacity, admission, and reuse across active requests.
Links containing ?t= open the video at a specific second.
Video summary
The ideas to retain
What the KV cache actually stores
In causal attention, once the model has processed a token, that token’s K and V projections do not need to be recomputed from scratch at every later decoding step. The runtime can retain…
A useful memory formula — with explicit limits
For an autoregressive full-attention transformer where each cacheable layer stores K and V in the standard layout, an estimate for one sequence is:
Illustrative arithmetic, not a benchmark
Assume a fictional architecture with:
Key moments
Jump directly to a section
Next step
