--- title: KV cache, memory hierarchy, and continuous batching seo_title: KV cache, memory hierarchy, and continuous batching — video description: The KV cache trades memory for less repeated compute; paging and continuous batching coordinate capacity, admission, and reuse across active requests. keywords: KV cache, PagedAttention, continuous batching, LLM serving, GPU memory, memory hierarchy, inference date: '2026-09-12T00:00:00+00:00' robots: index,follow,max-snippet:-1,max-image-preview:large,max-video-preview:-1 hide: - toc - navigation video_watch_page: true ---
All videosOther topics0:36

KV cache, memory hierarchy, and continuous batching

The KV cache trades memory for less repeated compute; paging and continuous batching coordinate capacity, admission, and reuse across active requests.

Links containing ?t= open the video at a specific second.

Video summary

The ideas to retain

01

What the KV cache actually stores

In causal attention, once the model has processed a token, that token’s K and V projections do not need to be recomputed from scratch at every later decoding step. The runtime can retain…

02

A useful memory formula — with explicit limits

For an autoregressive full-attention transformer where each cacheable layer stores K and V in the standard layout, an estimate for one sequence is:

03

Illustrative arithmetic, not a benchmark

Assume a fictional architecture with:

Key moments

Jump directly to a section

  1. The KV cache consumes capacity per sequence
  2. Continuous batching rebuilds the batch each step
  3. Paging turns capacity into assignable pages
Next step