Tools · Infrastructure · 04

See how much context fits in the KV cache.

KV-cache memory grows linearly with resident tokens and concurrent sequences. Change attention geometry, precision and the memory budget to see when context becomes the constraint.

Contexttokens × sequences

Per tokenlayers × K/V × KV heads

GQA/MQAfewer KV than query heads

Architecture

Preset Presets use architecture geometry published by Meta. Every field remains editable.

Layers

Hidden size

Attention heads

KV headsGQA/MQA reduces this value relative to the query-head count.

Resident workload

KV-cache precision · bits

Concurrent sequences

Context per sequence · tokens

KV budget · GiBMemory available only for KV cache after weights and runtime reserve have been accounted for.

8K 32K 64K 128K

Copy link Export CSV Reset

Selected KV cache—context × concurrency

KV / token / sequence—marginal cost of one resident token

Resident tokens—context × sequences

Budget utilization—of the configured KV-only budget

—

—

—

How KV-cache memory scaleslogarithmic X axis

Selected geometry Full MHA KV budget Preset maximum

Memory-limited max context—

Memory-limited max sequences—

KV versus MHA—

Saving versus MHA—

Equivalent full-MHA KV—

Head dimension—

Method

KV-cache memory scales with the tokens that remain resident.

Per token. In standard decoder attention, every layer retains one key and one value per KV head. Head dimension is hidden_size / attention_heads.

KV_token ≈ layers × 2 × KV_heads × head_dim × KV_bytes

Context and concurrency. Total memory then multiplies by the number of tokens in each sequence and by the number of sequences resident at the same time.

KV_total ≈ KV_token × context_tokens × concurrent_sequences

GQA and MQA. Fewer KV heads reduce cache memory in the same proportion, independently of precision or context length. The chart compares the selected geometry with full MHA using the same query-head count.

Budget. The memory limit applies only to KV cache. It is not total VRAM: weights, activations, CUDA graphs, buffers and other reserves must be subtracted first. Use the AI Inference VRAM Calculator to model the full deployment budget.

This approximation does not model MLA, hybrid/sliding-window attention, recurrent state, backend-specific paging/fragmentation or compressed caches. Validate those deployments against real serving-engine metrics.

Primary sources: Meta · Llama architecture definitions, Meta · Llama 3.1 model card, and vLLM · KV-cache configuration. Verified 2026-08-21.

Connect context, memory and architecture

01Total inference VRAMAdd weights and runtime reserve around the KV-cache budget.Tool 02How the Transformer worksAttention, heads and representations that determine cache geometry.Concept