Tools · Infrastructure · 03

Calculate what consumes VRAM during inference.

Separate model weights, KV cache and runtime reserve. Change precision, context length, concurrency and GPU count to see which part of the memory budget is constraining the deployment.

Weightsparameters × bits

KV cachelayers × K/V × heads × tokens

Capacityideal GPU sharding

Architecture

Preset Presets use architecture geometry published by Meta. Every field remains editable.

Parameters · billions

Layers

Hidden size

Attention heads

KV headsGQA/MQA reduces this value relative to the total query-head count.

Inference scenario

Weight precision · bitsFor example: 16 for BF16/FP16, or 8/4 as a quantized approximation.

KV-cache precision · bits

Context per sequence · tokens

Concurrent sequencesSequences simultaneously resident in the KV cache.

Hardware and reserve

GPUs

VRAM per GPU · GiB

Weight format metadata · %Add scales, metadata or format overhead when you know it.

Runtime reserve · %Explicit reserve for temporary activations, kernels and allocator headroom. It is not a universal constant.

Copy link Export JSON Reset

Estimated total VRAM—weights + KV cache + reserve

Weights—includes configured format overhead

KV cache—context × concurrent sequences

Memory per GPU——

—

—

—

Where the memory goestotal estimate

WeightsKV cacheRuntime reserve

KV / token / sequence—

Approx. max context—

Approx. max sequences—

Head dimension—

KV versus MHA—

VRAM headroom—

Method

A planning estimate, not a profiler.

Weights. The minimum approximation is parameter count × bits per stored weight. Quantized formats may add scales, codebooks, padding or other metadata, so format overhead is exposed as a separate input rather than hidden.

weight_memory ≈ parameters × weight_bits / 8 × (1 + format_overhead)

KV cache. For standard decoder attention, each layer retains one key and one value vector per KV head, token and sequence. GQA/MQA can use fewer KV heads than query heads. Meta publishes this geometry for Llama 3.1, while vLLM exposes the KV cache as an explicit GPU-memory budget.

KV ≈ layers × 2 × KV_heads × head_dim × KV_bytes × tokens × sequences

Multiple GPUs. The per-GPU figure assumes ideal, even sharding of both weights and KV cache. It is useful as a planning lower bound, but it does not model replicas, pipeline parallelism, indivisible layers, communication buffers, offload or a specific backend layout.

Runtime reserve. The tool applies an editable percentage to weights + KV. Temporary activations, CUDA graphs, kernels and allocator behavior vary by engine and workload, so no fixed percentage is presented as an observed fact.

The KV formula is not a faithful model for MLA, hybrid/sliding-window attention, recurrent state or architecture-specific compressed caches. In those cases, use the result as a comparison estimate and validate the deployment with real serving-engine metrics.

Primary sources: Meta · Llama architecture definitions, Meta · Llama 3.1 model card, and vLLM · KV-cache configuration. Verified 2026-08-21.

Use the memory estimate in context

01How the Transformer worksAttention, heads and representations before turning them into a memory budget.Concept 02LLM cost and latencyConnect deployment memory with workload cost and response time.Tool