VRAM calculation methodology

Base KV-cache formula

During autoregressive inference, each attention layer stores a key tensor and a value tensor for previous tokens. Hugging Face documents each tensor with shape [batch_size, num_heads, seq_len, head_dim].

The leading 2 represents K and V. “KV heads” is deliberately not always the model’s query-head count: GQA and MQA share fewer KV heads, which is why the config field num_key_value_heads matters.

The calculator displays bytes using binary units: 1 KiB = 1,024 bytes, 1 MiB = 1,024² bytes, and 1 GiB = 1,024³ bytes.

Hybrid and sliding-window layers

A sliding-window layer needs only the most recent window after that window is full. When a model mixes full and sliding layers, the calculator uses:

This is useful for architectures with alternating layer types. It does not model compressed, recurrent, latent-attention, draft-model, indexer, or other specialized state unless you provide a custom measured bytes-per-token value.

Estimated total VRAM

The displayed total has three parts:

1. Resident model weights: parameters × approximate effective bits per weight ÷ 8 × GPU residency.
2. KV cache: the architecture calculation above.
3. Runtime reserve: a user-controlled fixed budget for GPU context, graphs, workspaces, temporary tensors, display use, and other processes.

Weight-format labels such as Q4_K_M use an approximate effective bits-per-weight value, not the nominal integer alone. Actual file and resident sizes vary with tensor types, metadata, alignment, and runtime loading behavior.

Maximum context calculation

The reverse calculation subtracts estimated resident weights and runtime reserve from the GPU budget. It then divides the remaining bytes by the model’s KV cost per token and per sequence.

For hybrid sliding attention, the calculation is piecewise. Before the sliding window fills, all selected layers grow. After it fills, sliding layers are held at the window cost and only full-attention layers continue to grow. The result is capped at the preset’s model context limit when one is supplied.

KV precision assumptions

llama.cpp exposes separate K and V cache-type flags. This calculator uses one shared precision to keep the primary interaction compact; if your K and V types differ, enter a weighted custom bytes-per-token value or calculate the two tensors separately.

Model preset data

Presets use config fields such as num_hidden_layers, num_key_value_heads, hidden_size, num_attention_heads, and max_position_embeddings. Head dimension is taken from head_dim when provided, otherwise hidden_size ÷ num_attention_heads.

Each selected preset links to its source config beside the model selector. Presets are conveniences, not promises that every runtime implements the full advertised context without additional rope scaling or configuration.

Known limitations

• Activation and workspace memory is represented only by the user’s reserve.
• Tensor parallel sharding and pipeline placement are not modeled.
• Prompt batch workspaces and throughput are not calculated.
• Multimodal encoder state, draft-model cache, and cross-attention cache are not included.
• Block allocators may round cache allocations above the continuous formula.
• Unified-memory systems can spill instead of producing a simple out-of-memory boundary.
• A memory fit does not mean the model supports or performs well at that context.

Primary sources

• Hugging Face Transformers — Caching for cache tensor shape, per-layer storage, and dynamic versus sliding-window cache behavior.
• llama.cpp server documentation for context-size, KV offload, cache-type, and serving vocabulary.
• LMCache KV Cache Size Calculator for the paired cache-size and maximum-token planning jobs.

Corrections are welcome at support@contextwindowvramcalculator.com. Include a primary source or reproducible runtime measurement when possible.