More tokens, more cache
Full-attention KV memory grows linearly with context length. Doubling 32K to 64K doubles the cache for the same architecture and precision.
Free · private · formula-first
Context Window VRAM Calculator
Calculate an LLM’s KV cache, estimated total GPU memory, and maximum context length. See exactly how layers, KV heads, precision, and concurrent sequences change the result.
Runs in your browser
Formula shown
No sign-up
The short answer
Context uses VRAM through the KV cache
Model weights are mostly fixed after loading. The KV cache grows as inference stores the key and value vectors for each token in each attention layer.
KV heads matter
GQA and MQA use fewer key/value heads than classic MHA. That can make two similarly sized models have very different memory per token.
Concurrency multiplies it
Each active sequence needs its own cache. Four simultaneous 32K contexts can use roughly the KV memory of one 128K context.
Llama 3.1 8B at 128K context
With 32 layers, 8 KV heads, a 128-dimensional head, one sequence, and FP16 cache, the context alone uses 16 GiB. A Q4 copy of the model plus runtime buffers must still fit beside it.
At 8K, the same FP16 cache is 1 GiB; at 32K it is 4 GiB. Weight quantization does not change those numbers unless KV cache precision is changed separately.
2 × 32 × 8 × 128 × 131,072 × 1 × 2 bytes
= 17,179,869,184 bytes
= 16 GiB KV cache
= 17,179,869,184 bytes
= 16 GiB KV cache
How much VRAM for 8K, 32K, and 128K context?
FP16 KV cache only, one concurrent sequence. Add model weights and runtime reserve for total VRAM. An asterisk marks a value beyond the linked preset’s default config limit, shown only to explain linear cache scaling.
| Model architecture | Per token | 8K context | 32K context | 128K context |
|---|---|---|---|---|
| Llama 3.1 8B · 8 KV heads | 128 KiB | 1.00 GiB | 4.00 GiB | 16.0 GiB |
| Qwen2.5 7B · 4 KV heads | 56 KiB | 0.44 GiB | 1.75 GiB | 7.00 GiB* |
| Mistral 7B v0.3 · 8 KV heads | 128 KiB | 1.00 GiB | 4.00 GiB | 16.0 GiB* |
| Phi-3 Mini 128K · 32 KV heads | 384 KiB | 3.00 GiB | 12.0 GiB | 48.0 GiB |
What this calculator handles
Architecture detail without a spreadsheet
Presets make common cases fast. Advanced inputs keep the result auditable when your model or runtime is different.
MHA, GQA, and MQA
Enter the actual number of KV heads rather than guessing cache size from parameter count.
Hybrid sliding windows
Split full-attention and sliding layers, then cap only the sliding portion at its configured window.
Nonstandard caches
Override bytes per token for MLA, recurrent, compressed, or runtime-measured cache layouts.
Partial GPU offload
Change the percentage of model weights resident on the GPU without pretending CPU offload removes KV memory.
Reverse capacity
After weights and reserve, the remaining budget is converted back into a maximum context length.
Shareable state
Copy a URL containing the current assumptions so a teammate can reproduce the same estimate.
Runtime vocabulary
Translate the result to Ollama and llama.cpp
A calculator is only useful if its inputs map to the knobs you actually run. Verify model support before setting a context above its trained or configured limit.
Ollama
`num_ctx` sets the context length. Availability and defaults can vary with the model and installed Ollama version.
OLLAMA_CONTEXT_LENGTH=32768 ollama serve
llama.cpp server
`--ctx-size` sets context, while K and V cache types can be selected separately on supported backends.
llama-server -c 32768 -ctk q8_0 -ctv q8_0
Primary references: Ollama context length and llama.cpp server options.
Avoid false confidence
Four common VRAM calculation mistakes
Using only parameter count
Parameters approximate weight memory, not KV cache. Layers, KV heads, head dimension, and context determine cache growth.
Assuming Q4 weights mean Q4 cache
Cache precision is independent. Many runtimes default to FP16 KV even when the model weights are heavily quantized.
Spending every last byte
Graphs, CUDA or Metal state, temporary tensors, and other apps need room. Keep an explicit runtime reserve.
Confusing batch with sequences
Prompt-processing batch size affects workspaces and speed. Simultaneous cached sequences multiply persistent KV memory.
FAQ
Context window and VRAM questions
How do you calculate context window VRAM?
For standard attention, multiply 2 (key plus value) by the number of layers, KV heads, head dimension, stored tokens, concurrent sequences, and bytes per cache element. Then add resident weights and a runtime reserve for an estimated total.
How much VRAM does a 128K context window use?
There is no model-independent answer. Llama 3.1 8B uses about 16 GiB for one 128K FP16 KV cache, while a model with fewer KV heads can use far less. Use the architecture preset or enter the model’s config values.
Does context length increase VRAM linearly?
A normal full-attention KV cache grows linearly with stored tokens and sequences. Sliding-window layers stop growing after their window. Hybrid, latent-attention, recurrent, or compressed caches need their implementation-specific layout.
Why do GQA and MQA reduce KV memory?
They share key/value heads across multiple query heads. Because KV cache stores key and value tensors—not query tensors—the smaller KV-head count reduces bytes per token.
Does weight quantization reduce KV cache?
Not by itself. Model weights and KV cache have separate data types. Select the cache precision your runtime actually supports instead of copying the weight format.
What runtime reserve should I use?
Start with 1–2 GiB for a small local model, then validate with the target runtime. Large models, multimodal encoders, high prompt batches, CUDA graphs, and a display attached to the same GPU can need more.
Why might measured VRAM differ from this estimate?
Runtimes use different allocators, padding, block sizes, workspaces, graph capture, metadata, and cache layouts. Quantized formats add scales and packing overhead. Treat the total as a planning estimate and the expanded assumptions as a checklist for measurement.
Are my calculator inputs uploaded?
No. The calculation runs in your browser and no model file is uploaded. Aggregate usage analytics are described in the privacy policy.
Found a model or runtime edge case?
Send the model config, expected result, and runtime measurement. Good reports become test cases.