How much VRAM does context length need?

Quick answers

How much VRAM does context length use?

Every token in the context window stores a key and a value per attention layer (the KV cache). For Qwen3-8B that's about 144 KiB per token at FP16, so 32k tokens add ~4.8 GB and 128k add ~19.3 GB on top of the weights. Llama-3.3-70B-Instruct needs ~42.9 GB of cache at 128k.

How much VRAM do I need for 32k context?

It depends on the model. Qwen3-32B needs about 31.4 GB at Q4_K_M with an FP16 cache at 32k (vs 23.8 GB at ~4k); Qwen3.8-27B, which caches only 16 of 64 layers, needs 22.2 GB. See the table above for more models.

Does a quantized KV cache help?

Yes. llama.cpp and Ollama can store the cache at Q8_0 (about 53% of FP16) or Q4_0 (about 28%). A Q8 cache roughly halves the cache part with little quality loss; Q4 cache saves more but can hurt long-context accuracy. The weights' quantization (Q4_K_M etc.) is separate.

Which models use the least memory at long context?

Models with sliding-window or linear-attention layers only keep a growing cache on some layers. Of the 535 models we model, NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 needs the least cache at 128k (~0.8 GB) and OLMo-2-1124-13B-Instruct the most (~107.4 GB). gpt-oss-20b uses a 128-token sliding window on half its layers, so its 128k cache is just ~3.2 GB.

Is a model's maximum context the same as what I can run?

No. The model card states the longest context it was trained for; whether you can use it depends on memory. We don't track each model's maximum, so check the model card before relying on long contexts.

Why are some models missing?

120 models we track use attention layouts our formula doesn't model yet (for example multi-head latent attention, or Gemma 4's separate global layers), or their config is gated. Their pages say "cache not modeled".