Q4 vs Q8 vs FP16: which quantization should you run?

Quick answers

What is the difference between Q4, Q8 and FP16?

They are how many bits store each weight. FP16 is the full 16-bit release most models ship in. Q8_0 uses about 8.5 bits per weight and Q4_K_M about 4.85 (the extra fraction is scaling data), so a Q4_K_M file is about 30% the size of FP16 and Q8_0 about 53%. Smaller files need less VRAM and run faster, at some cost in quality.

Is Q4 good enough?

For chat and most everyday tasks Q4_K_M is the common default in llama.cpp, Ollama and LM Studio, and the quality drop versus FP16 is usually small for models of 7B and up. Small models (under ~4B) and precise work like coding or math lose more, so use Q8 there if it fits. Our VRAM figures default to Q4_K_M for this reason.

Should I run a bigger model at Q4 or a smaller one at Q8?

The common rule of thumb is that a bigger model at Q4 beats a smaller model at Q8 or FP16 that uses the same memory. For example, Qwen3-32B at Q4 needs ~23.8 GB, about what Qwen3-8B needs at FP16 (~19.8 GB). Below about 4 bits (Q3, Q2), quality falls off faster.

How much VRAM does a 32B model need at Q4 vs Q8?

Qwen3-32B needs ~23.8 GB at Q4_K_M, ~41.0 GB at Q8_0 and ~76.4 GB at FP16, with ~4k context. The cheapest GPU we track that fits it at Q4 is the Intel Arc Pro B70 32GB ($949); at Q8 it's the NVIDIA RTX 6000 Ada 48GB ($6,799).

Does quantization make a model faster?

Yes, for generating tokens. Decode speed is limited by memory bandwidth, and a Q4_K_M file is about 57% the bytes of Q8_0, so it runs roughly 1.8× as fast on the same hardware. Our estimates for Llama-3.3-70B-Instruct reflect that: compare the tok/s columns above.

Do KV cache and context length change this?

Yes. Our figures include ~4k tokens of context. Long contexts add KV-cache memory on top of the weights; see each model's "Memory by context length" table or set the context in the VRAM calculator.