# Q4 vs Q8 vs FP16: LLM quantization and VRAM (2026) > How much VRAM do local LLMs need at Q4_K_M, Q8_0 and FP16? Memory for 10 popular models, the cheapest GPU and Mac for each quant, and est. tok/s. Source: https://whatcanirunai.vercel.app/quantization/ # Q4 vs Q8 vs FP16: which quantization should you run? Quantization shrinks an open model so it fits your GPU or Mac. This page shows how much memory 10 popular models need at Q4\_K\_M, Q8\_0 and FP16, the cheapest current GPU and Mac or mini PC that runs each at each level, and what every device we track can run at each level. Q4\_K\_M 4.85 bits/weight 30% of FP16 Q8\_0 8.5 bits/weight 53% of FP16 FP16 16 bits/weight 100% of FP16 Short answer: run **Q4\_K\_M** unless you have memory to spare. It needs about 30% of FP16's memory and generates tokens roughly 1.8× as fast as Q8\_0 on the same hardware. Step up to **Q8\_0** for small models or coding when it fits; FP16 rarely pays off for local use. ## Memory needed at each quantization Weights plus \~15% overhead and 1 GB, with \~4k context ([our formula](https://whatcanirunai.vercel.app/method/)). Estimates, not file sizes. __Memory needed by popular models at Q4\_K\_M, Q8\_0 and FP16__ | Model | Q4\_K\_M | Q8\_0 | FP16 | | --------------------------------------------------------------------- | -------- | -------- | -------- | | [Qwen3-8B](https://whatcanirunai.vercel.app/models/qwen3-8b/)8.2B params | 6.7 GB | 11.0 GB | 19.8 GB | | [Gemma 4 12B](https://whatcanirunai.vercel.app/models/gemma-4-12b-it/)12.0B params | 9.3 GB | 15.6 GB | 28.5 GB | | [Qwen3-14B](https://whatcanirunai.vercel.app/models/qwen3-14b/)14.8B params | 11.3 GB | 19.0 GB | 35.0 GB | | [gpt-oss-20b](https://whatcanirunai.vercel.app/models/gpt-oss-20b/)20.9B params · MoE | 15.6 GB | 26.6 GB | 49.1 GB | | [Qwen3-30B-A3B](https://whatcanirunai.vercel.app/models/qwen3-30b-a3b/)30.5B params · MoE | 22.3 GB | 38.3 GB | 71.2 GB | | [Qwen3.8-27B](https://whatcanirunai.vercel.app/models/qwen3.8-27b/)27.8B params | 20.4 GB | 34.9 GB | 64.9 GB | | [Qwen3-32B](https://whatcanirunai.vercel.app/models/qwen3-32b/)32.8B params | 23.8 GB | 41.0 GB | 76.4 GB | | [Gemma 4 31B](https://whatcanirunai.vercel.app/models/gemma-4-31b-it/)31.3B params | 22.8 GB | 39.2 GB | 72.9 GB | | [Llama-3.3-70B-Instruct](https://whatcanirunai.vercel.app/models/llama-3.3-70b-instruct/)70.6B params | 50.2 GB | 87.2 GB | 163.3 GB | | [gpt-oss-120b](https://whatcanirunai.vercel.app/models/gpt-oss-120b/)116.8B params · MoE | 82.5 GB | 143.8 GB | 269.7 GB | ## Cheapest hardware at Q4\_K\_M The cheapest current device that fits each model at Q4\_K\_M with \~4k context, at its dated price, with estimated decode speed ([how we estimate](https://whatcanirunai.vercel.app/method/#speed); MoE models show an upper bound). __Cheapest GPU and Mac or mini PC for each model at Q4\_K\_M__ | Model | Cheapest GPU | Cheapest Mac / mini PC | | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | [Qwen3-8B](https://whatcanirunai.vercel.app/models/qwen3-8b/)needs \~6.7 GB | [Intel Arc B570](https://whatcanirunai.vercel.app/hardware/arc-b570/)$219 · 10 GB usable · est. 46–65 tok/s | [Mac mini M6 16GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m6-16/)$899 · 12 GB usable · est. 14–25 tok/s | | [Gemma 4 12B](https://whatcanirunai.vercel.app/models/gemma-4-12b-it/)needs \~9.3 GB | [Intel Arc B580](https://whatcanirunai.vercel.app/hardware/arc-b580/)$249 · 12 GB usable · est. 38–53 tok/s | [Mac mini M6 16GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m6-16/)$899 · 12 GB usable · est. 9–17 tok/s | | [Qwen3-14B](https://whatcanirunai.vercel.app/models/qwen3-14b/)needs \~11.3 GB | [Radeon RX 7600 XT](https://whatcanirunai.vercel.app/hardware/rx-7600-xt/)$329 · 16 GB usable · est. 19–27 tok/s | [Mac mini M6 24GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m6-24/)$1,299 · 18 GB usable · est. 9–15 tok/s | | [gpt-oss-20b](https://whatcanirunai.vercel.app/models/gpt-oss-20b/)needs \~15.6 GB | [Radeon RX 7900 XT](https://whatcanirunai.vercel.app/hardware/rx-7900-xt/)$899 · 20 GB usable · est. up to 312 tok/s | [Mac mini M6 24GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m6-24/)$1,299 · 18 GB usable · est. up to 62 tok/s | | [Qwen3-30B-A3B](https://whatcanirunai.vercel.app/models/qwen3-30b-a3b/)needs \~22.3 GB | [Intel Arc Pro B70 32GB](https://whatcanirunai.vercel.app/hardware/arc-pro-b70/)$949 · 32 GB usable · est. up to 284 tok/s | [Mac mini M5 Pro 64GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m5-pro-64/)$3,199 · 48 GB usable · est. up to 135 tok/s | | [Qwen3.8-27B](https://whatcanirunai.vercel.app/models/qwen3.8-27b/)needs \~20.4 GB | [Intel Arc Pro B70 32GB](https://whatcanirunai.vercel.app/hardware/arc-pro-b70/)$949 · 32 GB usable · est. 22–31 tok/s | [Mac mini M6 32GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m6-32/)$1,799 · 24 GB usable · est. 5–8 tok/s | | [Qwen3-32B](https://whatcanirunai.vercel.app/models/qwen3-32b/)needs \~23.8 GB | [Intel Arc Pro B70 32GB](https://whatcanirunai.vercel.app/hardware/arc-pro-b70/)$949 · 32 GB usable · est. 18–26 tok/s | [Mac mini M5 Pro 64GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m5-pro-64/)$3,199 · 48 GB usable · est. 7–12 tok/s | | [Gemma 4 31B](https://whatcanirunai.vercel.app/models/gemma-4-31b-it/)needs \~22.8 GB | [Intel Arc Pro B70 32GB](https://whatcanirunai.vercel.app/hardware/arc-pro-b70/)$949 · 32 GB usable · est. 19–27 tok/s | [Mac mini M5 Pro 64GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m5-pro-64/)$3,199 · 48 GB usable · est. 7–13 tok/s | | [Llama-3.3-70B-Instruct](https://whatcanirunai.vercel.app/models/llama-3.3-70b-instruct/)needs \~50.2 GB | [NVIDIA RTX Pro 6000 Blackwell 96GB](https://whatcanirunai.vercel.app/hardware/rtx-pro-6000-blackwell/)$8,565 · 96 GB usable · est. 25–36 tok/s | [Framework Desktop (Ryzen AI Max+ 395) 128GB](https://whatcanirunai.vercel.app/hardware/framework-desktop-128/)$3,449 · 96 GB usable · est. 3–5 tok/s | | [gpt-oss-120b](https://whatcanirunai.vercel.app/models/gpt-oss-120b/)needs \~82.5 GB | [NVIDIA RTX Pro 6000 Blackwell 96GB](https://whatcanirunai.vercel.app/hardware/rtx-pro-6000-blackwell/)$8,565 · 96 GB usable · est. up to 493 tok/s | [Framework Desktop (Ryzen AI Max+ 395) 128GB](https://whatcanirunai.vercel.app/hardware/framework-desktop-128/)$3,449 · 96 GB usable · est. up to 66 tok/s | ## Cheapest hardware at Q8\_0 The cheapest current device that fits each model at Q8\_0 with \~4k context, at its dated price, with estimated decode speed ([how we estimate](https://whatcanirunai.vercel.app/method/#speed); MoE models show an upper bound). __Cheapest GPU and Mac or mini PC for each model at Q8\_0__ | Model | Cheapest GPU | Cheapest Mac / mini PC | | ------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | | [Qwen3-8B](https://whatcanirunai.vercel.app/models/qwen3-8b/)needs \~11.0 GB | [Radeon RX 7600 XT](https://whatcanirunai.vercel.app/hardware/rx-7600-xt/)$329 · 16 GB usable · est. 20–28 tok/s | [Mac mini M6 24GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m6-24/)$1,299 · 18 GB usable · est. 9–16 tok/s | | [Gemma 4 12B](https://whatcanirunai.vercel.app/models/gemma-4-12b-it/)needs \~15.6 GB | [Radeon RX 7900 XT](https://whatcanirunai.vercel.app/hardware/rx-7900-xt/)$899 · 20 GB usable · est. 38–54 tok/s | [Mac mini M6 24GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m6-24/)$1,299 · 18 GB usable · est. 6–11 tok/s | | [Qwen3-14B](https://whatcanirunai.vercel.app/models/qwen3-14b/)needs \~19.0 GB | [Intel Arc Pro B70 32GB](https://whatcanirunai.vercel.app/hardware/arc-pro-b70/)$949 · 32 GB usable · est. 23–33 tok/s | [Mac mini M6 32GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m6-32/)$1,799 · 24 GB usable · est. 5–9 tok/s | | [gpt-oss-20b](https://whatcanirunai.vercel.app/models/gpt-oss-20b/)needs \~26.6 GB | [Intel Arc Pro B70 32GB](https://whatcanirunai.vercel.app/hardware/arc-pro-b70/)$949 · 32 GB usable · est. up to 135 tok/s | [Mac mini M5 Pro 64GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m5-pro-64/)$3,199 · 48 GB usable · est. up to 64 tok/s | | [Qwen3-30B-A3B](https://whatcanirunai.vercel.app/models/qwen3-30b-a3b/)needs \~38.3 GB | [NVIDIA RTX 6000 Ada 48GB](https://whatcanirunai.vercel.app/hardware/rtx-6000-ada/)$6,799 · 48 GB usable · est. up to 256 tok/s | [Mac mini M5 Pro 64GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m5-pro-64/)$3,199 · 48 GB usable · est. up to 77 tok/s | | [Qwen3.8-27B](https://whatcanirunai.vercel.app/models/qwen3.8-27b/)needs \~34.9 GB | [NVIDIA RTX 6000 Ada 48GB](https://whatcanirunai.vercel.app/hardware/rtx-6000-ada/)$6,799 · 48 GB usable · est. 20–28 tok/s | [Mac mini M5 Pro 64GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m5-pro-64/)$3,199 · 48 GB usable · est. 5–8 tok/s | | [Qwen3-32B](https://whatcanirunai.vercel.app/models/qwen3-32b/)needs \~41.0 GB | [NVIDIA RTX 6000 Ada 48GB](https://whatcanirunai.vercel.app/hardware/rtx-6000-ada/)$6,799 · 48 GB usable · est. 17–23 tok/s | [Mac mini M5 Pro 64GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m5-pro-64/)$3,199 · 48 GB usable · est. 4–7 tok/s | | [Gemma 4 31B](https://whatcanirunai.vercel.app/models/gemma-4-31b-it/)needs \~39.2 GB | [NVIDIA RTX 6000 Ada 48GB](https://whatcanirunai.vercel.app/hardware/rtx-6000-ada/)$6,799 · 48 GB usable · est. 17–25 tok/s | [Mac mini M5 Pro 64GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m5-pro-64/)$3,199 · 48 GB usable · est. 4–7 tok/s | | [Llama-3.3-70B-Instruct](https://whatcanirunai.vercel.app/models/llama-3.3-70b-instruct/)needs \~87.2 GB | none we track | [Mac Studio M5 Ultra 256GB](https://whatcanirunai.vercel.app/hardware/mac-studio-m5-ultra-256/)$9,499 · 192 GB usable · est. 7–13 tok/s | | [gpt-oss-120b](https://whatcanirunai.vercel.app/models/gpt-oss-120b/)needs \~143.8 GB | none we track | [Mac Studio M5 Ultra 256GB](https://whatcanirunai.vercel.app/hardware/mac-studio-m5-ultra-256/)$9,499 · 192 GB usable · est. up to 177 tok/s | ## Cheapest hardware at FP16 The cheapest current device that fits each model at FP16 with \~4k context, at its dated price, with estimated decode speed ([how we estimate](https://whatcanirunai.vercel.app/method/#speed); MoE models show an upper bound). __Cheapest GPU and Mac or mini PC for each model at FP16__ | Model | Cheapest GPU | Cheapest Mac / mini PC | | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | [Qwen3-8B](https://whatcanirunai.vercel.app/models/qwen3-8b/)needs \~19.8 GB | [Intel Arc Pro B70 32GB](https://whatcanirunai.vercel.app/hardware/arc-pro-b70/)$949 · 32 GB usable · est. 22–32 tok/s | [Mac mini M6 32GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m6-32/)$1,799 · 24 GB usable · est. 5–8 tok/s | | [Gemma 4 12B](https://whatcanirunai.vercel.app/models/gemma-4-12b-it/)needs \~28.5 GB | [Intel Arc Pro B70 32GB](https://whatcanirunai.vercel.app/hardware/arc-pro-b70/)$949 · 32 GB usable · est. 15–22 tok/s | [Mac mini M5 Pro 64GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m5-pro-64/)$3,199 · 48 GB usable · est. 6–10 tok/s | | [Qwen3-14B](https://whatcanirunai.vercel.app/models/qwen3-14b/)needs \~35.0 GB | [NVIDIA RTX 6000 Ada 48GB](https://whatcanirunai.vercel.app/hardware/rtx-6000-ada/)$6,799 · 48 GB usable · est. 20–28 tok/s | [Mac mini M5 Pro 64GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m5-pro-64/)$3,199 · 48 GB usable · est. 5–8 tok/s | | [gpt-oss-20b](https://whatcanirunai.vercel.app/models/gpt-oss-20b/)needs \~49.1 GB | [NVIDIA RTX Pro 6000 Blackwell 96GB](https://whatcanirunai.vercel.app/hardware/rtx-pro-6000-blackwell/)$8,565 · 96 GB usable · est. up to 212 tok/s | [Framework Desktop (Ryzen AI Max+ 395) 128GB](https://whatcanirunai.vercel.app/hardware/framework-desktop-128/)$3,449 · 96 GB usable · est. up to 28 tok/s | | [Qwen3-30B-A3B](https://whatcanirunai.vercel.app/models/qwen3-30b-a3b/)needs \~71.2 GB | [NVIDIA RTX Pro 6000 Blackwell 96GB](https://whatcanirunai.vercel.app/hardware/rtx-pro-6000-blackwell/)$8,565 · 96 GB usable · est. up to 254 tok/s | [Framework Desktop (Ryzen AI Max+ 395) 128GB](https://whatcanirunai.vercel.app/hardware/framework-desktop-128/)$3,449 · 96 GB usable · est. up to 34 tok/s | | [Qwen3.8-27B](https://whatcanirunai.vercel.app/models/qwen3.8-27b/)needs \~64.9 GB | [NVIDIA RTX Pro 6000 Blackwell 96GB](https://whatcanirunai.vercel.app/hardware/rtx-pro-6000-blackwell/)$8,565 · 96 GB usable · est. 19–27 tok/s | [Framework Desktop (Ryzen AI Max+ 395) 128GB](https://whatcanirunai.vercel.app/hardware/framework-desktop-128/)$3,449 · 96 GB usable · est. 2–4 tok/s | | [Qwen3-32B](https://whatcanirunai.vercel.app/models/qwen3-32b/)needs \~76.4 GB | [NVIDIA RTX Pro 6000 Blackwell 96GB](https://whatcanirunai.vercel.app/hardware/rtx-pro-6000-blackwell/)$8,565 · 96 GB usable · est. 16–23 tok/s | [Framework Desktop (Ryzen AI Max+ 395) 128GB](https://whatcanirunai.vercel.app/hardware/framework-desktop-128/)$3,449 · 96 GB usable · est. 2–3 tok/s | | [Gemma 4 31B](https://whatcanirunai.vercel.app/models/gemma-4-31b-it/)needs \~72.9 GB | [NVIDIA RTX Pro 6000 Blackwell 96GB](https://whatcanirunai.vercel.app/hardware/rtx-pro-6000-blackwell/)$8,565 · 96 GB usable · est. 17–24 tok/s | [Framework Desktop (Ryzen AI Max+ 395) 128GB](https://whatcanirunai.vercel.app/hardware/framework-desktop-128/)$3,449 · 96 GB usable · est. 2–3 tok/s | | [Llama-3.3-70B-Instruct](https://whatcanirunai.vercel.app/models/llama-3.3-70b-instruct/)needs \~163.3 GB | none we track | [Mac Studio M5 Ultra 256GB](https://whatcanirunai.vercel.app/hardware/mac-studio-m5-ultra-256/)$9,499 · 192 GB usable · est. 4–7 tok/s | | [gpt-oss-120b](https://whatcanirunai.vercel.app/models/gpt-oss-120b/)needs \~269.7 GB | none we track | [Mac Studio M5 Ultra 512GB](https://whatcanirunai.vercel.app/hardware/mac-studio-m5-ultra-512/)$22,299 · 384 GB usable · est. up to 94 tok/s | ## Biggest model each device runs, by quantization Every current device we track with 8 GB or more usable memory, smallest first, with the biggest model that fits with \~4k context and how many of our 656 models fit. __Biggest model that fits each device at Q4\_K\_M, Q8\_0 and FP16__ | Device | Q4\_K\_M | Q8\_0 | FP16 | | --------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | [GeForce RTX 3050 8GB](https://whatcanirunai.vercel.app/hardware/rtx-3050-8gb/)GPU · 8 GB usable · $249 | [NVIDIA-Nemotron-Nano-9B-v2](https://whatcanirunai.vercel.app/models/nvidia-nemotron-nano-9b-v2/)315 models fit | [Molmo2-ER](https://whatcanirunai.vercel.app/models/molmo2-er/)192 models fit | [Zamba2-2.7B](https://whatcanirunai.vercel.app/models/zamba2-2.7b/)120 models fit | | [Radeon RX 7600](https://whatcanirunai.vercel.app/hardware/rx-7600/)GPU · 8 GB usable · $269 | [NVIDIA-Nemotron-Nano-9B-v2](https://whatcanirunai.vercel.app/models/nvidia-nemotron-nano-9b-v2/)315 models fit | [Molmo2-ER](https://whatcanirunai.vercel.app/models/molmo2-er/)192 models fit | [Zamba2-2.7B](https://whatcanirunai.vercel.app/models/zamba2-2.7b/)120 models fit | | [GeForce RTX 4060](https://whatcanirunai.vercel.app/hardware/rtx-4060/)GPU · 8 GB usable · $299 | [NVIDIA-Nemotron-Nano-9B-v2](https://whatcanirunai.vercel.app/models/nvidia-nemotron-nano-9b-v2/)315 models fit | [Molmo2-ER](https://whatcanirunai.vercel.app/models/molmo2-er/)192 models fit | [Zamba2-2.7B](https://whatcanirunai.vercel.app/models/zamba2-2.7b/)120 models fit | | [GeForce RTX 5060](https://whatcanirunai.vercel.app/hardware/rtx-5060/)GPU · 8 GB usable · $299 | [NVIDIA-Nemotron-Nano-9B-v2](https://whatcanirunai.vercel.app/models/nvidia-nemotron-nano-9b-v2/)315 models fit | [Molmo2-ER](https://whatcanirunai.vercel.app/models/molmo2-er/)192 models fit | [Zamba2-2.7B](https://whatcanirunai.vercel.app/models/zamba2-2.7b/)120 models fit | | [Radeon RX 9060 XT 8GB](https://whatcanirunai.vercel.app/hardware/rx-9060-xt-8gb/)GPU · 8 GB usable · $299 | [NVIDIA-Nemotron-Nano-9B-v2](https://whatcanirunai.vercel.app/models/nvidia-nemotron-nano-9b-v2/)315 models fit | [Molmo2-ER](https://whatcanirunai.vercel.app/models/molmo2-er/)192 models fit | [Zamba2-2.7B](https://whatcanirunai.vercel.app/models/zamba2-2.7b/)120 models fit | | [GeForce RTX 5060 Ti 8GB](https://whatcanirunai.vercel.app/hardware/rtx-5060-ti-8gb/)GPU · 8 GB usable · $379 | [NVIDIA-Nemotron-Nano-9B-v2](https://whatcanirunai.vercel.app/models/nvidia-nemotron-nano-9b-v2/)315 models fit | [Molmo2-ER](https://whatcanirunai.vercel.app/models/molmo2-er/)192 models fit | [Zamba2-2.7B](https://whatcanirunai.vercel.app/models/zamba2-2.7b/)120 models fit | | [GeForce RTX 3060 Ti](https://whatcanirunai.vercel.app/hardware/rtx-3060-ti/)GPU · 8 GB usable · $399 | [NVIDIA-Nemotron-Nano-9B-v2](https://whatcanirunai.vercel.app/models/nvidia-nemotron-nano-9b-v2/)315 models fit | [Molmo2-ER](https://whatcanirunai.vercel.app/models/molmo2-er/)192 models fit | [Zamba2-2.7B](https://whatcanirunai.vercel.app/models/zamba2-2.7b/)120 models fit | | [GeForce RTX 3070](https://whatcanirunai.vercel.app/hardware/rtx-3070/)GPU · 8 GB usable · $499 | [NVIDIA-Nemotron-Nano-9B-v2](https://whatcanirunai.vercel.app/models/nvidia-nemotron-nano-9b-v2/)315 models fit | [Molmo2-ER](https://whatcanirunai.vercel.app/models/molmo2-er/)192 models fit | [Zamba2-2.7B](https://whatcanirunai.vercel.app/models/zamba2-2.7b/)120 models fit | | [Intel Arc B570](https://whatcanirunai.vercel.app/hardware/arc-b570/)GPU · 10 GB usable · $219 | [HyperCLOVAX-SEED-Omni-8B](https://whatcanirunai.vercel.app/models/hyperclovax-seed-omni-8b/)340 models fit | [YuE-s1-7B-anneal-en-cot](https://whatcanirunai.vercel.app/models/yue-s1-7b-anneal-en-cot/)198 models fit | [granite-4.1-3b](https://whatcanirunai.vercel.app/models/granite-4.1-3b/)144 models fit | | [GeForce RTX 3080 10GB](https://whatcanirunai.vercel.app/hardware/rtx-3080-10gb/)GPU · 10 GB usable · $699 | [HyperCLOVAX-SEED-Omni-8B](https://whatcanirunai.vercel.app/models/hyperclovax-seed-omni-8b/)340 models fit | [YuE-s1-7B-anneal-en-cot](https://whatcanirunai.vercel.app/models/yue-s1-7b-anneal-en-cot/)198 models fit | [granite-4.1-3b](https://whatcanirunai.vercel.app/models/granite-4.1-3b/)144 models fit | | [Intel Arc B580](https://whatcanirunai.vercel.app/hardware/arc-b580/)GPU · 12 GB usable · $249 | [OLMo-2-1124-13B-Instruct](https://whatcanirunai.vercel.app/models/olmo-2-1124-13b-instruct/)350 models fit | [Nemotron 3.5 Bolt E4B](https://whatcanirunai.vercel.app/models/nemotron-3.5-bolt-e4b/)253 models fit | [Nanbeige4.2-3B](https://whatcanirunai.vercel.app/models/nanbeige4.2-3b/)176 models fit | | [GeForce RTX 3060 12GB](https://whatcanirunai.vercel.app/hardware/rtx-3060-12gb/)GPU · 12 GB usable · $329 | [OLMo-2-1124-13B-Instruct](https://whatcanirunai.vercel.app/models/olmo-2-1124-13b-instruct/)350 models fit | [Nemotron 3.5 Bolt E4B](https://whatcanirunai.vercel.app/models/nemotron-3.5-bolt-e4b/)253 models fit | [Nanbeige4.2-3B](https://whatcanirunai.vercel.app/models/nanbeige4.2-3b/)176 models fit | | [Radeon RX 7700 XT](https://whatcanirunai.vercel.app/hardware/rx-7700-xt/)GPU · 12 GB usable · $449 | [OLMo-2-1124-13B-Instruct](https://whatcanirunai.vercel.app/models/olmo-2-1124-13b-instruct/)350 models fit | [Nemotron 3.5 Bolt E4B](https://whatcanirunai.vercel.app/models/nemotron-3.5-bolt-e4b/)253 models fit | [Nanbeige4.2-3B](https://whatcanirunai.vercel.app/models/nanbeige4.2-3b/)176 models fit | | [GeForce RTX 5070](https://whatcanirunai.vercel.app/hardware/rtx-5070/)GPU · 12 GB usable · $549 | [OLMo-2-1124-13B-Instruct](https://whatcanirunai.vercel.app/models/olmo-2-1124-13b-instruct/)350 models fit | [Nemotron 3.5 Bolt E4B](https://whatcanirunai.vercel.app/models/nemotron-3.5-bolt-e4b/)253 models fit | [Nanbeige4.2-3B](https://whatcanirunai.vercel.app/models/nanbeige4.2-3b/)176 models fit | | [GeForce RTX 4070](https://whatcanirunai.vercel.app/hardware/rtx-4070/)GPU · 12 GB usable · $599 | [OLMo-2-1124-13B-Instruct](https://whatcanirunai.vercel.app/models/olmo-2-1124-13b-instruct/)350 models fit | [Nemotron 3.5 Bolt E4B](https://whatcanirunai.vercel.app/models/nemotron-3.5-bolt-e4b/)253 models fit | [Nanbeige4.2-3B](https://whatcanirunai.vercel.app/models/nanbeige4.2-3b/)176 models fit | | [GeForce RTX 4070 Super](https://whatcanirunai.vercel.app/hardware/rtx-4070-super/)GPU · 12 GB usable · $599 | [OLMo-2-1124-13B-Instruct](https://whatcanirunai.vercel.app/models/olmo-2-1124-13b-instruct/)350 models fit | [Nemotron 3.5 Bolt E4B](https://whatcanirunai.vercel.app/models/nemotron-3.5-bolt-e4b/)253 models fit | [Nanbeige4.2-3B](https://whatcanirunai.vercel.app/models/nanbeige4.2-3b/)176 models fit | | [Mac mini M6 16GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m6-16/)unified memory · 12 GB usable · $899 | [OLMo-2-1124-13B-Instruct](https://whatcanirunai.vercel.app/models/olmo-2-1124-13b-instruct/)350 models fit | [Nemotron 3.5 Bolt E4B](https://whatcanirunai.vercel.app/models/nemotron-3.5-bolt-e4b/)253 models fit | [Nanbeige4.2-3B](https://whatcanirunai.vercel.app/models/nanbeige4.2-3b/)176 models fit | | [GeForce RTX 3080 Ti](https://whatcanirunai.vercel.app/hardware/rtx-3080-ti/)GPU · 12 GB usable · $1,199 | [OLMo-2-1124-13B-Instruct](https://whatcanirunai.vercel.app/models/olmo-2-1124-13b-instruct/)350 models fit | [Nemotron 3.5 Bolt E4B](https://whatcanirunai.vercel.app/models/nemotron-3.5-bolt-e4b/)253 models fit | [Nanbeige4.2-3B](https://whatcanirunai.vercel.app/models/nanbeige4.2-3b/)176 models fit | | [Radeon RX 7600 XT](https://whatcanirunai.vercel.app/hardware/rx-7600-xt/)GPU · 16 GB usable · $329 | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | [HyperCLOVAX-SEED-Omni-8B](https://whatcanirunai.vercel.app/models/hyperclovax-seed-omni-8b/)340 models fit | [gemma-3n-E2B-it](https://whatcanirunai.vercel.app/models/gemma-3n-e2b-it/)195 models fit | | [Radeon RX 9060 XT 16GB](https://whatcanirunai.vercel.app/hardware/rx-9060-xt-16gb/)GPU · 16 GB usable · $349 | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | [HyperCLOVAX-SEED-Omni-8B](https://whatcanirunai.vercel.app/models/hyperclovax-seed-omni-8b/)340 models fit | [gemma-3n-E2B-it](https://whatcanirunai.vercel.app/models/gemma-3n-e2b-it/)195 models fit | | [GeForce RTX 5060 Ti 16GB](https://whatcanirunai.vercel.app/hardware/rtx-5060-ti-16gb/)GPU · 16 GB usable · $429 | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | [HyperCLOVAX-SEED-Omni-8B](https://whatcanirunai.vercel.app/models/hyperclovax-seed-omni-8b/)340 models fit | [gemma-3n-E2B-it](https://whatcanirunai.vercel.app/models/gemma-3n-e2b-it/)195 models fit | | [GeForce RTX 4060 Ti 16GB](https://whatcanirunai.vercel.app/hardware/rtx-4060-ti-16gb/)GPU · 16 GB usable · $499 | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | [HyperCLOVAX-SEED-Omni-8B](https://whatcanirunai.vercel.app/models/hyperclovax-seed-omni-8b/)340 models fit | [gemma-3n-E2B-it](https://whatcanirunai.vercel.app/models/gemma-3n-e2b-it/)195 models fit | | [Radeon RX 7800 XT](https://whatcanirunai.vercel.app/hardware/rx-7800-xt/)GPU · 16 GB usable · $499 | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | [HyperCLOVAX-SEED-Omni-8B](https://whatcanirunai.vercel.app/models/hyperclovax-seed-omni-8b/)340 models fit | [gemma-3n-E2B-it](https://whatcanirunai.vercel.app/models/gemma-3n-e2b-it/)195 models fit | | [Radeon RX 7900 GRE](https://whatcanirunai.vercel.app/hardware/rx-7900-gre/)GPU · 16 GB usable · $549 | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | [HyperCLOVAX-SEED-Omni-8B](https://whatcanirunai.vercel.app/models/hyperclovax-seed-omni-8b/)340 models fit | [gemma-3n-E2B-it](https://whatcanirunai.vercel.app/models/gemma-3n-e2b-it/)195 models fit | | [Radeon RX 9070](https://whatcanirunai.vercel.app/hardware/rx-9070/)GPU · 16 GB usable · $549 | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | [HyperCLOVAX-SEED-Omni-8B](https://whatcanirunai.vercel.app/models/hyperclovax-seed-omni-8b/)340 models fit | [gemma-3n-E2B-it](https://whatcanirunai.vercel.app/models/gemma-3n-e2b-it/)195 models fit | | [Radeon RX 9070 XT](https://whatcanirunai.vercel.app/hardware/rx-9070-xt/)GPU · 16 GB usable · $599 | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | [HyperCLOVAX-SEED-Omni-8B](https://whatcanirunai.vercel.app/models/hyperclovax-seed-omni-8b/)340 models fit | [gemma-3n-E2B-it](https://whatcanirunai.vercel.app/models/gemma-3n-e2b-it/)195 models fit | | [NVIDIA RTX 2000 Ada 16GB](https://whatcanirunai.vercel.app/hardware/rtx-2000-ada/)GPU · 16 GB usable · $649 | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | [HyperCLOVAX-SEED-Omni-8B](https://whatcanirunai.vercel.app/models/hyperclovax-seed-omni-8b/)340 models fit | [gemma-3n-E2B-it](https://whatcanirunai.vercel.app/models/gemma-3n-e2b-it/)195 models fit | | [GeForce RTX 5070 Ti](https://whatcanirunai.vercel.app/hardware/rtx-5070-ti/)GPU · 16 GB usable · $749 | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | [HyperCLOVAX-SEED-Omni-8B](https://whatcanirunai.vercel.app/models/hyperclovax-seed-omni-8b/)340 models fit | [gemma-3n-E2B-it](https://whatcanirunai.vercel.app/models/gemma-3n-e2b-it/)195 models fit | | [GeForce RTX 4070 Ti Super](https://whatcanirunai.vercel.app/hardware/rtx-4070-ti-super/)GPU · 16 GB usable · $799 | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | [HyperCLOVAX-SEED-Omni-8B](https://whatcanirunai.vercel.app/models/hyperclovax-seed-omni-8b/)340 models fit | [gemma-3n-E2B-it](https://whatcanirunai.vercel.app/models/gemma-3n-e2b-it/)195 models fit | | [GeForce RTX 4080 Super](https://whatcanirunai.vercel.app/hardware/rtx-4080-super/)GPU · 16 GB usable · $999 | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | [HyperCLOVAX-SEED-Omni-8B](https://whatcanirunai.vercel.app/models/hyperclovax-seed-omni-8b/)340 models fit | [gemma-3n-E2B-it](https://whatcanirunai.vercel.app/models/gemma-3n-e2b-it/)195 models fit | | [GeForce RTX 5080](https://whatcanirunai.vercel.app/hardware/rtx-5080/)GPU · 16 GB usable · $999 | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | [HyperCLOVAX-SEED-Omni-8B](https://whatcanirunai.vercel.app/models/hyperclovax-seed-omni-8b/)340 models fit | [gemma-3n-E2B-it](https://whatcanirunai.vercel.app/models/gemma-3n-e2b-it/)195 models fit | | [Mac mini M6 24GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m6-24/)unified memory · 18 GB usable · $1,299 | [gpt-oss-20b](https://whatcanirunai.vercel.app/models/gpt-oss-20b/)393 models fit | [Mistral-Nemo-Instruct-2407](https://whatcanirunai.vercel.app/models/mistral-nemo-instruct-2407/)349 models fit | [YuE-s1-7B-anneal-en-cot](https://whatcanirunai.vercel.app/models/yue-s1-7b-anneal-en-cot/)198 models fit | | [Radeon RX 7900 XT](https://whatcanirunai.vercel.app/hardware/rx-7900-xt/)GPU · 20 GB usable · $899 | [Voxtral-Small-24B-2507](https://whatcanirunai.vercel.app/models/voxtral-small-24b-2507/)404 models fit | [OLMo-2-1124-13B-Instruct](https://whatcanirunai.vercel.app/models/olmo-2-1124-13b-instruct/)350 models fit | [LLaDA-MoE-7B-A1B-Base](https://whatcanirunai.vercel.app/models/llada-moe-7b-a1b-base/)218 models fit | | [NVIDIA RTX 4000 Ada 20GB](https://whatcanirunai.vercel.app/hardware/rtx-4000-ada/)GPU · 20 GB usable · $1,250 | [Voxtral-Small-24B-2507](https://whatcanirunai.vercel.app/models/voxtral-small-24b-2507/)404 models fit | [OLMo-2-1124-13B-Instruct](https://whatcanirunai.vercel.app/models/olmo-2-1124-13b-instruct/)350 models fit | [LLaDA-MoE-7B-A1B-Base](https://whatcanirunai.vercel.app/models/llada-moe-7b-a1b-base/)218 models fit | | [Radeon RX 7900 XTX](https://whatcanirunai.vercel.app/hardware/rx-7900-xtx/)GPU · 24 GB usable · $999 | [ERNIE-4.5-VL-28B-A3B-PT](https://whatcanirunai.vercel.app/models/ernie-4.5-vl-28b-a3b-pt/)423 models fit | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | [NVIDIA-Nemotron-Nano-9B-v2](https://whatcanirunai.vercel.app/models/nvidia-nemotron-nano-9b-v2/)315 models fit | | [GeForce RTX 3090](https://whatcanirunai.vercel.app/hardware/rtx-3090/)GPU · 24 GB usable · $1,499 | [ERNIE-4.5-VL-28B-A3B-PT](https://whatcanirunai.vercel.app/models/ernie-4.5-vl-28b-a3b-pt/)423 models fit | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | [NVIDIA-Nemotron-Nano-9B-v2](https://whatcanirunai.vercel.app/models/nvidia-nemotron-nano-9b-v2/)315 models fit | | [GeForce RTX 4090](https://whatcanirunai.vercel.app/hardware/rtx-4090/)GPU · 24 GB usable · $1,599 | [ERNIE-4.5-VL-28B-A3B-PT](https://whatcanirunai.vercel.app/models/ernie-4.5-vl-28b-a3b-pt/)423 models fit | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | [NVIDIA-Nemotron-Nano-9B-v2](https://whatcanirunai.vercel.app/models/nvidia-nemotron-nano-9b-v2/)315 models fit | | [Mac mini M6 32GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m6-32/)unified memory · 24 GB usable · $1,799 | [ERNIE-4.5-VL-28B-A3B-PT](https://whatcanirunai.vercel.app/models/ernie-4.5-vl-28b-a3b-pt/)423 models fit | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | [NVIDIA-Nemotron-Nano-9B-v2](https://whatcanirunai.vercel.app/models/nvidia-nemotron-nano-9b-v2/)315 models fit | | [GeForce RTX 3090 Ti](https://whatcanirunai.vercel.app/hardware/rtx-3090-ti/)GPU · 24 GB usable · $1,999 | [ERNIE-4.5-VL-28B-A3B-PT](https://whatcanirunai.vercel.app/models/ernie-4.5-vl-28b-a3b-pt/)423 models fit | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | [NVIDIA-Nemotron-Nano-9B-v2](https://whatcanirunai.vercel.app/models/nvidia-nemotron-nano-9b-v2/)315 models fit | | [MacBook Pro M6 32GB](https://whatcanirunai.vercel.app/hardware/macbook-pro-m6-32/)unified memory · 24 GB usable · $2,399 | [ERNIE-4.5-VL-28B-A3B-PT](https://whatcanirunai.vercel.app/models/ernie-4.5-vl-28b-a3b-pt/)423 models fit | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | [NVIDIA-Nemotron-Nano-9B-v2](https://whatcanirunai.vercel.app/models/nvidia-nemotron-nano-9b-v2/)315 models fit | | [Intel Arc Pro B70 32GB](https://whatcanirunai.vercel.app/hardware/arc-pro-b70/)GPU · 32 GB usable · $949 | [IQuest-Coder-V1-40B-Loop-Instruct](https://whatcanirunai.vercel.app/models/iquest-coder-v1-40b-loop-instruct/)504 models fit | [EuroLLM-22B-Instruct-2512](https://whatcanirunai.vercel.app/models/eurollm-22b-instruct-2512/)398 models fit | [Gemma 4 12B](https://whatcanirunai.vercel.app/models/gemma-4-12b-it/)341 models fit | | [GeForce RTX 5090](https://whatcanirunai.vercel.app/hardware/rtx-5090/)GPU · 32 GB usable · $1,999 | [IQuest-Coder-V1-40B-Loop-Instruct](https://whatcanirunai.vercel.app/models/iquest-coder-v1-40b-loop-instruct/)504 models fit | [EuroLLM-22B-Instruct-2512](https://whatcanirunai.vercel.app/models/eurollm-22b-instruct-2512/)398 models fit | [Gemma 4 12B](https://whatcanirunai.vercel.app/models/gemma-4-12b-it/)341 models fit | | [Mac mini M5 Pro 64GB](https://whatcanirunai.vercel.app/hardware/mac-mini-m5-pro-64/)unified memory · 48 GB usable · $3,199 | [AI21-Jamba2-Mini](https://whatcanirunai.vercel.app/models/ai21-jamba2-mini/)511 models fit | [EXAONE-4.5-33B](https://whatcanirunai.vercel.app/models/exaone-4.5-33b/)482 models fit | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | | [NVIDIA DGX Spark 64GB](https://whatcanirunai.vercel.app/hardware/dgx-spark-64/)unified memory · 48 GB usable · $4,999 | [AI21-Jamba2-Mini](https://whatcanirunai.vercel.app/models/ai21-jamba2-mini/)511 models fit | [EXAONE-4.5-33B](https://whatcanirunai.vercel.app/models/exaone-4.5-33b/)482 models fit | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | | [NVIDIA RTX 6000 Ada 48GB](https://whatcanirunai.vercel.app/hardware/rtx-6000-ada/)GPU · 48 GB usable · $6,799 | [AI21-Jamba2-Mini](https://whatcanirunai.vercel.app/models/ai21-jamba2-mini/)511 models fit | [EXAONE-4.5-33B](https://whatcanirunai.vercel.app/models/exaone-4.5-33b/)482 models fit | [Ling-Coder-lite](https://whatcanirunai.vercel.app/models/ling-coder-lite/)388 models fit | | [Framework Desktop (Ryzen AI Max+ 395) 128GB](https://whatcanirunai.vercel.app/hardware/framework-desktop-128/)unified memory · 96 GB usable · $3,449 | [Mistral-Small-4-119B-2603](https://whatcanirunai.vercel.app/models/mistral-small-4-119b-2603/)545 models fit | [LongCat-Flash-Lite](https://whatcanirunai.vercel.app/models/longcat-flash-lite/)512 models fit | [Seed-OSS-36B-Instruct](https://whatcanirunai.vercel.app/models/seed-oss-36b-instruct/)500 models fit | | [MacBook Pro M5 Max 128GB](https://whatcanirunai.vercel.app/hardware/macbook-pro-m5-max-128/)unified memory · 96 GB usable · $6,699 | [Mistral-Small-4-119B-2603](https://whatcanirunai.vercel.app/models/mistral-small-4-119b-2603/)545 models fit | [LongCat-Flash-Lite](https://whatcanirunai.vercel.app/models/longcat-flash-lite/)512 models fit | [Seed-OSS-36B-Instruct](https://whatcanirunai.vercel.app/models/seed-oss-36b-instruct/)500 models fit | | [NVIDIA DGX Spark 128GB](https://whatcanirunai.vercel.app/hardware/dgx-spark-128/)unified memory · 96 GB usable · $6,950 | [Mistral-Small-4-119B-2603](https://whatcanirunai.vercel.app/models/mistral-small-4-119b-2603/)545 models fit | [LongCat-Flash-Lite](https://whatcanirunai.vercel.app/models/longcat-flash-lite/)512 models fit | [Seed-OSS-36B-Instruct](https://whatcanirunai.vercel.app/models/seed-oss-36b-instruct/)500 models fit | | [NVIDIA RTX Pro 6000 Blackwell 96GB](https://whatcanirunai.vercel.app/hardware/rtx-pro-6000-blackwell/)GPU · 96 GB usable · $8,565 | [Mistral-Small-4-119B-2603](https://whatcanirunai.vercel.app/models/mistral-small-4-119b-2603/)545 models fit | [LongCat-Flash-Lite](https://whatcanirunai.vercel.app/models/longcat-flash-lite/)512 models fit | [Seed-OSS-36B-Instruct](https://whatcanirunai.vercel.app/models/seed-oss-36b-instruct/)500 models fit | | [Mac Studio M5 Ultra 256GB](https://whatcanirunai.vercel.app/hardware/mac-studio-m5-ultra-256/)unified memory · 192 GB usable · $9,499 | [Intern-S1](https://whatcanirunai.vercel.app/models/intern-s1/)574 models fit | [Ling-3.0-flash-Sante](https://whatcanirunai.vercel.app/models/ling-3.0-flash-sante/)553 models fit | [Qwen2.5-VL-72B-Instruct](https://whatcanirunai.vercel.app/models/qwen2.5-vl-72b-instruct/)522 models fit | | [Mac Studio M5 Ultra 512GB](https://whatcanirunai.vercel.app/hardware/mac-studio-m5-ultra-512/)unified memory · 384 GB usable · $22,299 | [Qwen3-Coder-480B-A35B-Instruct](https://whatcanirunai.vercel.app/models/qwen3-coder-480b-a35b-instruct/)610 models fit | [Inkling-Small](https://whatcanirunai.vercel.app/models/inkling-small/)577 models fit | [Mixtral-8x22B-Instruct-v0.1](https://whatcanirunai.vercel.app/models/mixtral-8x22b-instruct-v0.1/)554 models fit | Other quant levels (Q5\_K\_M, Q6\_K, Q3, IQ formats) sit between these; see [GGUF VRAM requirements](https://whatcanirunai.vercel.app/gguf-vram-requirements/) for real Q3\_K\_M–Q8\_0 file sizes. The [VRAM calculator](https://whatcanirunai.vercel.app/vram-calculator/) checks any memory size at Q4, Q8 or FP16. ## Quick answers What is the difference between Q4, Q8 and FP16? They are how many bits store each weight. FP16 is the full 16-bit release most models ship in. Q8\_0 uses about 8.5 bits per weight and Q4\_K\_M about 4.85 (the extra fraction is scaling data), so a Q4\_K\_M file is about 30% the size of FP16 and Q8\_0 about 53%. Smaller files need less VRAM and run faster, at some cost in quality. Is Q4 good enough? For chat and most everyday tasks Q4\_K\_M is the common default in llama.cpp, Ollama and LM Studio, and the quality drop versus FP16 is usually small for models of 7B and up. Small models (under \~4B) and precise work like coding or math lose more, so use Q8 there if it fits. Our VRAM figures default to Q4\_K\_M for this reason. Should I run a bigger model at Q4 or a smaller one at Q8? The common rule of thumb is that a bigger model at Q4 beats a smaller model at Q8 or FP16 that uses the same memory. For example, Qwen3-32B at Q4 needs \~23.8 GB, about what Qwen3-8B needs at FP16 (\~19.8 GB). Below about 4 bits (Q3, Q2), quality falls off faster. How much VRAM does a 32B model need at Q4 vs Q8? Qwen3-32B needs \~23.8 GB at Q4\_K\_M, \~41.0 GB at Q8\_0 and \~76.4 GB at FP16, with \~4k context. The cheapest GPU we track that fits it at Q4 is the Intel Arc Pro B70 32GB ($949); at Q8 it's the NVIDIA RTX 6000 Ada 48GB ($6,799). Does quantization make a model faster? Yes, for generating tokens. Decode speed is limited by memory bandwidth, and a Q4\_K\_M file is about 57% the bytes of Q8\_0, so it runs roughly 1.8× as fast on the same hardware. Our estimates for Llama-3.3-70B-Instruct reflect that: compare the tok/s columns above. Do KV cache and context length change this? Yes. Our figures include \~4k tokens of context. Long contexts add KV-cache memory on top of the weights; see each model's "Memory by context length" table or set the context in the VRAM calculator. [How much VRAM do you need?](https://whatcanirunai.vercel.app/how-much-vram/) · [Tokens per second](https://whatcanirunai.vercel.app/tokens-per-second/) · [Best GPU for local LLMs](https://whatcanirunai.vercel.app/best-gpu-for-local-llm/) · [Best Mac](https://whatcanirunai.vercel.app/best-mac-for-local-llm/) · [VRAM calculator](https://whatcanirunai.vercel.app/vram-calculator/) --- Agent guide: https://whatcanirunai.vercel.app/llms.txt · JSON API: https://whatcanirunai.vercel.app/developers/