Best local LLMs for 96 GB of VRAM
Every open model we track that runs fully in 96 GB of GPU memory, biggest first. Checks are at Q4_K_M with about 4k context.
- Run fully at Q4
- 545 of 656
- Run at Q8 too
- 512
- Biggest at Q4
- Mistral-Small-4-119B-2603
With 96 GB, the biggest model you can run comfortably is Mistral-Small-4-119B-2603 (~84.2 GB at Q4). If you want higher quality at Q8, LongCat-Flash-Lite is the biggest that fits.
Models that fit in 96 GB
Bar: Q4 memory needed vs. 96 GB. The tick marks 90%, our comfort line.FitsTight
| Model | Q4 need vs. 96 GB | Q4 / Q8 |
|---|---|---|
| gpt-oss-120b116.8B params, openai | 82.5 / 96.0 GB | Q4FitsQ8Won’t fit |
| Kolibri-178.1B params, Aleph-Alpha | 55.5 / 96.0 GB | Q4FitsQ8Won’t fit |
| Qwen2.5-72B-Instruct72.7B params, Qwen | 51.7 / 96.0 GB | Q4FitsQ8Tight |
| Llama-3.3-70B-Instruct70.6B params, meta-llama | 50.2 / 96.0 GB | Q4FitsQ8Tight |
| DeepSeek-R1-Distill-Llama-70B70.6B params, deepseek-ai | 50.2 / 96.0 GB | Q4FitsQ8Tight |
| Recursive Flint39.8B params, RecursiveSI | 28.7 / 96.0 GB | Q4FitsQ8Fits |
| Ornith-1.5-35B-A3B36.0B params, ornith-ai | 26.1 / 96.0 GB | Q4FitsQ8Fits |
| Sage-A1.536.0B params, InternScience | 26.1 / 96.0 GB | Q4FitsQ8Fits |
| MemT-1.0-Nano35.1B params, MemTensor | 25.5 / 96.0 GB | Q4FitsQ8Fits |
| Nanbeige4.3-33B33.1B params, Nanbeige | 24.1 / 96.0 GB | Q4FitsQ8Fits |
| Qwen2.5-Coder-32B-Instruct32.8B params, Qwen | 23.8 / 96.0 GB | Q4FitsQ8Fits |
| DeepSeek-R1-Distill-Qwen-32B32.8B params, deepseek-ai | 23.8 / 96.0 GB | Q4FitsQ8Fits |
| Qwen3-32B32.8B params, Qwen | 23.8 / 96.0 GB | Q4FitsQ8Fits |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF1631.6B params, nvidia | 23.0 / 96.0 GB | Q4FitsQ8Fits |
| Gemma 4 31B31.3B params, google | 22.8 / 96.0 GB | Q4FitsQ8Fits |
| Qwen3-30B-A3B30.5B params, Qwen | 22.3 / 96.0 GB | Q4FitsQ8Fits |
| Qwen3-Coder-30B-A3B-Instruct30.5B params, Qwen | 22.3 / 96.0 GB | Q4FitsQ8Fits |
| Hy-Memory-30B-A3B30.1B params, tencent | 22.0 / 96.0 GB | Q4FitsQ8Fits |
| Muse-Glimmer-30B29.8B params, meta-models | 21.8 / 96.0 GB | Q4FitsQ8Fits |
| Raon-1 mini preview28.8B params, KRAFTON | 21.1 / 96.0 GB | Q4FitsQ8Fits |
| Qwen3.8-27B27.8B params, Qwen | 20.4 / 96.0 GB | Q4FitsQ8Fits |
| Tess-4.5-27B27.8B params, migtissera | 20.4 / 96.0 GB | Q4FitsQ8Fits |
| gemma-3-27b-it27.4B params, google | 20.1 / 96.0 GB | Q4FitsQ8Fits |
| Gemma 4 26B A4B25.8B params, google | 19.0 / 96.0 GB | Q4FitsQ8Fits |
| Mistral-Small-3.2-24B-Instruct-250624.0B params, mistralai | 17.7 / 96.0 GB | Q4FitsQ8Fits |
| LFM2.5-24B-A2B24.0B params, LiquidAI | 17.7 / 96.0 GB | Q4FitsQ8Fits |
| gpt-oss-20b20.9B params, openai | 15.6 / 96.0 GB | Q4FitsQ8Fits |
| Carbon-X (DNA)14.8B params, HuggingFaceBio | 11.3 / 96.0 GB | Q4FitsQ8Fits |
| Qwen2.5-Coder-14B-Instruct14.8B params, Qwen | 11.3 / 96.0 GB | Q4FitsQ8Fits |
| DeepSeek-R1-Distill-Qwen-14B14.8B params, deepseek-ai | 11.3 / 96.0 GB | Q4FitsQ8Fits |
| Qwen3-14B14.8B params, Qwen | 11.3 / 96.0 GB | Q4FitsQ8Fits |
| phi-414.7B params, microsoft | 11.2 / 96.0 GB | Q4FitsQ8Fits |
| OLMo-2-1124-13B-Instruct13.7B params, allenai | 10.6 / 96.0 GB | Q4FitsQ8Fits |
| Mistral-Nemo-Instruct-240712.2B params, mistralai | 9.5 / 96.0 GB | Q4FitsQ8Fits |
| gemma-3-12b-it12.2B params, google | 9.5 / 96.0 GB | Q4FitsQ8Fits |
| Mellum2.1-12B-A2.5B-Thinking12.1B params, JetBrains | 9.5 / 96.0 GB | Q4FitsQ8Fits |
| Gemma 4 12B12.0B params, google | 9.3 / 96.0 GB | Q4FitsQ8Fits |
| Ornith-1.5-9B9.7B params, ornith-ai | 7.7 / 96.0 GB | Q4FitsQ8Fits |
| Qwen3-8B8.2B params, Qwen | 6.7 / 96.0 GB | Q4FitsQ8Fits |
| granite-3.3-8b-instruct8.2B params, ibm-granite | 6.7 / 96.0 GB | Q4FitsQ8Fits |
| Llama-3.1-8B-Instruct8.0B params, meta-llama | 6.6 / 96.0 GB | Q4FitsQ8Fits |
| Nemotron 3.5 Bolt E4B8.0B params, nvidia | 6.6 / 96.0 GB | Q4FitsQ8Fits |
| Gemma 4 E4B8.0B params, google | 6.6 / 96.0 GB | Q4FitsQ8Fits |
| Qwen2.5-Coder-7B-Instruct7.6B params, Qwen | 6.3 / 96.0 GB | Q4FitsQ8Fits |
| DeepSeek-R1-Distill-Qwen-7B7.6B params, deepseek-ai | 6.3 / 96.0 GB | Q4FitsQ8Fits |
| Mistral-7B-Instruct-v0.37.2B params, mistralai | 6.1 / 96.0 GB | Q4FitsQ8Fits |
| Gemma 4 E2B5.1B params, google | 4.6 / 96.0 GB | Q4FitsQ8Fits |
| gemma-3-4b-it4.3B params, google | 4.0 / 96.0 GB | Q4FitsQ8Fits |
| Spark-X2.6-4B4.1B params, XHToken | 3.9 / 96.0 GB | Q4FitsQ8Fits |
| Qwen3-4B4.0B params, Qwen | 3.8 / 96.0 GB | Q4FitsQ8Fits |
| Nemotron Labs Dory 1.0 4B4.0B params, nvidia | 3.8 / 96.0 GB | Q4FitsQ8Fits |
| Phi-4-mini-instruct3.8B params, microsoft | 3.7 / 96.0 GB | Q4FitsQ8Fits |
| Llama-3.2-3B-Instruct3.2B params, meta-llama | 3.2 / 96.0 GB | Q4FitsQ8Fits |
| MiniCPM5-2B2.5B params, openbmb | 2.8 / 96.0 GB | Q4FitsQ8Fits |
| Qwen3-1.7B2.0B params, Qwen | 2.4 / 96.0 GB | Q4FitsQ8Fits |
| LFM3-1.7B1.7B params, LiquidAI | 2.2 / 96.0 GB | Q4FitsQ8Fits |
| Llama-3.2-1B-Instruct1.2B params, meta-llama | 1.9 / 96.0 GB | Q4FitsQ8Fits |
| gemma-3-1b-it1.0B params, google | 1.7 / 96.0 GB | Q4FitsQ8Fits |
| Qwen3-0.6B0.8B params, Qwen | 1.5 / 96.0 GB | Q4FitsQ8Fits |
| Ling-3.0-flash-Sante127.5B params, inclusionAI | 89.9 / 96.0 GB | Q4TightQ8Won’t fit |
| Ornith-1.5-122B125.1B params, ornith-ai | 88.2 / 96.0 GB | Q4TightQ8Won’t fit |
Hardware we track with 96–127 GB usable
- Framework Desktop (Ryzen AI Max+ 395) 128GB96 GB usable, $3449
- MacBook Pro M4 Max 128GB96 GB usable, $4999
- MacBook Pro M5 Max 128GB96 GB usable, $6699
- NVIDIA DGX Spark 128GB96 GB usable, $6950
- NVIDIA RTX Pro 6000 Blackwell 96GB96 GB usable, $8565
Quick answers
- What is the biggest LLM I can run with 96 GB of VRAM?
- Mistral-Small-4-119B-2603 (119.4B parameters) is the biggest model we track that runs comfortably in 96 GB at Q4_K_M, needing about 84.2 GB with a ~4k context. At Q8 the biggest is LongCat-Flash-Lite.
- How many models fit in 96 GB of VRAM?
- 545 of the 656 open models we track run fully in 96 GB at Q4 with room to spare, 512 also fit at Q8, and 8 more fit only with a short context.
- Can I run a model that needs more than 96 GB?
- Yes, with llama.cpp or Ollama you can offload some layers to system RAM, but speed drops sharply. For full-speed local inference the whole model plus its context cache should fit in GPU memory; we use 90% of 96 GB as the comfort line.
- Which hardware has about 96 GB of usable memory?
- The cheapest current device we track in this range is the Framework Desktop (Ryzen AI Max+ 395) 128GB (96 GB usable, $3,449 list price as of our last snapshot). 5 tracked devices have 96–127 GB usable.
Other memory sizes
4 GB · 6 GB · 8 GB · 10 GB · 12 GB · 16 GB · 20 GB · 24 GB · 32 GB · 48 GB · 64 GB · 128 GB