HomeCan I run it?Mistral Small 24BMistral Small 24B on an RTX 3070 Ti
No. Mistral Small 24B at Q4_K_M needs about 17.7 GB, and the GeForce RTX 3070 Ti has 8.0 GB usable at a ~4k context.
Can I run Mistral Small 24B on an RTX 3070 Ti?
Mistral Small 24B at Q4_K_M needs about 17.7 GB, and the GeForce RTX 3070 Ti has 8.0 GB usable. That figure is 8 GB of VRAM on the spec sheet, of which 8.0 GB is usable for weights and cache, at 608 GB/s memory bandwidth. Mistral’s 24B Small instruct model, a dense mid-size alternative to the 27B–32B class. These are estimates from our formula, not measured benchmarks.
- Quant that fits
- None of Q4, Q8 or FP16 needs 17.7 GB
- Memory needed
- 17.7 GB at Q4_K_M, ~4k context
- Usable VRAM
- 8.0 GB of 8 GB VRAM
- Est. tokens/sec
- n/a does not fit
Memory at each quant
| Quant | Need vs 8.0 GB | Verdict |
|---|---|---|
| Q4_K_M | 17.7 GB | Q4Won’t fit |
| Q8_0 | 30.3 GB | Q8Won’t fit |
| FP16 | 56.2 GB | FP16Won’t fit |
Estimated tokens per second
No speed estimate: Mistral Small 24B does not fit in the 8.0 GB this machine can use, so we do not publish a tokens/sec figure for CPU offload.
Context headroom
Even at a short ~4k context, the weights do not fit. Extending context would only increase the gap.
| Context | KV cache (FP16) | Total at Q4_K_M | Verdict |
|---|---|---|---|
| 4k | 0.7 GB | 17.7 GB | Won’t fit |
| 8k | 1.3 GB | 18.4 GB | Won’t fit |
| 16k | 2.7 GB | 19.8 GB | Won’t fit |
| 32k | 5.4 GB | 22.4 GB | Won’t fit |
| 64k | 10.7 GB | 27.8 GB | Won’t fit |
| 128k | 21.5 GB | 38.5 GB | Won’t fit |
What to do instead
- Run Qwen3-8B instead (6.7 GB at Q4). It is a different model, not a drop-in, but it does fit this machine.
- To hold Mistral Small 24B at Q4, you need about 17.7 GB. The Mac mini M6 24GB has 18.0 GB usable ($1299 Apple list price, M6 12C CPU/12C GPU, 24GB/512GB (MHQM4LL/A)).
- CPU offload can load the weights anyway, but it is slow. We do not publish an offload tokens/sec figure.
- Full hardware list: Mistral-Small-3.2-24B-Instruct-2506. Specs for this machine: GeForce RTX 3070 Ti.
Other hardware for Mistral Small 24B
- Mistral Small 24B on an RTX 3060 12GBWon’t fit
- Mistral Small 24B on an RTX 3080Won’t fit
- Mistral Small 24B on an RTX 3090Fits
- Mistral Small 24B on an RTX 4060 Ti 16GBWon’t fit
- Mistral Small 24B on an RTX 4070 Ti SuperWon’t fit
- Mistral Small 24B on an RTX 4090Fits
Other models on an RTX 3070 Ti
- Qwen3-32B on an RTX 3070 TiWon’t fit
- Qwen3-30B-A3B on an RTX 3070 TiWon’t fit
- Qwen3.8-27B on an RTX 3070 TiWon’t fit
- Gemma 3 27B on an RTX 3070 TiWon’t fit
- Gemma 4 31B on an RTX 3070 TiWon’t fit
- gpt-oss-20b on an RTX 3070 TiWon’t fit
Quick answers
- Can I run Mistral Small 24B on an RTX 3070 Ti?
- No. Mistral Small 24B at Q4_K_M needs about 17.7 GB, and the GeForce RTX 3070 Ti has 8.0 GB usable.
- How much VRAM does Mistral Small 24B need?
- About 17.7 GB at Q4_K_M, 30.3 GB at Q8_0 and 56.2 GB at FP16, estimated for a ~4k context.
- How fast is Mistral Small 24B on an RTX 3070 Ti?
- We do not estimate tokens/sec when the model does not fit. CPU offload is possible but slow.
- What context length can I use?
- The weights already do not fit, so longer context is not the limiting factor.