# How we calculate | What Can I Run AI?

> Fit formula

Source: https://whatcanirunai.vercel.app/method/

# How we calculate

Every verdict on this site comes from one formula. It's an estimate: real use varies with context length and runtime.

## Memory a model needs

weights GB = params × bits per weight ÷ 8  
needed GB = weights × 1.15 + 1 GB

Bits per weight: Q4\_K\_M ≈ 4.85, Q8\_0 ≈ 8.5, FP16 = 16\. The 15% plus 1 GB covers the KV cache at \~4k context and runtime overhead.

## Verdicts

Fits needs 90% or less of usable memory.

Tight needs 90 to 100%. It loads, but keep the context short.

Won’t fit needs more than you have, so layers spill to the CPU and it gets slow.

## Context length and the KV cache

The base estimate covers about 4k tokens of context. For longer contexts we add the KV cache: full-attention layers × KV heads × head size × 2 (keys and values) × 2 bytes (FP16) per token, read from each model’s config.json on Hugging Face. Sliding-window layers only cache their window, and linear-attention, Mamba and conv layers keep a small fixed state, so hybrid models grow much more slowly. A Q8\_0 cache is about 53% of FP16 and Q4\_0 about 28%. Models with compressed or shared attention (for example MLA) aren’t sized yet and show “cache not modeled”.

## Usable memory

GPUs: all of the VRAM. Macs and unified-memory mini-PCs: about 75% of memory, a rule of thumb for what the GPU may use.

Want to plug in your own numbers? Use the [VRAM calculator](https://whatcanirunai.vercel.app/vram-calculator/).

## Estimated speed (tokens per second)

tok/s ≈ memory bandwidth (GB/s) ÷ model weights (GB) × efficiency

Generating each token reads every active weight once, so decode speed is capped by memory bandwidth. Efficiency bands: **60–85%** for discrete GPUs, **45–80%** for unified memory (Macs, Ryzen AI Max, DGX Spark). We show the range, not one number.

Calibration (Llama 2 7B Q4\_0, 3.82 GB, tg128): [llama.cpp CUDA scoreboard #15013](https://github.com/ggml-org/llama.cpp/discussions/15013) gives RTX 3090 162 tok/s (66% of 936 GB/s), RTX 4090 189 (72%), RTX 5090 290 (62%), RTX 3080 140 (70%), RTX 4070 Ti Super 132 (75%), RTX 4060 Ti 64 (85%). [llama.cpp Apple Silicon scoreboard #4167](https://github.com/ggml-org/llama.cpp/discussions/4167) gives M1 Ultra 84 tok/s (40%), M2 Ultra 94 (45%), M4 Max 83 (58%), M5 Max 120 (75%), M6 36 (80%). Expect real numbers up to about ±25% outside the band: longer contexts, other runtimes and other quant formats all move it.

Mixture-of-experts models: when active parameters are known (from the name, e.g. 30B-A3B, or OpenAI's gpt-oss model card), we show only an **upper bound** ("up to") from the active weights, because routing and attention overhead isn't calibrated. Otherwise we use total parameters, which underestimates MoE speed. Multi-GPU: memory adds up, but llama.cpp's default layer split runs cards in turn, so speed is about one card's on the whole model. Prompt processing (prefill) is compute-bound and not estimated.

## Power and fit checks

PSU ≥ (GPU board power × count + CPU TDP + 150 W) × 1.3

GPU board power, length and slot width come from NVIDIA Founders Edition and AMD reference spec pages; partner cards are often longer and hungrier. Case limits and PSU connectors come from Corsair product pages. Running cost is load watts × your electricity rate (the $0.18/kWh default is an example, not a sourced average). Max context is not shown yet: it needs per-model KV-cache data we haven't sourced.

---
Agent guide: https://whatcanirunai.vercel.app/llms.txt · JSON API: https://whatcanirunai.vercel.app/developers/