Can I run it?
Pick your hardware and context length to see which models fit, at the best quality that fits, with the memory each needs and a rough generation speed. Memory counts the weights, the context cache (from each model's real attention layout) and runtime buffers. Each model page has copy-paste commands for Ollama, llama.cpp, LM Studio and MLX.
RTX 4060 Ti · 16 GB: about 15 GB usable for a model. The best fit at 8K context is Nanbeige4.2 3B (index 70.4) at 8-bit, needing 5.8 GB and generating roughly 46 tokens/s.
| Model | Index | Runs at | Memory needed | Speed | Params |
|---|---|---|---|---|---|
| Ternary Bonsai 2 27B PrismML | 74.6 | Q2_0 (as released) Q2_0 | 9.3 GB | ≈ 26 tok/s | 27.4B |
| Nanbeige4.2 3B Nanbeige (Kanzhun) | 70.4 | 8-bit Q8_0 | 5.8 GB | ≈ 46 tok/s | 4.2B |
| Spark-X2.6 4B SparkLLM (XHToken) | 67.5 | 8-bit Q8_0 | 5.3 GB | ≈ 46 tok/s | 4.1B |
| Spark-X2.5 4B SparkLLM (XHToken) | 65.5 | 8-bit Q8_0 | 5.3 GB | ≈ 46 tok/s | 4.1B |
| gpt-oss-20b OpenAI | 64.0 | 4-bit Q4_K_M | 14 GB | ≈ 46 tok/s | 20.9B (3.6B active) |
| Qwen3.5 9B Alibaba (Qwen) | 63.9 | 8-bit Q8_0 | 11 GB | ≈ 21 tok/s | 9B |
| Granite 4.2 8B IBM (Granite) | 63.5 | 8-bit Q8_0 | 11 GB | ≈ 24 tok/s | 8B |
| Apriel 1.6 15B Thinker ServiceNow (Apriel) | 60.9 | 6-bit Q6_K | 15 GB | ≈ 17 tok/s | 14.9B |
| LFM3 1.7B Liquid AI | 60.0 | 8-bit Q8_0 | 2.5 GB | ≈ 110 tok/s | 1.7B |
| Nemotron Labs Dory 1.0 4B NVIDIA | 60.0 | 8-bit Q8_0 | 5.2 GB | ≈ 48 tok/s | 4B |
| Ministral 3 14B Mistral AI | 60.0 | 6-bit Q6_K | 14 GB | ≈ 18 tok/s | 14B |
| Qwen3.5 4B Alibaba (Qwen) | 57.8 | 8-bit Q8_0 | 5.1 GB | ≈ 47 tok/s | 4B |
| MiniCPM5 2B OpenBMB (MiniCPM) | 57.5 | 8-bit Q8_0 | 3.6 GB | ≈ 75 tok/s | 2.5B |
| Falcon H1R 7B TII (Falcon) | 57.5 | 8-bit Q8_0 | 9.1 GB | ≈ 25 tok/s | 7.6B |
| Gemma 4 12B Google | 56.9 | 8-bit Q8_0 | 14 GB | ≈ 16 tok/s | 11.9B |
| Qwen3 4B Thinking 2507 Alibaba (Qwen) | 55.7 | 8-bit Q8_0 | 6.0 GB | ≈ 47 tok/s | 4B |
| Ministral 3 8B Mistral AI | 55.2 | 8-bit Q8_0 | 10 GB | ≈ 24 tok/s | 8B |
| Granite 4.2 3B IBM (Granite) | 54.8 | 8-bit Q8_0 | 4.4 GB | ≈ 63 tok/s | 3B |
| Phi-4-reasoning-plus Microsoft (Phi) | 54.4 | 6-bit Q6_K | 15 GB | ≈ 17 tok/s | 14.7B |
| DeepSeek-R1-0528-Qwen3-8B DeepSeek | 53.1 | 8-bit Q8_0 | 11 GB | ≈ 23 tok/s | 8.2B |
| MiniCPM-SALA OpenBMB (MiniCPM) | 52.6 | 8-bit Q8_0 | 11 GB | ≈ 20 tok/s | 9.5B |
| Hunyuan 7B Instruct Tencent (Hunyuan) | 52.5 | 8-bit Q8_0 | 9.7 GB | ≈ 25 tok/s | 7.5B |
| Nemotron Nano 9B v2 NVIDIA | 52.5 | 8-bit Q8_0 | 10 GB | ≈ 21 tok/s | 8.9B |
| Ling 3.0 Tiny InclusionAI (Ling) | 51.9 | 8-bit Q8_0 | 9.3 GB | ≈ 68 tok/s | 7.9B (1.4B active) |
| Spark-X2.6 1.7B SparkLLM (XHToken) | 50.1 | 8-bit Q8_0 | 2.5 GB | ≈ 111 tok/s | 1.7B |
| Ministral 3 3B Mistral AI | 49.4 | 8-bit Q8_0 | 4.6 GB | ≈ 63 tok/s | 3B |
| Qwen3 8B Alibaba (Qwen) | 48.5 | 8-bit Q8_0 | 11 GB | ≈ 23 tok/s | 8.2B |
| Edge0 8B-A1B (preview) Edge0 | 47.2 | 8-bit Q8_0 | 9.3 GB | ≈ 68 tok/s | 7.9B (1.4B active) |
| Phi-4 Microsoft (Phi) | 44.7 | 6-bit Q6_K | 15 GB | ≈ 17 tok/s | 14.7B |
| Tri-21B Think Trillion Labs | 43.6 | 4-bit Q4_K_M | 15 GB | ≈ 16 tok/s | 20.7B |
| OLMo 3 7B Think Ai2 (OLMo) | 43.4 | 8-bit Q8_0 | 11 GB | ≈ 26 tok/s | 7.3B |
| LFM2.5 2.6B Liquid AI | 43.2 | 8-bit Q8_0 | 3.6 GB | ≈ 70 tok/s | 2.7B |
| MiMo-7B-RL Xiaomi (MiMo) | 40.7 | 8-bit Q8_0 | 10 GB | ≈ 24 tok/s | 7.8B |
| DeepSeek-R1-Distill-Qwen-7B DeepSeek | 39.8 | 8-bit Q8_0 | 9.2 GB | ≈ 25 tok/s | 7.6B |
| Gemma 4 E4B Google | 38.2 | 8-bit Q8_0 | 9.3 GB | ≈ 42 tok/s | 8B |
| EXAONE 4.0 1.2B LG AI Research (EXAONE) | 36.1 | 8-bit Q8_0 | 2.4 GB | ≈ 148 tok/s | 1.3B |
| Qwen3.5 2B Alibaba (Qwen) | 33.0 | 8-bit Q8_0 | 3.1 GB | ≈ 82 tok/s | 2.3B |
| Gemma 4 E2B Google | 31.9 | 8-bit Q8_0 | 6.1 GB | ≈ 82 tok/s | 5.1B |
| SmolLM3 3B Hugging Face (SmolLM) | 30.4 | 8-bit Q8_0 | 4.4 GB | ≈ 62 tok/s | 3.1B |
| LFM2.5 1.2B Thinking Liquid AI | 30.1 | 8-bit Q8_0 | 1.9 GB | ≈ 162 tok/s | 1.2B |
| LLaDA2.2 mini InclusionAI (Ling) | 29.7 | 6-bit Q6_K | 15 GB | ≈ 88 tok/s | 16.3B (1.4B active) |
| Gemma 3 12B Google | 28.4 | 8-bit Q8_0 | 15 GB | ≈ 16 tok/s | 12.2B |
| Llama 3.1 8B Meta | 21.3 | 8-bit Q8_0 | 10 GB | ≈ 24 tok/s | 8B |
| LFM2.5 1.2B Instruct Liquid AI | 20.6 | 8-bit Q8_0 | 1.9 GB | ≈ 162 tok/s | 1.2B |
| Apertus 1.5 8B Swiss AI Initiative (Apertus) | 20.1 | 8-bit Q8_0 | 11 GB | ≈ 21 tok/s | 8.9B |
| Gemma 3n E4B Google | 18.2 | 8-bit Q8_0 | 9.1 GB | ≈ 47 tok/s | 7.8B |
| Gemma 3 4B Google | 17.5 | 8-bit Q8_0 | 5.4 GB | ≈ 44 tok/s | 4.3B |
| Gemma 3n E2B Google | 14.3 | 8-bit Q8_0 | 6.5 GB | ≈ 95 tok/s | 5.4B |
| LFM2.5 350M Liquid AI | 12.6 | 8-bit Q8_0 | 1.0 GB | ≈ 542 tok/s | 350M |
| Qwen3.5 0.8B Alibaba (Qwen) | 10.7 | 8-bit Q8_0 | 1.6 GB | ≈ 211 tok/s | 900M |
| Gemma 3 1B Google | 6.1 | 8-bit Q8_0 | 1.6 GB | ≈ 190 tok/s | 1B |
| Gemma 3 270M Google | — | 8-bit Q8_0 | 0.8 GB | ≈ 703 tok/s | 270M |
| Baguettotron PleIAs | — | 8-bit Q8_0 · max 4K | 1.1 GB | ≈ 593 tok/s | 320M |
| ERNIE 4.5 0.3B Baidu (ERNIE) | — | 8-bit Q8_0 | 1.0 GB | ≈ 527 tok/s | 360M |
| MiniCPM5 1B OpenBMB (MiniCPM) | — | 8-bit Q8_0 | 1.9 GB | ≈ 176 tok/s | 1.1B |
| Llama 3.2 1B Meta | — | 8-bit Q8_0 | 2.1 GB | ≈ 158 tok/s | 1.2B |
| MiniCPM-V 4.6 Thinking OpenBMB (MiniCPM) | — | 8-bit Q8_0 | 2.0 GB | ≈ 146 tok/s | 1.3B |
| MiniCPM-V 4.6 OpenBMB (MiniCPM) | — | 8-bit Q8_0 | 2.0 GB | ≈ 146 tok/s | 1.3B |
| MiniCPM-V 4.7 1B OpenBMB (MiniCPM) | — | 8-bit Q8_0 | 2.3 GB | ≈ 122 tok/s | 1.6B |
| LFM3-VL 2B Liquid AI | — | 8-bit Q8_0 | 2.7 GB | ≈ 99 tok/s | 1.9B |
| LFM3-Omni 2B Liquid AI | — | 8-bit Q8_0 | 3.0 GB | ≈ 87 tok/s | 2.2B |
| LFM2.5-VL 3B Liquid AI | — | 8-bit Q8_0 | 4.0 GB | ≈ 61 tok/s | 3.1B |
| Llama 3.2 3B Meta | — | 8-bit Q8_0 | 4.9 GB | ≈ 59 tok/s | 3.2B |
| Phi-4-mini Microsoft (Phi) | — | 8-bit Q8_0 | 5.7 GB | ≈ 50 tok/s | 3.8B |
| Nemotron 3 Nano 4B NVIDIA | — | 8-bit Q8_0 | 5.6 GB | ≈ 47 tok/s | 4B |
| AgentCPM-Explore 4B OpenBMB (MiniCPM) | — | 8-bit Q8_0 | 6.1 GB | ≈ 47 tok/s | 4B |
| FrogNano 4B Microsoft (Phi) | — | 8-bit Q8_0 | 5.8 GB | ≈ 41 tok/s | 4.7B |
| Teuken 7B Instruct v0.6 OpenGPT-X (Teuken) | — | 8-bit Q8_0 · max 4K | 8.7 GB | ≈ 25 tok/s | 7.5B |
| Salamandra 7B Instruct Barcelona Supercomputing Center (ALIA) | — | 8-bit Q8_0 | 10.0 GB | ≈ 24 tok/s | 7.8B |
| Nemotron 3.5 Bolt E4B NVIDIA | — | 8-bit Q8_0 | 9.3 GB | ≈ 42 tok/s | 8B |
| Luciole 8B Instruct OpenLLM-France (Luciole) | — | 8-bit Q8_0 | 9.4 GB | ≈ 23 tok/s | 8.1B |
| MiMo-VL-7B-RL Xiaomi (MiMo) | — | 8-bit Q8_0 | 11 GB | ≈ 23 tok/s | 8.3B |
| Rnj-1.5 Instruct Essential AI (Rnj) | — | 8-bit Q8_0 | 11 GB | ≈ 23 tok/s | 8.3B |
| LFM2.5 8B-A1B Liquid AI | — | 8-bit Q8_0 | 9.8 GB | ≈ 95 tok/s | 8.5B (1B active) |
| GLM-4.6V-Flash Z.ai (GLM) | — | 8-bit Q8_0 | 11 GB | ≈ 21 tok/s | 9B |
| EuroLLM 9B Instruct (2512) EuroLLM (UTTER consortium) | — | 8-bit Q8_0 | 12 GB | ≈ 21 tok/s | 9.2B |
| MiniCPM-o 4.5 OpenBMB (MiniCPM) | — | 8-bit Q8_0 | 12 GB | ≈ 20 tok/s | 9.4B |
| MiMo-V2.6-Distill-Qwen-9B Xiaomi (MiMo) | — | 8-bit Q8_0 | 12 GB | ≈ 20 tok/s | 9.4B |
| Step3-VL-10B StepFun | — | 8-bit Q8_0 | 13 GB | ≈ 19 tok/s | 10B |
| HyperCLOVA X SEED Omni 8B NAVER (HyperCLOVA X) | — | 8-bit Q8_0 · max 8K | 13 GB | ≈ 18 tok/s | 10.7B |
| Bielik 11B v3 Instruct SpeakLeash (Bielik) | — | 8-bit Q8_0 | 14 GB | ≈ 17 tok/s | 11.2B |
| Phi-4-reasoning-vision-15B Microsoft (Phi) | — | 6-bit Q6_K | 15 GB | ≈ 16 tok/s | 15B |
| Ring-mini-2.0 InclusionAI (Ling) | — | 6-bit Q6_K | 15 GB | ≈ 88 tok/s | 16.3B (1.4B active) |
| Ling-mini-2.0 InclusionAI (Ling) | — | 6-bit Q6_K | 15 GB | ≈ 88 tok/s | 16.3B (1.4B active) |
| Kimi-VL-A3B-Thinking-2506 Moonshot AI (Kimi) | — | 6-bit Q6_K | 15 GB | ≈ 44 tok/s | 16.4B (2.8B active) |
| gpt-oss-safeguard-20b OpenAI | — | 4-bit Q4_K_M | 14 GB | ≈ 46 tok/s | 20.9B (3.6B active) |
| ERNIE 4.5 21B-A3B Thinking Baidu (ERNIE) | — | 4-bit Q4_K_M | 15 GB | ≈ 55 tok/s | 21.8B (3B active) |
Estimates for llama.cpp-style GGUF inference (Ollama, LM Studio and llama.cpp itself). Macs reserve part of their memory for the system by default; phones leave about half of it to apps. Vision encoders and very long prompts add more on top. See the methodology.
Questions
How much memory does an LLM need to run locally?
Three parts add up: the weights (parameters × bits per weight ÷ 8), the context cache, which grows with every token in the conversation, and about half a gigabyte of runtime buffers. At 4-bit (Q4_K_M, about 4.85 bits per weight) an 8B model's weights take about 4.8 GB, and Llama 3.1 8B needs another 4.3 GB for a 32K-token context at 16-bit.
Why does the context length change what fits?
Each token in the context keeps keys and values for every attention layer. Llama 3.1 8B stores about 128 KB per token, so 128K tokens need 17 GB, more than its 4-bit weights. Newer hybrid designs such as Qwen3.8 27B use full attention in only some layers, so their cache is far smaller: 8.6 GB at 128K. An 8-bit context cache halves these numbers.
Can I run a model that is bigger than my GPU memory?
Yes, with llama.cpp or Ollama part of the model can stay in system RAM, at a cost in speed. It works best for Mixture-of-Experts models, which read only a few experts per token: Qwen3.6 35B-A3B has 35B parameters but reads only 3B per token, about 1.8 GB at 4-bit. Turn on "Spill into system RAM" above to see it.
How accurate are the speed estimates?
Generating text is limited by memory bandwidth: every new token reads the active weights once. The estimate divides the hardware's bandwidth by that amount and applies a typical efficiency, so expect real speeds within about a third either way. It covers generation with a short prompt; reading a long prompt and very full contexts are slower.