Overview
An LLM’s VRAM need is the weight file plus the KV cache plus runtime overhead. The weight formula and the KV formula are defined on llm-quantization-explained and repeated below with the model each line uses.
How much VRAM does an LLM need?
Add three pieces. Weights are params × bits_per_weight ÷ 8 bytes. KV cache is 2 × layers × kv_heads × head_dim × context × bytes_per_value bytes. Overhead of 1 to 2 GB is an assumption, not a measured allocator reserve. Decimal GB on this page is bytes ÷ 1e9.
Bits per weight for the Qwen estimates are llama.cpp’s Q4_K_M value 4.8944 and Q8_0 value 8.5008, from the quantize README read on 2026-10-12. Parameter counts are the Qwen2.5 model cards. Llama 3.1 sizes below are the README’s published file sizes, not a recomputed estimate.
How much VRAM for a 70B model?
The quantize README lists Llama 3.1 70B Q4_K_M at 43.1 GB and the original checkpoint at 280.9 GB. Those labels are GB in that table. 43.1 GB of weights does not fit in 24 GB or 32 GB of VRAM before KV cache is added. A round line of 70 × 4 ÷ 8 = 35 GB uses 4.000 bits and 70e9 parameters. It is not the README file size.
Qwen2.5-72B is 72.7B parameters on its model card, not a 70B Llama file. At 4.8944 bits per weight the weight estimate is 72.7e9 × 4.8944 ÷ 8 ÷ 1e9 = 44.4779 decimal GB. At Q8_0 it is 77.2510 GB. Neither fits a 24 GB card or a 32 GB card on weights alone.
The 72B card lists 80 layers, 64 query heads, and 8 KV heads. config.json lists hidden_size 8192, so head dimension is 8192 ÷ 64 = 128. With 2 bytes per value (bfloat16 checkpoint dtype): KV bytes = 2 × 80 × 8 × 128 × context × 2 = 327,680 × context. At 4,096 tokens that is 1.3422 decimal GB. At 32,768 tokens it is 10.7374 GB. The card’s full 131,072-token context is larger still and is not claimed to fit here.
How much VRAM for 7B, 14B and 32B models?
Qwen card counts and the same bits per weight. KV uses each config.json: head dimension 128 in every case below, and 2 bytes per value because torch_dtype is bfloat16.
| Card | Layers | KV heads | Q4_K_M weights | Q8_0 weights | KV at 4,096 | KV at 32,768 |
|---|---|---|---|---|---|---|
| Qwen2.5-7B (7.61B) | 28 | 4 | 4.6558 GB | 8.0864 GB | 0.2349 GB | 1.8790 GB |
| Qwen2.5-14B (14.7B) | 48 | 8 | 8.9935 GB | 15.6202 GB | 0.8053 GB | 6.4425 GB |
| Qwen2.5-32B (32.5B) | 64 | 8 | 19.8835 GB | 34.5345 GB | 1.0737 GB | 8.5899 GB |
7B Q4_K_M example at 4,096 context: 4.6558 + 0.2349 + an assumed 1 to 2 GB overhead = 5.8907 to 6.8907 decimal GB. 32B Q4_K_M at 4,096: 19.8835 + 1.0737 + 1 to 2 = 21.9572 to 22.9572 GB. 32B Q8_0 weights alone are 34.5345 GB, which is already over 32 GB. Llama 3.1 8B Q4_K_M is the README’s 4.9 GB file (4.58 GiB in the other table), not the Qwen 7.61B estimate.
How does context length add to VRAM?
KV bytes scale in a straight line with context length. For Qwen2.5-32B the product is 2 × 64 × 8 × 128 × context × 2 = 262,144 × context bytes. At 4,096 tokens that is 1.0737 GB. At 32,768 it is 8.5899 GB. At the card’s 131,072-token maximum it is 34.3597 GB of KV cache alone, before weights. A card can hold the weights and still fail the long context.
Why do calculators give different answers?
They pick different bits per weight (4.000 versus the README’s 4.8944), different parameter counts (a round 70e9 versus the card’s 72.7B or the README’s 43.1 GB file), GB versus GiB, and different context lengths. Some add a KV term and some do not. This site shows each term. It does not publish a calculator yet, so there is no software-application listing for one.
System RAM for a Mac is a different page: how much RAM for a local LLM. When the GPU does not hold the file, run-llm-on-cpu is the offload page.
Which figure should I plan a 70B Q4 download around?
Use 43.1 GB from the llama.cpp Memory/Disk table for Llama 3.1 70B Q4_K_M, and still add KV cache and overhead for VRAM. Do not plan that file as 35 GB.
Is the 1 to 2 GB overhead a measurement?
No. It is an assumption so a result that lands within 1 GB of the card size is not called a fit.
Sources read on 2026-10-12
- llama.cpp quantize README for bits per weight, GiB sizes, and the Memory/Disk table.
- Qwen2.5-7B, 14B, 32B, and 72B model cards, plus each repo’s
config.json, for parameter counts, layers, heads, hidden size, andtorch_dtype. - RTX 5090, RTX 4090, RTX 3090 family, and RTX 4060 family.
- Radeon RX 7900 XTX and Radeon RX 9070 XT.
- Ollama GPU, Ollama FAQ, Ollama API, and llama.cpp CLI options.
- LM Studio system requirements.
- Intel Arc B580 specifications at
https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.htmlreturned HTTP 403. No Intel VRAM figure is cited.
Related
- llm-quantization-explained is the formula.
- vram-requirements applies it by model size.
- 24gb-vs-16gb-vs-12gb-vram applies it by card size.
- best-gpu-for-local-llm names the cards.
- rtx-3090-vs-4090-vs-5090-llm compares three NVIDIA cards.
- local-llm-pc-build covers the rest of the PC.
- run-llm-on-cpu covers CPU and partial offload.
- methodology is the citation rule.
- Mac memory questions: what fits my RAM and how much RAM for a local LLM.