LLM VRAM Calculator: Weights, KV Cache and Overhead

Overview

An LLM VRAM calculator is one sum: total VRAM = weights + KV cache + overhead, where weights = params × bits per weight ÷ 8, KV cache comes from the model’s config.json, and overhead is an assumption of 1 to 2 GB. Every worked number below is arithmetic you can repeat by hand, and the page is plain HTML, so it needs no JavaScript. Mac users: the unified-memory version is what fits my RAM, which uses 2 GB, the top of the range here.

How do I calculate VRAM for an LLM?

Use this, following methodology and llm-quantization-explained:

VRAM_GB = (params × bits_per_weight ÷ 8 ÷ 1e9) + (2 × layers × kv_heads × head_dim × context × bytes_per_value ÷ 1e9) + overhead_GB

GB here is bytes ÷ 1e9. The README’s per-model size column is GiB (bytes ÷ 1024³), so do not mix the two.

How can I estimate memory from a GGUF file size?

For a GGUF file, the file size is already the weights term. The README’s Memory/Disk table lists Llama 3.1 8B Q4_K_M at 4.9 GB and 70B Q4_K_M at 43.1 GB (the table’s GB label). Add KV cache and overhead to that file size. The GGUF itself does not hold your context length, so the KV term must still be computed.

What does a worked example look like?

Qwen2.5-32B (card, config.json): 32.5B parameters, 64 layers, 8 KV heads, hidden_size 5120 with 40 attention heads so head_dim = 5120 ÷ 40 = 128, torch_dtype bfloat16 (2 bytes).

• Weights at Q4_K_M: 32.5e9 × 4.8944 ÷ 8 ÷ 1e9 = 19.8835 GB
• KV per token: 2 × 64 × 8 × 128 × 2 = 262,144 bytes
• KV at 4,096 tokens: 262,144 × 4,096 ÷ 1e9 = 1.0737 GB
• Total: 19.8835 + 1.0737 + (1 to 2) = 21.9572 to 22.9572 GB

Results table at default settings

Q4_K_M bits (4.8944), 4,096 context, bf16 cache, overhead 1 to 2 GB.

Qwen3.8-27B (card) lists 27B parameters; the weights use 27B. Its 64 layers are 48 linear-attention and 16 full-attention layers, with 4 KV heads and head dimension 256 in the full-attention layers. The KV figure counts only those 16 layers: 2 × 16 × 4 × 256 × 2 = 65,536 bytes per token. The memory held by the 48 linear-attention layers is unknown here, because neither the card nor the config states it, so that row is a lower bound. Llama 3.1 70B uses the README file size, not a recomputed estimate; see vram-for-70b-model.

What LLM can I run with 16GB VRAM?

A model fits 16 GB when weights + KV + overhead stay under 16 GB, which at Q4_K_M means roughly 20B parameters or fewer at modest context. The table shows 27B and 32B Q4_K_M miss it. The Llama 3.1 8B Q4_K_M file is 4.9 GB per the README, leaving the rest for KV and overhead. OpenAI’s gpt-oss-20b card says the model has 21B parameters (3.6B active) and runs “within 16GB of memory” with MXFP4 MoE weights; a computed bits-per-weight for MXFP4 is unknown here, so use the card’s statement. More: 16gb-vram-llm and gpu-vram-list.

How do MoE and hybrid models change the math?

MoE total parameters still sit in memory even though fewer are active per token. The gpt-oss-120b card lists 117B parameters (5.1B active) and says it fits a single 80GB GPU. The gpt-oss-20b config.json shows 24 layers, 8 KV heads, 64-dimension heads, and alternating sliding-window (128) and full-attention layers, so the plain KV formula overstates its cache; the real size is unknown here. llama.cpp’s README describes “CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity”, covered in moe-cpu-offload and run-llm-on-cpu. Speed after offload is unknown; no figure is published here.

Why do two VRAM calculators give different answers?

They use different inputs, not different arithmetic: round 4.000 bits versus 4.8944, GB versus GiB, whether KV cache is counted, the context length, and the overhead assumed. Overhead differs even between this site (1 to 2 GB) and the Mac page (2 GB). Compare assumptions before comparing totals. See vram-requirements for the by-size tables.

Questions

Can I run this LLM on my GPU?

Compute weights + KV + overhead for the quant and context you want, and compare to your card’s VRAM. If the total is within 1 GB of the card, treat it as not fitting, since overhead is an assumption.

Does context length change VRAM?

Yes. KV cache grows in a straight line with context. For Qwen2.5-32B it is 262,144 bytes per token: 1.0737 GB at 4,096 tokens and 8.5899 GB at 32,768.

Is a Q4 model really 4 bits per weight?

No. The llama.cpp README lists Q4_K_M at 4.8944 bits per weight, so 4.000 bits understates the file by about 18 percent.

What if the model does not fit?

Lower the quant, shorten the context, or offload layers to system RAM. Offloading is slower, and no speed is stated here. See cuda-out-of-memory-llm and ram-vs-vram.

Sources read on 2026-10-13

• llama.cpp quantize README: bits per weight and GiB sizes (Llama 3.1 8B table), Memory/Disk GB table.
• llama.cpp README: hybrid CPU+GPU inference sentence.
• Qwen2.5-32B card and config.json: parameters, layers, heads, hidden size, dtype.
• Qwen3.8-27B card and config.json: 27B, layer layout, KV heads, head dimension.
• gpt-oss-20b card, config.json, and gpt-oss-120b card: parameters, 16GB and 80GB statements, MXFP4, layer and head fields.
• LLM for Mac: what fits my RAM: 2 GB overhead (read from the sister repo’s page source; the live URL returned HTTP 404 on fetch).
• arxiv.org/abs/2311.09735 was suggested but is the GEO paper, not a memory source, so it is not used.

Related

• llm-quantization-explained is the canonical formula.
• vram-requirements applies it by model size.
• methodology is the citation rule.
• gpu-vram-list, 16gb-vram-llm, vram-for-70b-model apply it by card and size.
• ram-vs-vram, moe-cpu-offload, run-llm-on-cpu, cuda-out-of-memory-llm cover what to do when it does not fit.