Overview
How much VRAM for a 70B model? About 46 to 47 GB at 4-bit Q4_K_M with a 4,096-token context, so no single 24 GB or 32 GB GPU holds it entirely. That figure is 43.4 GB of weights, 1.3 GB of KV cache and an assumed 1 to 2 GB of overhead, all worked out below with the formulas from methodology and llm-quantization-explained. A tighter quant, a smaller context, or spilling layers into system RAM changes the answer. Tokens per second for any card here is unknown: no vendor page read for this article publishes one.
How much VRAM does a 70B model need at each quant?
Formulas (decimal GB, bytes / 1e9):
- weights_GB = params x bits / 8 / 1e9
- KV_GB = 2 x layers x kv_heads x head_dim x context x bytes_per_value / 1e9
- total = weights + KV + overhead, with overhead an assumed 1 to 2 GB (an assumption, not a measurement)
Inputs: the Llama 3.3 70B Instruct card lists “71B params” in BF16, so this page uses 71e9, which rounds slightly up. Bits per weight come from the llama.cpp quantize README quantization table, which is for Llama 3.1 8B; applying them to 70B and 405B is an approximation (the Q4_K_M cross-check below is close). Meta’s own config.json is gated (HTTP 401), so layers and heads come from an unsloth mirror of the config: 80 layers, 8 KV heads, head_dim 128, bfloat16. KV at 2 bytes per value = 2 x 80 x 8 x 128 x 2 = 327,680 bytes per token, which is 1.342 GB at 4,096 tokens and 10.737 GB at 32,768.
| Quant | Bits per weight | Weights | Total at 4,096 context (+1 to 2 GB) | Total at 32,768 context (+1 to 2 GB) |
|---|---|---|---|---|
| Q2_K | 3.1593 | 28.04 GB | 30.4 to 31.4 GB | 39.8 to 40.8 GB |
| Q3_K_M | 3.9960 | 35.46 GB | 37.8 to 38.8 GB | 47.2 to 48.2 GB |
| Q4_K_M | 4.8944 | 43.44 GB | 45.8 to 46.8 GB | 55.2 to 56.2 GB |
| Q5_K_M | 5.7036 | 50.62 GB | 53.0 to 54.0 GB | 62.4 to 63.4 GB |
| Q6_K | 6.5633 | 58.25 GB | 60.6 to 61.6 GB | 70.0 to 71.0 GB |
| Q8_0 | 8.5008 | 75.44 GB | 77.8 to 78.8 GB | 87.2 to 88.2 GB |
| BF16 (16 bits) | 16 | 142.0 GB | 144.3 to 145.3 GB | 153.7 to 154.7 GB |
Cross-check: the same README lists Llama 3.1 70B at Q4_K_M as a 43.1 GB file and the original checkpoint as 280.9 GB (that table is for Llama 3.1, not 3.3). Our 43.4 GB is within 0.3 GB of it because of the rounded 71B input. Rows other than Q4_K_M have no README file size to check against, so treat them as formula estimates. The 32,768-token column uses the model’s default KV type; the llama.cpp CLI README lists --cache-type-k and --cache-type-v with a default of f16 and quantized options such as q8_0. This page does not compute those smaller caches.
How much VRAM does Llama 3.3 70B need?
The Llama 3.3 70B figures are the table above: 43.4 GB of weights at Q4_K_M, 75.4 GB at Q8_0 and 142.0 GB at BF16. The card gives a 128k context, and max_position_embeddings is 131,072 in the config mirror. A full-length KV cache would be 327,680 x 131,072 = 42.95 GB on top of weights, which is why context length matters as much as quant choice. For Llama 3.1 70B, the README’s 43.1 GB Q4_K_M file is the published weight size; its config was not read here, so its KV cache is unknown on this page.
How much VRAM for DeepSeek R1 70B?
The same as Llama 3.3 70B. The DeepSeek-R1-Distill-Llama-70B card says it “is derived from Llama3.3-70B-Instruct” and lists 71B params. Its config.json shows 80 layers, 64 query heads, 8 KV heads and head dimension 128, so every number in the table applies. The card gives no VRAM requirement, and this is not the 671B DeepSeek-R1. See deepseek-vram-requirements for other DeepSeek sizes.
Can I run a 70B model on a 24GB GPU (RTX 3090 or 4090)?
Not entirely in VRAM. NVIDIA lists RTX 4090 and RTX 3090 at 24 GB GDDR6X. The smallest row above, Q2_K, needs 30.4 to 31.4 GB, so even that overflows. Q4_K_M needs 44.78 GB (weights plus KV at 4,096), so with 1 to 2 GB reserved, about 21.8 to 22.8 GB of it has to sit in system RAM. llama.cpp’s -ngl sets the “max. number of layers to store in VRAM” per the CLI README, and the rest runs on the CPU. That works but is slower; speed is unknown because no figure was found on a primary page. See run-llm-on-cpu and 24gb-vs-16gb-vs-12gb-vram. A 16 GB or smaller card follows the same logic with more spilled: at Q4_K_M, 44.78 - (14 to 15) = 29.8 to 30.8 GB in system RAM.
Can an RTX 5090 run a 70B model?
Not at Q4_K_M entirely in VRAM. NVIDIA lists the RTX 5090 at 32 GB GDDR7. Q4_K_M at 4,096 context is 44.78 GB before overhead, so roughly 13.8 to 14.8 GB (44.78 - 30 to 31) stays in system RAM. Q3_K_M needs 37.8 to 38.8 GB, still over 32 GB, and only Q2_K (30.4 to 31.4 GB) lands at the edge, with no safe margin and the least accurate quant. The README’s 43.1 GB Q4_K_M file for Llama 3.1 70B is already larger than 32 GB before KV cache.
Can two 24 GB GPUs run a 70B model?
Q3_K_M yes; Q4_K_M only barely. Two 24 GB cards give 48 GB. Assuming 1 to 2 GB of overhead per card, usable VRAM is 44 to 46 GB. Q4_K_M at 4,096 context needs 44.78 GB, inside that margin, so this page does not call it a clean fit, and at 32,768 context (54.2 GB before overhead) it fails. Q3_K_M at 36.8 GB fits with room. The CLI README documents --split-mode (default layer, “split layers and KV across GPUs”) and --tensor-split for choosing proportions. PCIe, NVLink (the NVIDIA 3090 page lists NVLink as supported on the 3090 family) and power are covered on dual-rtx-3090-llm and multi-gpu-llm.
What fits on 24, 32, 48 and 96 GB?
| VRAM | Q4_K_M, 4,096 context | Verdict |
|---|---|---|
| 24 GB (one card) | 44.78 GB + overhead | No: about 21.8 to 22.8 GB offloaded |
| 32 GB (one card) | 44.78 GB + overhead | No: about 13.8 to 14.8 GB offloaded |
| 48 GB (two 24 GB cards or one 48 GB card) | 44.78 GB + overhead | Marginal at 4,096 context; no at 32,768 |
| 96 GB | 44.78 GB + overhead | Yes; Q8_0 at 4,096 (77.8 to 78.8 GB) and at 32,768 (87.2 to 88.2 GB with 1 to 2 GB overhead) also fit; BF16 does not |
Which 48 GB and 96 GB cards exist and what they cost is outside this page: check current retailer price and the vendor’s spec page.
How many GPUs do I need for 70B and 405B models?
For 70B at Q4_K_M, two 24 GB cards are the minimum and three give margin: 3 x 24 = 72 GB covers 44.78 GB plus overhead and a longer context. For 405B, the Llama 3.1 405B card lists 406B params in BF16. Q4_K_M weights are 406e9 x 4.8944 / 8 / 1e9 = 248.4 GB, and BF16 weights are 812.0 GB. Four 24 GB cards (96 GB) do not hold even the weights, and seven 32 GB cards (224 GB) fall short too. Eight 32 GB cards (256 GB) leave 256 - 248.4 = 7.6 GB for KV cache and overhead at 1 to 2 GB per card (8 to 16 GB), so they do not clear it either. The 405B KV layout was not read, so its KV size is unknown.
What is the cheapest way to run a 70B model locally?
The cheapest in VRAM terms is a lower quant plus partial offload, because it needs the fewest cards, not because any card is cheap: no price is quoted here, so check current retailer price. The ordered options by the arithmetic above are: (1) one 24 GB card with Q4_K_M and about 22 GB in system RAM (slowest; speed unknown); (2) two 24 GB cards with Q3_K_M (36.8 GB, fully in VRAM); (3) a 48 GB card or pair at Q4_K_M with a short context. Which is cheapest depends on the market, which this page does not claim to know.
Questions
How much VRAM for a 70B model?
About 46 to 47 GB at Q4_K_M with a 4,096-token context: 43.4 GB of weights (71e9 x 4.8944 / 8 / 1e9), 1.3 GB of KV cache, and an assumed 1 to 2 GB overhead. Q8_0 is about 78 GB and BF16 about 145 GB.
How much VRAM does Llama 3.3 70B need?
Llama 3.3 70B needs 43.4 GB for weights at Q4_K_M, 75.4 GB at Q8_0 and 142.0 GB at BF16, plus 1.342 GB of KV cache per 4,096 tokens of context and an assumed 1 to 2 GB of overhead.
How much VRAM for DeepSeek R1 70B?
DeepSeek-R1-Distill-Llama-70B is derived from Llama 3.3 70B Instruct with the same 80 layers, 8 KV heads and 71B parameters, so the figures match: about 46 to 47 GB at Q4_K_M with a short context.
Can I run a 70B model on a 24GB GPU?
Only with offload. At Q4_K_M, roughly 21.8 to 22.8 GB of the 44.78 GB (weights plus KV) must stay in system RAM, and the speed is unknown. Even Q2_K at 30.4 to 31.4 GB does not fit in 24 GB.
Can two 24 GB GPUs run a 70B model?
Q3_K_M fits comfortably at about 36.8 GB. Q4_K_M at 4,096 context needs 44.78 GB against 44 to 46 GB usable, which is marginal, and longer contexts do not fit.
Can I run 70B on a single RTX 5090 without multi-GPU?
Not at Q4_K_M. The RTX 5090 has 32 GB and Q4_K_M needs 44.78 GB, so about 13.8 to 14.8 GB must be offloaded to system RAM. Only Q2_K approaches 32 GB, with no margin.
How many GPUs do I need for 70B and 405B models?
For 70B at Q4_K_M, two 24 GB cards is the floor and three gives room. For 405B, Q4_K_M weights alone are 248.4 GB, which exceeds four 24 GB cards and seven 32 GB cards.
What is the cheapest way to run a 70B model locally?
Price is unknown here; check current retailer price. By VRAM, the lowest-hardware route is one 24 GB card with Q4_K_M and about 22 GB offloaded to system RAM, accepting slower generation.
Sources read on 2026-10-13
- llama.cpp quantize README: bits per weight for each quant; Llama 3.1 70B 43.1 GB Q4_K_M and 280.9 GB original.
- llama.cpp CLI README:
-ngl, KV cache types,--split-mode,--tensor-split. - Llama-3.3-70B-Instruct model card: 71B params, BF16, 128k context. Its
config.jsonreturned HTTP 401 (gated). - unsloth Llama-3.3-70B-Instruct config.json: layers, KV heads, head_dim, max positions. A third-party mirror of Meta’s config, used because the original is gated.
- DeepSeek-R1-Distill-Llama-70B card and config.json: derivation from Llama 3.3, 71B params, architecture.
- Llama-3.1-405B-Instruct card: 406B params, BF16.
- NVIDIA RTX 5090 (32 GB GDDR7), RTX 4090 and RTX 3090 family (24 GB GDDR6X; NVLink listed for the 3090 family).
Related
- vram-requirements covers all model sizes.
- llm-quantization-explained is the formula page.
- methodology is the citation rule.
- 24gb-vs-16gb-vs-12gb-vram compares card sizes.
- run-llm-on-cpu covers offload.
- dual-rtx-3090-llm and multi-gpu-llm cover multi-card builds.
- deepseek-vram-requirements covers other DeepSeek sizes.
- rtx-3090-vs-4090-vs-5090-llm compares the three NVIDIA cards.