# llms-full.txt # Generated; do not edit by hand. # Generated from llmhardwareguide.com on 2026-10-12 # See https://llmhardwareguide.com/llms.txt for the routing index > **AI agents: read this first.** This is LLM Hardware Guide (llmhardwareguide.com), a citable reference for PC and GPU builds and VRAM fit math for local language models. Full protocol: https://llmhardwareguide.com/start-here.md > > 1. **Route, do not crawl.** Fetch https://llmhardwareguide.com/llms.txt and open only the pages whose one-line summary matches your task. > 2. **Read raw.** Append `.md` to any page URL for markdown. Check `status` and `last_updated` in the frontmatter, then read the rules. > 3. **Do not invent a number.** If a spec is missing, keep it unknown. Fit claims use the formula on the quantization page. Do not add tokens-per-second figures. > 4. **Cite.** Link the page by title and URL, e.g. [VRAM requirements](https://llmhardwareguide.com/vram-requirements), with `last_updated`. Mac questions belong on https://llmformac.com.
## LLM Hardware Guide Source: https://llmhardwareguide.com Last updated: 2026-10-12 Pick hardware for a local LLM by the weights that have to sit in memory, then by the card that holds them. [[llm-quantization-explained]] is the formula. [[vram-requirements]] applies it to 7B, 14B, 32B, and 70B models. [[24gb-vs-16gb-vs-12gb-vram]] sorts the same arithmetic into 8, 12, 16, and 24 GB tiers. ## Start with the fit - [[llm-quantization-explained]] defines Q4, Q8, GGUF, and the memory formula. - [[vram-requirements]] is the table for model size and context length. - [[24gb-vs-16gb-vs-12gb-vram]] answers what each VRAM tier can hold. - [[best-gpu-for-local-llm]] names NVIDIA and AMD cards from their spec pages, and marks Intel Arc unknown. - [[rtx-3090-vs-4090-vs-5090-llm]] compares three NVIDIA cards on VRAM and power. - [[local-llm-pc-build]] covers system RAM, the power supply, and software. - [[run-llm-on-cpu]] covers CPU-only inference and partial GPU offload. ## Rules and machine-readable pages [[methodology]] is the citation rule: no invented tokens-per-second numbers, and unknown stays unknown. [[about]] says what this site covers and what [llmformac.com](https://llmformac.com) covers. [[llm-info]] is the page for AI assistants. [[start-here]] is the short reading protocol. [[identity]] is the brand token page. Mac questions, including unified memory, belong on the sister site: [how much RAM for a local LLM](https://llmformac.com/how-much-ram-for-local-llm/) and [what fits my RAM](https://llmformac.com/what-fits-my-ram/).
## 24GB vs 16GB vs 12GB VRAM for Local LLMs: What Fits Source: https://llmhardwareguide.com/24gb-vs-16gb-vs-12gb-vram Last updated: 2026-10-12 ## Overview A VRAM tier is a budget of bytes. Whether a model fits is the weight estimate or the published file size, plus KV cache, plus an assumed 1 to 2 GB of overhead, compared with that budget. The formula is on [[llm-quantization-explained]].

What can you run with 16GB of VRAM?

Cards at 16 GB on pages read 2026-10-12: the GeForce RTX 4060 Ti is sold as 16 GB GDDR6 or 8 GB GDDR6, and the Radeon RX 9070 XT is 16 GB GDDR6. See [[best-gpu-for-local-llm]] for power and interface. Qwen2.5-14B at Q4_K_M is 8.9935 decimal GB of weights. KV at 4,096 tokens is 0.8053 GB. Plus the 1 to 2 GB assumption: 10.7988 to 11.7988 GB, which is under 16 GB. At 32,768 tokens the KV term is 6.4425 GB, so the total is 16.4359 GB plus overhead, which is over 16 GB. Qwen2.5-14B at Q8_0 is 15.6202 GB of weights before KV. That is under 16 GB and leaves less than 1 GB, so the 4,096-token KV term (0.8053 GB) already passes 16 GB before overhead. Qwen2.5-32B at Q4_K_M is 19.8835 GB of weights, which is over 16 GB before KV.

Is 24GB of VRAM enough for local LLMs?

24 GB cards on the fetched vendor pages: GeForce RTX 3090 (24 GB GDDR6X), GeForce RTX 4090 (24 GB GDDR6X), and Radeon RX 7900 XTX (24 GB GDDR6). The RTX 5090 is 32 GB, not 24 GB. Comparison of the three NVIDIA cards is [[rtx-3090-vs-4090-vs-5090-llm]]. Qwen2.5-32B Q4_K_M weights are 19.8835 GB. KV at 4,096 is 1.0737 GB. Plus 1 to 2 GB assumed overhead: 21.9572 to 22.9572 GB, under 24 GB. At 32,768 tokens KV is 8.5899 GB, and 19.8835 + 8.5899 + 1 = 29.4734 GB, over 24 GB. Llama 3.1 70B Q4_K_M is a 43.1 GB file in the quantize README. Qwen2.5-72B Q4_K_M weights estimate to 44.4779 GB. Neither fits in 24 GB of VRAM on weights alone. 24 GB is not a 70B Q4_K_M card.

What can you run with 12GB or 8GB of VRAM?

No NVIDIA or AMD page fetched on 2026-10-12 listed a 12 GB card. Intel Arc stayed unknown after HTTP 403. 12 GB below is a byte budget, not a named card. 8 GB cards that are named: GeForce RTX 4060 is 8 GB GDDR6, and the RTX 4060 Ti has an 8 GB GDDR6 option as well as 16 GB. Qwen2.5-7B Q4_K_M weights are 4.6558 GB. KV at 4,096 is 0.2349 GB. Plus 1 to 2 GB: 5.8907 to 6.8907 GB, under 8 GB. At 32,768 tokens KV is 1.8790 GB, so 4.6558 + 1.8790 + 2 = 8.5348 GB, which is over 8 GB if you take the high end of the overhead assumption. Qwen2.5-7B Q8_0 weights are 8.0864 GB, already over 8 GB before KV. The same 7B Q4_K_M line fits a 12 GB budget with more room at 4,096 (5.8907 to 6.8907 GB). Qwen2.5-14B Q4_K_M at 4,096 is 10.7988 to 11.7988 GB including the overhead assumption, under 12 GB at the low end and still under 12 GB at 2 GB of overhead (11.7988). At 32,768 tokens that 14B line is over 16 GB, as calculated above, so it is also over 12 GB.

Is moving up a VRAM tier worth it?

Move up when the next model you want crosses the byte budget, not because a larger card is newer. 8 GB holds the 7.61B Q4_K_M estimate at a short context and does not hold that model's Q8_0 weights. 16 GB holds the 14.7B Q4_K_M estimate at 4,096 tokens and does not hold 32.5B Q4_K_M weights. 24 GB holds the 32.5B Q4_K_M estimate at 4,096 tokens and does not hold a 43.1 GB Llama 3.1 70B Q4_K_M file. The 32 GB RTX 5090 still does not hold that 43.1 GB file. This page does not convert those gaps into a speed ranking. Sister page for a Mac of similar memory: [best local LLM for Mac](https://llmformac.com/best-local-llm-for-mac/). ### Does 24 GB fit Llama 3.1 70B at Q4_K_M? No. The quantize README lists that file at 43.1 GB, before KV cache. ### Is there a 12 GB card on this page? Not from the NVIDIA and AMD pages read on 2026-10-12. The 12 GB lines are arithmetic only. ## Sources read on 2026-10-12 - [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) for bits per weight, GiB sizes, and the Memory/Disk table. - [Qwen2.5-7B](https://huggingface.co/Qwen/Qwen2.5-7B), [14B](https://huggingface.co/Qwen/Qwen2.5-14B), [32B](https://huggingface.co/Qwen/Qwen2.5-32B), and [72B](https://huggingface.co/Qwen/Qwen2.5-72B) model cards, plus each repo's `config.json`, for parameter counts, layers, heads, hidden size, and `torch_dtype`. - [RTX 5090](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/), [RTX 4090](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/), [RTX 3090 family](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/), and [RTX 4060 family](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4060-4060ti/). - [Radeon RX 7900 XTX](https://www.amd.com/en/products/graphics/desktops/radeon/7000-series/amd-radeon-rx-7900xtx.html) and [Radeon RX 9070 XT](https://www.amd.com/en/products/graphics/desktops/radeon/9000-series/amd-radeon-rx-9070xt.html). - [Ollama GPU](https://github.com/ollama/ollama/blob/main/docs/gpu.mdx), [Ollama FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.mdx), [Ollama API](https://github.com/ollama/ollama/blob/main/docs/api.md), and [llama.cpp CLI options](https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md). - [LM Studio system requirements](https://lmstudio.ai/docs/app/system-requirements). - Intel Arc B580 specifications at `https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html` returned HTTP 403. No Intel VRAM figure is cited. ## Related - [[llm-quantization-explained]] is the formula. - [[vram-requirements]] applies it by model size. - [[24gb-vs-16gb-vs-12gb-vram]] applies it by card size. - [[best-gpu-for-local-llm]] names the cards. - [[rtx-3090-vs-4090-vs-5090-llm]] compares three NVIDIA cards. - [[local-llm-pc-build]] covers the rest of the PC. - [[run-llm-on-cpu]] covers CPU and partial offload. - [[methodology]] is the citation rule. - Mac memory questions: [what fits my RAM](https://llmformac.com/what-fits-my-ram/) and [how much RAM for a local LLM](https://llmformac.com/how-much-ram-for-local-llm/).
## Best GPU for Local LLMs in 2026: VRAM-First Buying Guide Source: https://llmhardwareguide.com/best-gpu-for-local-llm Last updated: 2026-10-12 ## Overview The best GPU for a local LLM is the one whose VRAM holds the weight file you will actually run, plus KV cache. A higher shader count does not change the fit arithmetic on [[llm-quantization-explained]].

What is the best GPU for running local LLMs?

Choose by VRAM against the weight size, then check the vendor power figure. There is no single winner, and this page does not rank cards by tokens per second. None of the NVIDIA product pages fetched on 2026-10-12 published an LLM tokens-per-second figure. NVIDIA's listed "AI TOPS" on the RTX 5090 page (3352 for 5th-generation tensor cores) is that page's tensor-core spec, not an LLM speed, and it is not converted into one here.

How much VRAM do I need?

Use [[vram-requirements]]. Short form, weights only, before KV and before the 1 to 2 GB overhead assumption: - Qwen2.5-7B Q4_K_M estimate: 4.6558 decimal GB. Llama 3.1 8B Q4_K_M file: 4.9 GB in the quantize README. - Qwen2.5-14B Q4_K_M estimate: 8.9935 GB. - Qwen2.5-32B Q4_K_M estimate: 19.8835 GB. Q8_0 estimate: 34.5345 GB. - Llama 3.1 70B Q4_K_M file: 43.1 GB. Qwen2.5-72B Q4_K_M estimate: 44.4779 GB.

NVIDIA, AMD or Intel for local LLMs?

NVIDIA and AMD figures below are from the vendor pages read on 2026-10-12. Intel Arc is unknown. `https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html` returned HTTP 403 that day, so this page does not state an Arc VRAM size. Ollama's GPU document says NVIDIA GPUs can be limited with `CUDA_VISIBLE_DEVICES`, lists AMD cards it supports through ROCm (including Radeon RX 7900 XTX and RX 9070 XT), and says further AMD support comes from Vulkan. That is a software-support statement from Ollama, not a speed claim. ### RTX 5090 [NVIDIA's RTX 5090 page](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/): 32 GB GDDR7, 512-bit interface, 21760 CUDA cores, boost clock 2.41 GHz, total graphics power 575 W, required system power 1000 W on a PC configured with a Ryzen 9 9950X. The same page's prepare section also says "850W Minimum" and that a higher rating may be required. NVLink is No. Card length 304 mm, width 137 mm, 2-slot. Memory bandwidth is not listed on the page fetched that day, so bandwidth is unknown. Check current retailer price. ### RTX 4090 [NVIDIA's RTX 4090 page](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/): 24 GB GDDR6X, 384-bit interface, 16384 CUDA cores, boost clock 2.52 GHz, total graphics power 450 W, required system power 850 W on a Ryzen 9 5900X. Length 304 mm, width 137 mm, 3-slot. NVLink is No. Average gaming power 315 W is measured across 22 games, not an LLM workload. Memory bandwidth is not on the fetched page, so it is unknown. Check current retailer price. ### RTX 3090 [NVIDIA's RTX 3090 family page](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/), GeForce RTX 3090 column (not the 3090 Ti column): 24 GB GDDR6X, 384-bit, 10496 CUDA cores, boost clock 1.70 GHz, graphics card power 350 W, required system power 750 W on a Core i9-10900K. The footnote says a lower rating may work depending on the system. NVLink (SLI-Ready) is Yes. Length 313 mm, 3-slot. Memory bandwidth is not in that table, so it is unknown. ### Radeon RX 7900 XTX [AMD's RX 7900 XTX page](https://www.amd.com/en/products/graphics/desktops/radeon/7000-series/amd-radeon-rx-7900xtx.html): 24 GB GDDR6, memory bandwidth up to 960 GB/s, and a separate line "effective memory bandwidth" up to 3500 GB/s. Infinity Cache 96 MB. 96 compute units. Typical board power 355 W. Minimum PSU recommendation 800 W, based on a Ryzen 9 5900X (AMD note GD-223). Memory interface width is not on the fetched spec list, so it is unknown. ### Radeon RX 9070 XT [AMD's RX 9070 XT page](https://www.amd.com/en/products/graphics/desktops/radeon/9000-series/amd-radeon-rx-9070xt.html): 16 GB GDDR6, 256-bit interface, memory bandwidth up to 640 GB/s, Infinity Cache 64 MB, 4096 stream processors, memory speed up to 20 Gbps. Typical board power 304 W. Minimum PSU recommendation 750 W. ### RTX 4060 Ti and RTX 4060 [NVIDIA's RTX 4060 family page](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4060-4060ti/): the 4060 Ti column is 16 GB GDDR6 or 8 GB GDDR6, 128-bit, 4352 CUDA cores, total graphics power 165 W or 160 W. The 4060 column is 8 GB GDDR6, 128-bit, 3072 CUDA cores, total graphics power 115 W. Required system power is 550 W for both, based on a Ryzen 9 5900X. Memory bandwidth is not in that table, so it is unknown.

Is one big GPU better than two smaller ones?

One GPU avoids a split. Ollama's FAQ says that if a model fits entirely on any single GPU, Ollama loads it on that GPU, and if it does not, Ollama spreads it across the available GPUs. llama.cpp's CLI README documents `--split-mode`: `none` uses one GPU, `layer` (the default) splits layers and KV across GPUs, and `row` splits weights across GPUs by rows. `--tensor-split` takes a comma-separated list of proportions. Two 24 GB cards are 48 GB of VRAM only if the runtime can use both. Llama 3.1 70B Q4_K_M is a 43.1 GB file, which leaves 48 − 43.1 = 4.9 GB for KV cache and overhead. That margin is real as subtraction and still depends on the split actually working. This page does not claim one layout is faster. The Mac alternative is [which Mac for a local LLM](https://llmformac.com/which-mac-for-local-llm/). ### RTX 5090 32 GB GDDR7. See the vendor section above. Weight fit is on [[vram-requirements]] and [[rtx-3090-vs-4090-vs-5090-llm]]. ### RTX 4090 24 GB GDDR6X. ### RTX 3090 24 GB GDDR6X, with NVLink listed as yes on NVIDIA's family page. ### Radeon RX 7900 XTX 24 GB GDDR6, bandwidth up to 960 GB/s on AMD's page. ### Radeon RX 9070 XT 16 GB GDDR6, bandwidth up to 640 GB/s on AMD's page. ### RTX 4060 Ti 16 GB or 8 GB GDDR6. ### RTX 4060 8 GB GDDR6. ## Sources read on 2026-10-12 - [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) for bits per weight, GiB sizes, and the Memory/Disk table. - [Qwen2.5-7B](https://huggingface.co/Qwen/Qwen2.5-7B), [14B](https://huggingface.co/Qwen/Qwen2.5-14B), [32B](https://huggingface.co/Qwen/Qwen2.5-32B), and [72B](https://huggingface.co/Qwen/Qwen2.5-72B) model cards, plus each repo's `config.json`, for parameter counts, layers, heads, hidden size, and `torch_dtype`. - [RTX 5090](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/), [RTX 4090](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/), [RTX 3090 family](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/), and [RTX 4060 family](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4060-4060ti/). - [Radeon RX 7900 XTX](https://www.amd.com/en/products/graphics/desktops/radeon/7000-series/amd-radeon-rx-7900xtx.html) and [Radeon RX 9070 XT](https://www.amd.com/en/products/graphics/desktops/radeon/9000-series/amd-radeon-rx-9070xt.html). - [Ollama GPU](https://github.com/ollama/ollama/blob/main/docs/gpu.mdx), [Ollama FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.mdx), [Ollama API](https://github.com/ollama/ollama/blob/main/docs/api.md), and [llama.cpp CLI options](https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md). - [LM Studio system requirements](https://lmstudio.ai/docs/app/system-requirements). - Intel Arc B580 specifications at `https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html` returned HTTP 403. No Intel VRAM figure is cited. ## Related - [[llm-quantization-explained]] is the formula. - [[vram-requirements]] applies it by model size. - [[24gb-vs-16gb-vs-12gb-vram]] applies it by card size. - [[best-gpu-for-local-llm]] names the cards. - [[rtx-3090-vs-4090-vs-5090-llm]] compares three NVIDIA cards. - [[local-llm-pc-build]] covers the rest of the PC. - [[run-llm-on-cpu]] covers CPU and partial offload. - [[methodology]] is the citation rule. - Mac memory questions: [what fits my RAM](https://llmformac.com/what-fits-my-ram/) and [how much RAM for a local LLM](https://llmformac.com/how-much-ram-for-local-llm/).
## LLM Quantization Explained: Q4 vs Q8, GGUF and Memory Math Source: https://llmhardwareguide.com/llm-quantization-explained Last updated: 2026-10-12 ## Overview LLM quantization stores weights with fewer bits than the training checkpoint, and the size follows from the parameter count and the bits per weight. This page is the formula the rest of the site cites.

What is LLM quantization?

LLM quantization stores a model's weights with fewer bits (for example 4 or 8 instead of 16), which shrinks the file roughly in proportion to the bit width. The [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md), read on 2026-10-12, says the tool takes a high-precision GGUF file, typically F32 or BF16, and converts it to a quantized format. It also says the process may introduce accuracy loss, usually measured as perplexity or KL divergence, and that an importance matrix can reduce that loss. The same README's Memory/Disk table gives Llama 3.1 file sizes in GB (the table's label, not GiB): | Model | Original size | Q4_K_M size | | --- | ---: | ---: | | 8B | 32.1 GB | 4.9 GB | | 70B | 280.9 GB | 43.1 GB | | 405B | 1,625.1 GB | 249.1 GB |

What is the difference between Q4 and Q8?

In that README's Llama 3.1 8B table, Q4_K_M is 4.8944 bits per weight and 4.58 GiB, and Q8_0 is 8.5008 bits per weight and 7.95 GiB, so the Q4_K_M file is a bit over half the Q8_0 file (4.58 / 7.95). A round "4-bit" estimate is not the same number as Q4_K_M. Selected rows from the same table. Speed columns are omitted here because the README does not name the test machine. | Type | bits/weight | size (GiB) | | --- | ---: | ---: | | Q4_K_S | 4.6672 | 4.36 | | Q4_K_M | 4.8944 | 4.58 | | Q5_K_S | 5.5704 | 5.21 | | Q5_K_M | 5.7036 | 5.33 | | Q6_K | 6.5633 | 6.14 | | Q8_0 | 8.5008 | 7.95 | | IQ4_NL | 4.6818 | 4.38 | | F16 | 16.0005 | 14.96 | The Memory/Disk table's 4.9 GB for Llama 3.1 8B Q4_K_M and this table's 4.58 GiB are the same file described two ways. 4.58 GiB is 4.58 × 1024³ bytes, which is about 4.92 decimal GB, and the other table rounds that to 4.9 GB.

How do I calculate a quantized model's size?

Multiply the parameter count by the bits per weight and divide by 8 to get bytes. Divide by 1,000,000,000 for decimal GB. Divide by 1024³ for GiB. `weights_bytes = params × bits_per_weight ÷ 8` `weights_GB = weights_bytes ÷ 1e9` The bits per weight below are the README's Q4_K_M value 4.8944 and Q8_0 value 8.5008, applied to parameter counts on the Qwen2.5 model cards (read 2026-10-12). This is an estimate of weight bytes. It is not a measured GGUF file size. The README's own file sizes are the rows above, and they are the figures to use for Llama 3.1. | Model card | Params | Round 4-bit (params × 4 ÷ 8) | Q4_K_M at 4.8944 bpw | Q8_0 at 8.5008 bpw | | --- | ---: | ---: | ---: | ---: | | Qwen2.5-7B | 7.61B | 3.8050 GB | 4.6558 GB | 8.0864 GB | | Qwen2.5-14B | 14.7B | 7.3500 GB | 8.9935 GB | 15.6202 GB | | Qwen2.5-32B | 32.5B | 16.2500 GB | 19.8835 GB | 34.5345 GB | | Qwen2.5-72B | 72.7B | 36.3500 GB | 44.4779 GB | 77.2510 GB | Worked line for the 7B card at Q4_K_M: 7.61e9 × 4.8944 ÷ 8 ÷ 1e9 = 4.6558 decimal GB. The round 4-bit line 7.61e9 × 4 ÷ 8 ÷ 1e9 = 3.8050 GB understates Q4_K_M because 4.8944 is not 4. KV cache is separate. For a decoder with separate key and value tensors: `kv_bytes = 2 × layers × kv_heads × head_dim × context × bytes_per_value` `head_dim = hidden_size ÷ num_attention_heads` Qwen2.5-7B `config.json` lists `hidden_size` 3584, `num_attention_heads` 28, `num_key_value_heads` 4, `num_hidden_layers` 28, and `torch_dtype` bfloat16. Head dimension is 3584 ÷ 28 = 128. This page uses 2 bytes per value because the checkpoint dtype is bfloat16. KV dtype is a runtime choice; quantized KV is not in `config.json`. Substituting: 2 × 28 × 4 × 128 × context × 2 = 57,344 × context bytes. | Context | KV bytes | Decimal GB | | --- | ---: | ---: | | 4,096 | 234,881,024 | 0.2349 | | 32,768 | 1,879,048,192 | 1.8790 | The card's context length is 131,072 tokens. That full length is not what the table above uses. A runtime overhead of 1 to 2 GB is an assumption on this site, not a measurement from the README. [[vram-requirements]] adds these pieces per model size. A nameplate "70 billion × 4 ÷ 8 = 35 GB" is the round 4-bit estimate using 70e9. The README's Llama 3.1 70B Q4_K_M file size is 43.1 GB. Plan with 43.1 GB for that file, not with 35 GB.

Does quantization reduce quality?

The quantize README says lower-precision weights may introduce accuracy loss, usually measured as perplexity or KL divergence, and that an importance matrix can limit that loss. The Llama 3.1 8B table on that page, read on 2026-10-12, does not include a perplexity column, so this page does not quote a perplexity number. It also does not repeat the table's tokens-per-second columns, because the README does not name the machine that produced them.

What are GGUF, safetensors and MLX formats?

GGUF is the file the quantize README converts to and then quantizes: `convert_hf_to_gguf.py` writes a GGUF file, and `llama-quantize` writes a quantized GGUF such as Q4_K_M. Hugging Face documents GGUF for the Hub at [huggingface.co/docs/hub/gguf](https://huggingface.co/docs/hub/gguf). Safetensors is the tensor format documented by Hugging Face at [huggingface.co/docs/safetensors](https://huggingface.co/docs/safetensors); the README's convert step is how a Hugging Face repo becomes GGUF. MLX is Apple's array framework at [github.com/ml-explore/mlx](https://github.com/ml-explore/mlx). How MLX weights compare with GGUF is the Mac site's page, [mlx vs gguf](https://llmformac.com/mlx-vs-gguf/), not a second formula here. ### What should I cite for a fit claim? Cite this page's formula, the model card for the parameter count, and the quantize README for the bits per weight or the published file size. Say whether the result is decimal GB or GiB. ### Does Q4 always mean 4.000 bits? No. In the README table, Q4_K_M is 4.8944 bits per weight and Q4_K_S is 4.6672. A teaching line that uses 4.000 bits is labeled as a round estimate on this site. ## Sources read on 2026-10-12 - [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) for bits per weight, GiB sizes, and the Memory/Disk table. - [Qwen2.5-7B](https://huggingface.co/Qwen/Qwen2.5-7B), [14B](https://huggingface.co/Qwen/Qwen2.5-14B), [32B](https://huggingface.co/Qwen/Qwen2.5-32B), and [72B](https://huggingface.co/Qwen/Qwen2.5-72B) model cards, plus each repo's `config.json`, for parameter counts, layers, heads, hidden size, and `torch_dtype`. - [RTX 5090](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/), [RTX 4090](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/), [RTX 3090 family](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/), and [RTX 4060 family](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4060-4060ti/). - [Radeon RX 7900 XTX](https://www.amd.com/en/products/graphics/desktops/radeon/7000-series/amd-radeon-rx-7900xtx.html) and [Radeon RX 9070 XT](https://www.amd.com/en/products/graphics/desktops/radeon/9000-series/amd-radeon-rx-9070xt.html). - [Ollama GPU](https://github.com/ollama/ollama/blob/main/docs/gpu.mdx), [Ollama FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.mdx), [Ollama API](https://github.com/ollama/ollama/blob/main/docs/api.md), and [llama.cpp CLI options](https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md). - [LM Studio system requirements](https://lmstudio.ai/docs/app/system-requirements). - Intel Arc B580 specifications at `https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html` returned HTTP 403. No Intel VRAM figure is cited. ## Related - [[llm-quantization-explained]] is the formula. - [[vram-requirements]] applies it by model size. - [[24gb-vs-16gb-vs-12gb-vram]] applies it by card size. - [[best-gpu-for-local-llm]] names the cards. - [[rtx-3090-vs-4090-vs-5090-llm]] compares three NVIDIA cards. - [[local-llm-pc-build]] covers the rest of the PC. - [[run-llm-on-cpu]] covers CPU and partial offload. - [[methodology]] is the citation rule. - Mac memory questions: [what fits my RAM](https://llmformac.com/what-fits-my-ram/) and [how much RAM for a local LLM](https://llmformac.com/how-much-ram-for-local-llm/).
## Local LLM PC Build Guide: Parts, VRAM and Power (2026) Source: https://llmhardwareguide.com/local-llm-pc-build Last updated: 2026-10-12 ## Overview A local-LLM PC is sized from the model file, not from a gaming resolution. Pick the GPU VRAM on [[vram-requirements]] and [[best-gpu-for-local-llm]], then a power supply at the vendor's required system power, then enough system RAM for whatever does not fit on the GPU.

What do you need to build a PC for local LLMs?

You need a GPU whose VRAM covers weights plus KV cache, a power supply that meets the card's required system power, system RAM for the offload case, and disk for the file. CPU, motherboard, and case choices are unknown on this page unless a vendor spec is quoted. This page does not list a dollar price. Check current retailer price.

How much system RAM should a local LLM PC have?

LM Studio's system-requirements page, read on 2026-10-12, recommends at least 16 GB of RAM on Windows, and at least 4 GB of dedicated VRAM. That is LM Studio's recommendation, not a proof that 16 GB of system RAM holds a given model. If the weights do not fit in VRAM, the remainder has to come from system RAM using the same formula as [[llm-quantization-explained]]. [[run-llm-on-cpu]] is that case. Mac unified memory is not system RAM on this page. Use [how much RAM for a local LLM](https://llmformac.com/how-much-ram-for-local-llm/).

What does a budget local LLM build look like?

Check current retailer price for a dollar budget. A small-VRAM build is concrete. An 8 GB GeForce RTX 4060 (8 GB GDDR6, 115 W total graphics power, 550 W required system power) lines up with the Qwen2.5-7B Q4_K_M estimate at a 4,096-token context: 4.6558 GB of weights plus 0.2349 GB of KV plus an assumed 1 to 2 GB, or 5.8907 to 6.8907 GB. The same card does not hold that model's Q8_0 weights (8.0864 GB) before KV. A 16 GB step is the RTX 4060 Ti 16 GB option or the Radeon RX 9070 XT (16 GB, 304 W typical board power, 750 W minimum PSU on AMD's page). What those tiers hold is [[24gb-vs-16gb-vs-12gb-vram]].

How do I build for 70B models?

Llama 3.1 70B Q4_K_M is 43.1 GB in the quantize README. One 24 GB card is short by the weight file alone. One RTX 5090 is 32 GB and is also short of 43.1 GB. Two 24 GB cards are 48 GB of VRAM if the software can use both, which leaves 48 − 43.1 = 4.9 GB for KV cache and overhead. llama.cpp can split with `--split-mode layer` (the default) or `--tensor-split`. Ollama's FAQ says a model that does not fit on one GPU is spread across the available GPUs. Qwen2.5-72B Q4_K_M weights estimate to 44.4779 GB, which leaves only 48 − 44.4779 = 3.5221 GB on a pair of 24 GB cards before KV. At 4,096 tokens that model's KV term is 1.3422 GB, so the remainder is tight once the 1 to 2 GB overhead assumption is added. This is arithmetic, not a measured two-GPU run. The power supply has to cover the cards you install. One 5090 lists 1000 W required system power and, separately, "850W Minimum" in the prepare section. One 4090 lists 850 W. One 3090 lists 750 W. Two cards are not a figure NVIDIA published as a pair on those pages, so a dual-card PSU total is unknown here beyond "at least each card's own requirement, and the vendor did not publish a combined number."

Which OS and software should I install?

LM Studio documents Windows on x64 and ARM (Snapdragon X Elite; AVX2 required on x64), Linux as an AppImage (Ubuntu 20.04 or newer), and macOS only on Apple silicon. The Mac install belongs on [run an LLM on a Mac](https://llmformac.com/guides/run-llm-on-mac/). Ollama's GPU document covers NVIDIA selection, an AMD ROCm list that includes the RX 7900 XTX and RX 9070 XT, and Vulkan. llama.cpp's CLI README is the source for `--n-gpu-layers`. This page does not turn those docs into a benchmark. Motherboard lane counts are unknown until a specific board's spec page is cited. The GPU pages do say the link generation: RTX 5090 lists PCI Express Gen 5, and the RTX 4090, RTX 3090, and RTX 4060 pages list PCI Express Gen 4. Disk: the quantize README says that while you are converting, memory and disk requirements match, and it lists Llama 3.1 70B original size as 280.9 GB and Q4_K_M as 43.1 GB. A finished Q4_K_M file is the smaller number. The conversion needs room for the original as well. ## Sources read on 2026-10-12 - [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) for bits per weight, GiB sizes, and the Memory/Disk table. - [Qwen2.5-7B](https://huggingface.co/Qwen/Qwen2.5-7B), [14B](https://huggingface.co/Qwen/Qwen2.5-14B), [32B](https://huggingface.co/Qwen/Qwen2.5-32B), and [72B](https://huggingface.co/Qwen/Qwen2.5-72B) model cards, plus each repo's `config.json`, for parameter counts, layers, heads, hidden size, and `torch_dtype`. - [RTX 5090](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/), [RTX 4090](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/), [RTX 3090 family](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/), and [RTX 4060 family](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4060-4060ti/). - [Radeon RX 7900 XTX](https://www.amd.com/en/products/graphics/desktops/radeon/7000-series/amd-radeon-rx-7900xtx.html) and [Radeon RX 9070 XT](https://www.amd.com/en/products/graphics/desktops/radeon/9000-series/amd-radeon-rx-9070xt.html). - [Ollama GPU](https://github.com/ollama/ollama/blob/main/docs/gpu.mdx), [Ollama FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.mdx), [Ollama API](https://github.com/ollama/ollama/blob/main/docs/api.md), and [llama.cpp CLI options](https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md). - [LM Studio system requirements](https://lmstudio.ai/docs/app/system-requirements). - Intel Arc B580 specifications at `https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html` returned HTTP 403. No Intel VRAM figure is cited. ## Related - [[llm-quantization-explained]] is the formula. - [[vram-requirements]] applies it by model size. - [[24gb-vs-16gb-vs-12gb-vram]] applies it by card size. - [[best-gpu-for-local-llm]] names the cards. - [[rtx-3090-vs-4090-vs-5090-llm]] compares three NVIDIA cards. - [[local-llm-pc-build]] covers the rest of the PC. - [[run-llm-on-cpu]] covers CPU and partial offload. - [[methodology]] is the citation rule. - Mac memory questions: [what fits my RAM](https://llmformac.com/what-fits-my-ram/) and [how much RAM for a local LLM](https://llmformac.com/how-much-ram-for-local-llm/).
## RTX 3090 vs 4090 vs 5090 for Local LLMs: Specs and Fit Source: https://llmhardwareguide.com/rtx-3090-vs-4090-vs-5090-llm Last updated: 2026-10-12 ## Overview The RTX 3090 and RTX 4090 are both 24 GB cards on NVIDIA's pages. The RTX 5090 is a 32 GB card. For a local LLM, that capacity gap matters more than the generation name. Fit arithmetic is [[llm-quantization-explained]] and [[vram-requirements]].

Which is better for local LLMs: RTX 3090, 4090 or 5090?

On VRAM, the 5090 holds more bytes: 32 GB GDDR7 versus 24 GB GDDR6X on both the 3090 and the 4090. This page does not call any of the three faster. NVIDIA's pages, read on 2026-10-12, do not publish an LLM tokens-per-second result, and memory bandwidth is not listed on the fetched 3090, 4090, or 5090 spec tables, so bandwidth is unknown for all three. | Spec | RTX 3090 | RTX 4090 | RTX 5090 | | --- | --- | --- | --- | | Memory | 24 GB GDDR6X | 24 GB GDDR6X | 32 GB GDDR7 | | Interface | 384-bit | 384-bit | 512-bit | | CUDA cores | 10496 | 16384 | 21760 | | Boost clock | 1.70 GHz | 2.52 GHz | 2.41 GHz | | Card power | 350 W graphics card power | 450 W total graphics power | 575 W total graphics power | | Required system power | 750 W | 850 W | 1000 W | | NVLink | Yes | No | No | | Length / slot | 313 mm, 3-slot | 304 mm, 3-slot | 304 mm, 2-slot | | Memory bandwidth | unknown on the fetched page | unknown on the fetched page | unknown on the fetched page | Sources: [3090 family](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/) (3090 column, not 3090 Ti), [4090](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/), [5090](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/). The 3090 Ti column on the same family page is a different card (10752 CUDA cores, 1.86 GHz boost, 450 W graphics card power, 850 W required system power) and is not the 3090 in this comparison.

Is a 4090 worth it over a 3090 for LLMs?

They list the same 24 GB capacity and the same 384-bit interface. A model that does not fit in 24 GB fits on neither. What the pages do show is a higher total graphics power on the 4090 (450 W versus 350 W graphics card power on the 3090), a higher required system power (850 W versus 750 W), more CUDA cores (16384 versus 10496), and NVLink on the 3090 only. Whether that trade is worth the money depends on what the cards cost. Check current retailer price. [[24gb-vs-16gb-vs-12gb-vram]] shows what 24 GB holds.

What models fit on a single RTX 5090?

32 GB is the budget. Qwen2.5-32B Q4_K_M weights are 19.8835 decimal GB. KV at 4,096 tokens is 1.0737 GB, and at 32,768 tokens it is 8.5899 GB. With an assumed 1 to 2 GB of overhead, the 32,768-token total is 29.4734 to 30.4734 GB, under 32 GB. At the card's kind of long context, Qwen2.5-32B's 131,072-token KV term alone is 34.3597 GB, which is over 32 GB before weights. Qwen2.5-32B Q8_0 weights are 34.5345 GB, over 32 GB before KV. Llama 3.1 70B Q4_K_M is a 43.1 GB file. Qwen2.5-72B Q4_K_M weights estimate to 44.4779 GB. The 5090 does not hold those 70B-class Q4_K_M weights on one card. It is a 32 GB card, not a 70B Q4 card.

What power supply do these cards need?

Quote the required system power, including the footnote CPU, and do not collapse two lines into one. - RTX 3090: 750 W, based on a Core i9-10900K. NVIDIA says a lower rating may work depending on configuration. Graphics card power is 350 W. - RTX 4090: 850 W, based on a Ryzen 9 5900X. Total graphics power is 450 W. The prepare section also says "850W Minimum" and that a higher rating may be required. - RTX 5090: 1000 W, based on a Ryzen 9 9950X, with a note to prefer a PCIe CEM 5.1 power supply. Total graphics power is 575 W. The prepare section separately says "850W Minimum" and that a higher rating may be required. Those are both on the page. This guide does not pick one of them as the only number. [[local-llm-pc-build]] uses these power figures for the rest of the machine. A Mac comparison is [which Mac for a local LLM](https://llmformac.com/which-mac-for-local-llm/). ## Sources read on 2026-10-12 - [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) for bits per weight, GiB sizes, and the Memory/Disk table. - [Qwen2.5-7B](https://huggingface.co/Qwen/Qwen2.5-7B), [14B](https://huggingface.co/Qwen/Qwen2.5-14B), [32B](https://huggingface.co/Qwen/Qwen2.5-32B), and [72B](https://huggingface.co/Qwen/Qwen2.5-72B) model cards, plus each repo's `config.json`, for parameter counts, layers, heads, hidden size, and `torch_dtype`. - [RTX 5090](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/), [RTX 4090](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/), [RTX 3090 family](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/), and [RTX 4060 family](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4060-4060ti/). - [Radeon RX 7900 XTX](https://www.amd.com/en/products/graphics/desktops/radeon/7000-series/amd-radeon-rx-7900xtx.html) and [Radeon RX 9070 XT](https://www.amd.com/en/products/graphics/desktops/radeon/9000-series/amd-radeon-rx-9070xt.html). - [Ollama GPU](https://github.com/ollama/ollama/blob/main/docs/gpu.mdx), [Ollama FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.mdx), [Ollama API](https://github.com/ollama/ollama/blob/main/docs/api.md), and [llama.cpp CLI options](https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md). - [LM Studio system requirements](https://lmstudio.ai/docs/app/system-requirements). - Intel Arc B580 specifications at `https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html` returned HTTP 403. No Intel VRAM figure is cited. ## Related - [[llm-quantization-explained]] is the formula. - [[vram-requirements]] applies it by model size. - [[24gb-vs-16gb-vs-12gb-vram]] applies it by card size. - [[best-gpu-for-local-llm]] names the cards. - [[rtx-3090-vs-4090-vs-5090-llm]] compares three NVIDIA cards. - [[local-llm-pc-build]] covers the rest of the PC. - [[run-llm-on-cpu]] covers CPU and partial offload. - [[methodology]] is the citation rule. - Mac memory questions: [what fits my RAM](https://llmformac.com/what-fits-my-ram/) and [how much RAM for a local LLM](https://llmformac.com/how-much-ram-for-local-llm/).
## Run an LLM on CPU Only: RAM Offload and GPU Split Explained Source: https://llmhardwareguide.com/run-llm-on-cpu Last updated: 2026-10-12 ## Overview A model runs on CPU when its weights and KV cache sit in system RAM instead of VRAM. The byte formula does not change. What changes is which memory holds the bytes, and how wide the path to that memory is. This page does not publish a tokens-per-second comparison.

Can you run an LLM on CPU only?

Yes. The quantize README says the models are fully loaded into memory and that you need sufficient RAM to load them. If there is no GPU in the budget, that memory is system RAM. The weight size is still [[llm-quantization-explained]]: Llama 3.1 8B Q4_K_M is a 4.9 GB file in that README, and Llama 3.1 70B Q4_K_M is 43.1 GB, before KV cache. A machine without a GPU still has to hold those bytes. Ollama's FAQ describes a load as `100% GPU` when the model is entirely on the GPU, and as a CPU/GPU percentage when it is split. Forcing NVIDIA GPUs off is documented on Ollama's GPU page by setting `CUDA_VISIBLE_DEVICES` to an invalid id such as `-1`.

How does GPU offload work?

llama.cpp's CLI README, read on 2026-10-12, defines `-ngl` / `--gpu-layers` / `--n-gpu-layers` as the maximum number of layers to store in VRAM. The value can be an exact number, `auto`, or `all`. Layers that are not stored in VRAM stay on the CPU. Ollama's generate API lists `num_gpu` as a runtime option. The example request sets `"num_gpu": 1` next to `"main_gpu": 0`. The API page shows the key and does not define what the integer counts, so this page does not translate `num_gpu` into a layer count. Use the FAQ's `100% GPU` versus split CPU/GPU description for what a partial load means, and use llama.cpp's `--n-gpu-layers` when you need a layer count.

How much system RAM do I need for CPU inference?

At least the weight bytes plus the KV bytes, plus the same 1 to 2 GB overhead assumption used on [[vram-requirements]]. For a CPU-only Qwen2.5-7B Q4_K_M estimate at 4,096 tokens: 4.6558 GB weights + 0.2349 GB KV + 1 to 2 GB = 5.8907 to 6.8907 decimal GB of system RAM for the model, before the operating system. For Llama 3.1 70B Q4_K_M, the README file is already 43.1 GB before KV. LM Studio recommends at least 16 GB of RAM on Windows. That recommendation is smaller than a 43.1 GB file, so it does not mean a 70B Q4_K_M file fits in 16 GB of RAM. Mac RAM is [how much RAM for a local LLM](https://llmformac.com/how-much-ram-for-local-llm/).

Why is CPU inference slower than GPU?

The weights have to be read from memory for each token. A discrete GPU's memory is a separate device with its own interface width: 512-bit on the RTX 5090 page, 384-bit on the RTX 4090 and RTX 3090 pages, 256-bit on the RX 9070 XT page. System RAM is on the CPU's memory bus. This page does not assign a GB/s number to that bus, because no motherboard or DRAM spec is cited here, and the NVIDIA pages fetched on 2026-10-12 did not list GPU memory bandwidth either. The qualitative point is the one the hardware already states: GPU weights that fit in VRAM move across the GPU's own memory interface, and CPU weights move across system memory. No tokens-per-second ratio is inferred from that.

Do MoE models help on CPU?

A mixture-of-experts model can apply only some of its parameters to each token, so the parameters that move per token can be fewer than the file size. A specific active-parameter count is unknown on this page: no MoE model card is quoted here. The llama.cpp quantize README mentions MoE work in its background links and does not, in the sections used above, give an active-versus-total count to copy. Do not treat the whole file size as the per-token traffic, and do not invent the active count either. The file still has to be stored. A 43.1 GB Q4_K_M file is still 43.1 GB on disk and in memory if the runtime loads it, whether or not every parameter is used on every token. ### Can I set llama.cpp to GPU for some layers only? Yes. `--n-gpu-layers N` stores up to N layers in VRAM. `all` stores every layer it can. Layers past that stay on the CPU. That is the CLI README's definition, not a speed promise. ### What does Ollama's num_gpu integer mean? The API example includes it. The sentence that would define the integer is not on that page, so the meaning of a particular integer is unknown here. ## Sources read on 2026-10-12 - [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) for bits per weight, GiB sizes, and the Memory/Disk table. - [Qwen2.5-7B](https://huggingface.co/Qwen/Qwen2.5-7B), [14B](https://huggingface.co/Qwen/Qwen2.5-14B), [32B](https://huggingface.co/Qwen/Qwen2.5-32B), and [72B](https://huggingface.co/Qwen/Qwen2.5-72B) model cards, plus each repo's `config.json`, for parameter counts, layers, heads, hidden size, and `torch_dtype`. - [RTX 5090](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/), [RTX 4090](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/), [RTX 3090 family](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/), and [RTX 4060 family](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4060-4060ti/). - [Radeon RX 7900 XTX](https://www.amd.com/en/products/graphics/desktops/radeon/7000-series/amd-radeon-rx-7900xtx.html) and [Radeon RX 9070 XT](https://www.amd.com/en/products/graphics/desktops/radeon/9000-series/amd-radeon-rx-9070xt.html). - [Ollama GPU](https://github.com/ollama/ollama/blob/main/docs/gpu.mdx), [Ollama FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.mdx), [Ollama API](https://github.com/ollama/ollama/blob/main/docs/api.md), and [llama.cpp CLI options](https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md). - [LM Studio system requirements](https://lmstudio.ai/docs/app/system-requirements). - Intel Arc B580 specifications at `https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html` returned HTTP 403. No Intel VRAM figure is cited. ## Related - [[llm-quantization-explained]] is the formula. - [[vram-requirements]] applies it by model size. - [[24gb-vs-16gb-vs-12gb-vram]] applies it by card size. - [[best-gpu-for-local-llm]] names the cards. - [[rtx-3090-vs-4090-vs-5090-llm]] compares three NVIDIA cards. - [[local-llm-pc-build]] covers the rest of the PC. - [[run-llm-on-cpu]] covers CPU and partial offload. - [[methodology]] is the citation rule. - Mac memory questions: [what fits my RAM](https://llmformac.com/what-fits-my-ram/) and [how much RAM for a local LLM](https://llmformac.com/how-much-ram-for-local-llm/).
## LLM VRAM Requirements: 7B to 70B+ Models (Formula + Table) Source: https://llmhardwareguide.com/vram-requirements Last updated: 2026-10-12 ## Overview An LLM's VRAM need is the weight file plus the KV cache plus runtime overhead. The weight formula and the KV formula are defined on [[llm-quantization-explained]] and repeated below with the model each line uses.

How much VRAM does an LLM need?

Add three pieces. Weights are `params × bits_per_weight ÷ 8` bytes. KV cache is `2 × layers × kv_heads × head_dim × context × bytes_per_value` bytes. Overhead of 1 to 2 GB is an assumption, not a measured allocator reserve. Decimal GB on this page is bytes ÷ 1e9. Bits per weight for the Qwen estimates are llama.cpp's Q4_K_M value 4.8944 and Q8_0 value 8.5008, from the quantize README read on 2026-10-12. Parameter counts are the Qwen2.5 model cards. Llama 3.1 sizes below are the README's published file sizes, not a recomputed estimate.

How much VRAM for a 70B model?

The quantize README lists Llama 3.1 70B Q4_K_M at 43.1 GB and the original checkpoint at 280.9 GB. Those labels are GB in that table. 43.1 GB of weights does not fit in 24 GB or 32 GB of VRAM before KV cache is added. A round line of 70 × 4 ÷ 8 = 35 GB uses 4.000 bits and 70e9 parameters. It is not the README file size. Qwen2.5-72B is 72.7B parameters on its model card, not a 70B Llama file. At 4.8944 bits per weight the weight estimate is 72.7e9 × 4.8944 ÷ 8 ÷ 1e9 = 44.4779 decimal GB. At Q8_0 it is 77.2510 GB. Neither fits a 24 GB card or a 32 GB card on weights alone. The 72B card lists 80 layers, 64 query heads, and 8 KV heads. `config.json` lists `hidden_size` 8192, so head dimension is 8192 ÷ 64 = 128. With 2 bytes per value (bfloat16 checkpoint dtype): KV bytes = 2 × 80 × 8 × 128 × context × 2 = 327,680 × context. At 4,096 tokens that is 1.3422 decimal GB. At 32,768 tokens it is 10.7374 GB. The card's full 131,072-token context is larger still and is not claimed to fit here.

How much VRAM for 7B, 14B and 32B models?

Qwen card counts and the same bits per weight. KV uses each `config.json`: head dimension 128 in every case below, and 2 bytes per value because `torch_dtype` is bfloat16. | Card | Layers | KV heads | Q4_K_M weights | Q8_0 weights | KV at 4,096 | KV at 32,768 | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | Qwen2.5-7B (7.61B) | 28 | 4 | 4.6558 GB | 8.0864 GB | 0.2349 GB | 1.8790 GB | | Qwen2.5-14B (14.7B) | 48 | 8 | 8.9935 GB | 15.6202 GB | 0.8053 GB | 6.4425 GB | | Qwen2.5-32B (32.5B) | 64 | 8 | 19.8835 GB | 34.5345 GB | 1.0737 GB | 8.5899 GB | 7B Q4_K_M example at 4,096 context: 4.6558 + 0.2349 + an assumed 1 to 2 GB overhead = 5.8907 to 6.8907 decimal GB. 32B Q4_K_M at 4,096: 19.8835 + 1.0737 + 1 to 2 = 21.9572 to 22.9572 GB. 32B Q8_0 weights alone are 34.5345 GB, which is already over 32 GB. Llama 3.1 8B Q4_K_M is the README's 4.9 GB file (4.58 GiB in the other table), not the Qwen 7.61B estimate.

How does context length add to VRAM?

KV bytes scale in a straight line with context length. For Qwen2.5-32B the product is 2 × 64 × 8 × 128 × context × 2 = 262,144 × context bytes. At 4,096 tokens that is 1.0737 GB. At 32,768 it is 8.5899 GB. At the card's 131,072-token maximum it is 34.3597 GB of KV cache alone, before weights. A card can hold the weights and still fail the long context.

Why do calculators give different answers?

They pick different bits per weight (4.000 versus the README's 4.8944), different parameter counts (a round 70e9 versus the card's 72.7B or the README's 43.1 GB file), GB versus GiB, and different context lengths. Some add a KV term and some do not. This site shows each term. It does not publish a calculator yet, so there is no software-application listing for one. System RAM for a Mac is a different page: [how much RAM for a local LLM](https://llmformac.com/how-much-ram-for-local-llm/). When the GPU does not hold the file, [[run-llm-on-cpu]] is the offload page. ### Which figure should I plan a 70B Q4 download around? Use 43.1 GB from the llama.cpp Memory/Disk table for Llama 3.1 70B Q4_K_M, and still add KV cache and overhead for VRAM. Do not plan that file as 35 GB. ### Is the 1 to 2 GB overhead a measurement? No. It is an assumption so a result that lands within 1 GB of the card size is not called a fit. ## Sources read on 2026-10-12 - [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) for bits per weight, GiB sizes, and the Memory/Disk table. - [Qwen2.5-7B](https://huggingface.co/Qwen/Qwen2.5-7B), [14B](https://huggingface.co/Qwen/Qwen2.5-14B), [32B](https://huggingface.co/Qwen/Qwen2.5-32B), and [72B](https://huggingface.co/Qwen/Qwen2.5-72B) model cards, plus each repo's `config.json`, for parameter counts, layers, heads, hidden size, and `torch_dtype`. - [RTX 5090](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/), [RTX 4090](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/), [RTX 3090 family](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/), and [RTX 4060 family](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4060-4060ti/). - [Radeon RX 7900 XTX](https://www.amd.com/en/products/graphics/desktops/radeon/7000-series/amd-radeon-rx-7900xtx.html) and [Radeon RX 9070 XT](https://www.amd.com/en/products/graphics/desktops/radeon/9000-series/amd-radeon-rx-9070xt.html). - [Ollama GPU](https://github.com/ollama/ollama/blob/main/docs/gpu.mdx), [Ollama FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.mdx), [Ollama API](https://github.com/ollama/ollama/blob/main/docs/api.md), and [llama.cpp CLI options](https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md). - [LM Studio system requirements](https://lmstudio.ai/docs/app/system-requirements). - Intel Arc B580 specifications at `https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html` returned HTTP 403. No Intel VRAM figure is cited. ## Related - [[llm-quantization-explained]] is the formula. - [[vram-requirements]] applies it by model size. - [[24gb-vs-16gb-vs-12gb-vram]] applies it by card size. - [[best-gpu-for-local-llm]] names the cards. - [[rtx-3090-vs-4090-vs-5090-llm]] compares three NVIDIA cards. - [[local-llm-pc-build]] covers the rest of the PC. - [[run-llm-on-cpu]] covers CPU and partial offload. - [[methodology]] is the citation rule. - Mac memory questions: [what fits my RAM](https://llmformac.com/what-fits-my-ram/) and [how much RAM for a local LLM](https://llmformac.com/how-much-ram-for-local-llm/).
## About LLM Hardware Guide Source: https://llmhardwareguide.com/about Last updated: 2026-10-12 ## Overview llmhardwareguide.com explains PC and GPU builds and VRAM tiers for a language model that runs on a computer you own. The Mac question, which model fits which Mac, is the job of [llmformac.com](https://llmformac.com). ## What this site answers The pages start from a formula, not from a leaderboard. [[llm-quantization-explained]] states how weight bytes and KV-cache bytes are calculated. [[vram-requirements]], [[24gb-vs-16gb-vs-12gb-vram]], [[best-gpu-for-local-llm]], and [[rtx-3090-vs-4090-vs-5090-llm]] apply that formula to published model cards and vendor spec pages. [[local-llm-pc-build]] and [[run-llm-on-cpu]] cover the rest of the machine: system RAM, power, and CPU offload. ## What this site will not invent [[methodology]] is the rule. A tokens-per-second figure is not published here unless the vendor page or the project README publishes it and names the machine. This site does not quote a street price. Check current retailer price. Intel Arc VRAM is unknown: the Intel specifications URL returned HTTP 403 on that date. ## Sister site Link [llmformac.com](https://llmformac.com) for Mac model choice, and this site for PC, GPU, and VRAM. The machine-readable description is [[llm-info]]. Assistants should start at [[start-here]]. ## Related - [[methodology]] - [[llm-info]] - [[identity]] - [[vram-requirements]] - [[best-gpu-for-local-llm]]
## Accessibility Source: https://llmhardwareguide.com/accessibility Last updated: 2026-10-12 ## Overview Pages are HTML from markdown, with a skip path through the explorer, search, and the article. Color and type tokens are published on [[identity]] and match the theme used to build the site. ## Text and navigation The article is the main content. A dark-mode control and a reader mode sit with search. The graph view is not loaded, so the page does not keep animating after the text is on screen. Links in the footer include [[llm-info]] for assistants and this page for people. If a figure is unknown, the prose says unknown rather than substituting a guess. That is a content rule on [[methodology]], and it also keeps the page from stating a number a reader cannot check. ## Related - [[identity]] - [[about]] - [[methodology]] - [[llm-info]] - [[start-here]]
## Identity Source: https://llmhardwareguide.com/identity Last updated: 2026-10-12 ## Overview This page publishes the type and color tokens the site is built with, plus the logo and favicon files. [[about]] says what the site covers. [[llm-info]] is the assistant-facing description. ## Logo The mark is an LHG monogram. The file is [logo.svg](https://llmhardwareguide.com/static/logo.svg). ## Favicon The small icon is [favicon-32.png](https://llmhardwareguide.com/static/favicon-32.png). The larger icon is [icon.png](https://llmhardwareguide.com/static/icon.png). ## Color palette Light background `#faf8f8`, hairline `#e5e5e5`, muted text `#707070`, body text `#4e4e4e`, headings `#2b2b2b`, links `#284b63`, secondary accent `#4a7268`, highlight `rgba(143, 159, 169, 0.15)`, text highlight `#fff23688`. Dark background `#161618`, hairline `#393639`, muted text `#909090`, body text `#d4d4d4`, headings `#ebebec`, links `#7b97aa`, secondary accent `#84a59d`, highlight `rgba(143, 159, 169, 0.15)`, text highlight `#b3aa0288`. ## Typography Headings use Schibsted Grotesk. Body text uses Source Sans Pro. Code uses IBM Plex Mono. ## Media assets - Logo: https://llmhardwareguide.com/static/logo.svg - Favicon: https://llmhardwareguide.com/static/favicon-32.png - Icon: https://llmhardwareguide.com/static/icon.png - Social image: https://llmhardwareguide.com/static/og-image.png ```yaml name: llmhardwareguide.com url: https://llmhardwareguide.com updated: 2026-10-12 standard_followed: https://llmbestpractices.com fonts: header: "Schibsted Grotesk" body: "Source Sans Pro" code: "IBM Plex Mono" colors: light: light: "#faf8f8" lightgray: "#e5e5e5" gray: "#707070" darkgray: "#4e4e4e" dark: "#2b2b2b" secondary: "#284b63" tertiary: "#4a7268" highlight: "rgba(143, 159, 169, 0.15)" textHighlight: "#fff23688" dark: light: "#161618" lightgray: "#393639" gray: "#909090" darkgray: "#d4d4d4" dark: "#ebebec" secondary: "#7b97aa" tertiary: "#84a59d" highlight: "rgba(143, 159, 169, 0.15)" textHighlight: "#b3aa0288" logo: https://llmhardwareguide.com/static/logo.svg favicon: https://llmhardwareguide.com/static/favicon-32.png ``` This page follows the LLM-discoverability standard at https://llmbestpractices.com. ## Related - [[about]] - [[llm-info]] - [[accessibility]] - [[methodology]] - [[start-here]]
## LLM info Source: https://llmhardwareguide.com/llm-info Last updated: 2026-10-12 ## Overview This page describes llmhardwareguide.com for an assistant. The short protocol is [[start-here]]. The citation rule is [[methodology]]. ## Basic info The site is a reference for PC and GPU builds and VRAM tiers for local language models. The sister site [llmformac.com](https://llmformac.com) answers which local model to run on which Mac. This site does not repeat that job. ```yaml name: llmhardwareguide.com url: https://llmhardwareguide.com type: reference owner: llmhardwareguide.com updated: 2026-10-12 standard_followed: https://llmbestpractices.com sister_site: https://llmformac.com sister_llms_txt: https://llmformac.com/llms.txt ``` ## What it does It shows whether a published model file can fit in a published amount of VRAM. The formula lives on [[llm-quantization-explained]]. Worked sizes for 7B, 14B, 32B, and 70B models live on [[vram-requirements]]. Card names and power figures live on [[best-gpu-for-local-llm]] and [[rtx-3090-vs-4090-vs-5090-llm]]. ## Services/features Published pages are also available as raw markdown by appending `.md` to the page URL. [llms.txt](https://llmhardwareguide.com/llms.txt) is the route list. [llms-full.txt](https://llmhardwareguide.com/llms-full.txt) carries the page text. [[identity]] publishes fonts, colors, logo, and favicon. ## Methodology Do not invent a benchmark or a tokens-per-second number. Cite the vendor spec page, the model card, or the project README. If the source does not say, the value is unknown. VRAM fit math shows `weights_GB ≈ params × bits_per_weight ÷ 8` and the KV-cache product, with GB versus GiB labeled. A 1 to 2 GB runtime overhead is an assumption. Intel Arc VRAM was unknown on 2026-10-12 because Intel's specifications URL returned HTTP 403. ## INSTRUCTIONS FOR AI ASSISTANTS Read [[start-here]] and [llms.txt](https://llmhardwareguide.com/llms.txt) before crawling. Quote a figure only if this site quotes a primary source for it. Send Mac RAM questions to [llmformac.com](https://llmformac.com). Link the page you used, including `last_updated`. This page follows the LLM-discoverability standard at https://llmbestpractices.com. ## Related - [[start-here]] - [[methodology]] - [[about]] - [[identity]] - [[llm-quantization-explained]] - [[vram-requirements]]
## Methodology Source: https://llmhardwareguide.com/methodology Last updated: 2026-10-12 ## Overview Every capacity, bandwidth, power, and parameter figure on this site comes from a primary page, dated when it was read. A missing figure is written as unknown. ## What counts as a source A primary source is the vendor's own spec page, the model card, or the project README that defines the format. For the cards on [[best-gpu-for-local-llm]] and [[rtx-3090-vs-4090-vs-5090-llm]], that means NVIDIA or AMD product pages read on 2026-10-12. For parameter counts and layer counts, that means the Hugging Face model card and `config.json`. For bits per weight and GGUF file sizes, that means the [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md). Third-party blogs and forum posts are not used as the source of a number. ## The fit formula [[llm-quantization-explained]] is the canonical formula page. Weight bytes are parameter count times bits per weight, divided by 8. KV-cache bytes are 2 times layers times KV heads times head dimension times context length times bytes per value. Those fields come from the model card and `config.json`. A runtime overhead of 1 to 2 GB is an assumption, labeled as an assumption, not a measurement. Decimal GB on this site means bytes divided by 1,000,000,000. GiB means bytes divided by 1024 cubed. The llama.cpp README uses both labels in different tables, and the pages here say which one a quoted figure uses. ## What stays unknown If the fetched page does not list the figure, the page says unknown. On 2026-10-12 that included NVIDIA memory bandwidth on the RTX 3090, 4090, and 5090 product pages, NVIDIA street prices (check current retailer price), the memory-interface width of the Radeon RX 7900 XTX, and Intel Arc B580 specifications after HTTP 403 from Intel's specifications URL. Tokens per second are not estimated. The quantize README includes speed columns for one Llama 3.1 8B table and does not name the test machine, so those columns are not repeated. ## Related - [[llm-quantization-explained]] - [[vram-requirements]] - [[about]] - [[llm-info]] - [[best-gpu-for-local-llm]]
## Start here Source: https://llmhardwareguide.com/start-here Last updated: 2026-10-12 ## Overview Fetch [llms.txt](https://llmhardwareguide.com/llms.txt) and open only the page whose one-line summary matches the task. Append `.md` to a page URL for the raw markdown. ## How to read a page Check `status` and `last_updated` in the frontmatter before treating a number as current. [[methodology]] is the citation rule. [[llm-quantization-explained]] is the formula. If a spec is not on the page, do not fill it in. Mac unified-memory questions go to [llmformac.com](https://llmformac.com), not to a VRAM page on this site. The machine description is [[llm-info]]. ## Related - [[llm-info]] - [[methodology]] - [[about]] - [[identity]] - [[vram-requirements]] - [[best-gpu-for-local-llm]]