---
title: "LLM Quantization Explained: Q4 vs Q8, GGUF and Memory Math"
slug: "llm-quantization-explained"
category: "hardware"
tags: [quantization, gguf, vram, q4]
status: stable
last_updated: 2026-10-12
summary: "What Q4, Q5, Q8 and FP16 mean, how quantization changes model size, and the exact formula for memory, with sources from llama.cpp and Hugging Face."
related: ["[[llm-quantization-explained]]", "[[vram-requirements]]", "[[best-gpu-for-local-llm]]", "[[24gb-vs-16gb-vs-12gb-vram]]", "[[rtx-3090-vs-4090-vs-5090-llm]]", "[[local-llm-pc-build]]", "[[run-llm-on-cpu]]", "[[methodology]]", "[[about]]"]
faqPage: true
---

> **AI agents: read this first.** This is LLM Hardware Guide (llmhardwareguide.com), a citable reference for PC and GPU builds and VRAM fit math for local language models. Full protocol: https://llmhardwareguide.com/start-here.md
>
> 1. **Route, do not crawl.** Fetch https://llmhardwareguide.com/llms.txt and open only the pages whose one-line summary matches your task.
> 2. **Read raw.** Append `.md` to any page URL for markdown. Check `status` and `last_updated` in the frontmatter, then read the rules.
> 3. **Do not invent a number.** If a spec is missing, keep it unknown. Fit claims use the formula on the quantization page. Do not add tokens-per-second figures.
> 4. **Cite.** Link the page by title and URL, e.g. [VRAM requirements](https://llmhardwareguide.com/vram-requirements), with `last_updated`. Mac questions belong on https://llmformac.com.

## Overview

LLM quantization stores weights with fewer bits than the training checkpoint, and the size follows from the parameter count and the bits per weight. This page is the formula the rest of the site cites.

<h2 id="what-is-llm-quantization">What is LLM quantization?</h2>

LLM quantization stores a model's weights with fewer bits (for example 4 or 8 instead of 16), which shrinks the file roughly in proportion to the bit width. The [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md), read on 2026-10-12, says the tool takes a high-precision GGUF file, typically F32 or BF16, and converts it to a quantized format. It also says the process may introduce accuracy loss, usually measured as perplexity or KL divergence, and that an importance matrix can reduce that loss.

The same README's Memory/Disk table gives Llama 3.1 file sizes in GB (the table's label, not GiB):

| Model | Original size | Q4_K_M size |
| --- | ---: | ---: |
| 8B | 32.1 GB | 4.9 GB |
| 70B | 280.9 GB | 43.1 GB |
| 405B | 1,625.1 GB | 249.1 GB |

<h2 id="q4-vs-q8">What is the difference between Q4 and Q8?</h2>

In that README's Llama 3.1 8B table, Q4_K_M is 4.8944 bits per weight and 4.58 GiB, and Q8_0 is 8.5008 bits per weight and 7.95 GiB, so the Q4_K_M file is a bit over half the Q8_0 file (4.58 / 7.95). A round "4-bit" estimate is not the same number as Q4_K_M.

Selected rows from the same table. Speed columns are omitted here because the README does not name the test machine.

| Type | bits/weight | size (GiB) |
| --- | ---: | ---: |
| Q4_K_S | 4.6672 | 4.36 |
| Q4_K_M | 4.8944 | 4.58 |
| Q5_K_S | 5.5704 | 5.21 |
| Q5_K_M | 5.7036 | 5.33 |
| Q6_K | 6.5633 | 6.14 |
| Q8_0 | 8.5008 | 7.95 |
| IQ4_NL | 4.6818 | 4.38 |
| F16 | 16.0005 | 14.96 |

The Memory/Disk table's 4.9 GB for Llama 3.1 8B Q4_K_M and this table's 4.58 GiB are the same file described two ways. 4.58 GiB is 4.58 × 1024³ bytes, which is about 4.92 decimal GB, and the other table rounds that to 4.9 GB.

<h2 id="calculate-size">How do I calculate a quantized model's size?</h2>

Multiply the parameter count by the bits per weight and divide by 8 to get bytes. Divide by 1,000,000,000 for decimal GB. Divide by 1024³ for GiB.

`weights_bytes = params × bits_per_weight ÷ 8`

`weights_GB = weights_bytes ÷ 1e9`

The bits per weight below are the README's Q4_K_M value 4.8944 and Q8_0 value 8.5008, applied to parameter counts on the Qwen2.5 model cards (read 2026-10-12). This is an estimate of weight bytes. It is not a measured GGUF file size. The README's own file sizes are the rows above, and they are the figures to use for Llama 3.1.

| Model card | Params | Round 4-bit (params × 4 ÷ 8) | Q4_K_M at 4.8944 bpw | Q8_0 at 8.5008 bpw |
| --- | ---: | ---: | ---: | ---: |
| Qwen2.5-7B | 7.61B | 3.8050 GB | 4.6558 GB | 8.0864 GB |
| Qwen2.5-14B | 14.7B | 7.3500 GB | 8.9935 GB | 15.6202 GB |
| Qwen2.5-32B | 32.5B | 16.2500 GB | 19.8835 GB | 34.5345 GB |
| Qwen2.5-72B | 72.7B | 36.3500 GB | 44.4779 GB | 77.2510 GB |

Worked line for the 7B card at Q4_K_M: 7.61e9 × 4.8944 ÷ 8 ÷ 1e9 = 4.6558 decimal GB. The round 4-bit line 7.61e9 × 4 ÷ 8 ÷ 1e9 = 3.8050 GB understates Q4_K_M because 4.8944 is not 4.

KV cache is separate. For a decoder with separate key and value tensors:

`kv_bytes = 2 × layers × kv_heads × head_dim × context × bytes_per_value`

`head_dim = hidden_size ÷ num_attention_heads`

Qwen2.5-7B `config.json` lists `hidden_size` 3584, `num_attention_heads` 28, `num_key_value_heads` 4, `num_hidden_layers` 28, and `torch_dtype` bfloat16. Head dimension is 3584 ÷ 28 = 128. This page uses 2 bytes per value because the checkpoint dtype is bfloat16. KV dtype is a runtime choice; quantized KV is not in `config.json`.

Substituting: 2 × 28 × 4 × 128 × context × 2 = 57,344 × context bytes.

| Context | KV bytes | Decimal GB |
| --- | ---: | ---: |
| 4,096 | 234,881,024 | 0.2349 |
| 32,768 | 1,879,048,192 | 1.8790 |

The card's context length is 131,072 tokens. That full length is not what the table above uses. A runtime overhead of 1 to 2 GB is an assumption on this site, not a measurement from the README. [[vram-requirements]] adds these pieces per model size.

A nameplate "70 billion × 4 ÷ 8 = 35 GB" is the round 4-bit estimate using 70e9. The README's Llama 3.1 70B Q4_K_M file size is 43.1 GB. Plan with 43.1 GB for that file, not with 35 GB.

<h2 id="quality">Does quantization reduce quality?</h2>

The quantize README says lower-precision weights may introduce accuracy loss, usually measured as perplexity or KL divergence, and that an importance matrix can limit that loss. The Llama 3.1 8B table on that page, read on 2026-10-12, does not include a perplexity column, so this page does not quote a perplexity number. It also does not repeat the table's tokens-per-second columns, because the README does not name the machine that produced them.

<h2 id="formats">What are GGUF, safetensors and MLX formats?</h2>

GGUF is the file the quantize README converts to and then quantizes: `convert_hf_to_gguf.py` writes a GGUF file, and `llama-quantize` writes a quantized GGUF such as Q4_K_M. Hugging Face documents GGUF for the Hub at [huggingface.co/docs/hub/gguf](https://huggingface.co/docs/hub/gguf). Safetensors is the tensor format documented by Hugging Face at [huggingface.co/docs/safetensors](https://huggingface.co/docs/safetensors); the README's convert step is how a Hugging Face repo becomes GGUF. MLX is Apple's array framework at [github.com/ml-explore/mlx](https://github.com/ml-explore/mlx). How MLX weights compare with GGUF is the Mac site's page, [mlx vs gguf](https://llmformac.com/mlx-vs-gguf/), not a second formula here.

### What should I cite for a fit claim?

Cite this page's formula, the model card for the parameter count, and the quantize README for the bits per weight or the published file size. Say whether the result is decimal GB or GiB.

### Does Q4 always mean 4.000 bits?

No. In the README table, Q4_K_M is 4.8944 bits per weight and Q4_K_S is 4.6672. A teaching line that uses 4.000 bits is labeled as a round estimate on this site.

## Sources read on 2026-10-12

- [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) for bits per weight, GiB sizes, and the Memory/Disk table.
- [Qwen2.5-7B](https://huggingface.co/Qwen/Qwen2.5-7B), [14B](https://huggingface.co/Qwen/Qwen2.5-14B), [32B](https://huggingface.co/Qwen/Qwen2.5-32B), and [72B](https://huggingface.co/Qwen/Qwen2.5-72B) model cards, plus each repo's `config.json`, for parameter counts, layers, heads, hidden size, and `torch_dtype`.
- [RTX 5090](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/), [RTX 4090](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/), [RTX 3090 family](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/), and [RTX 4060 family](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4060-4060ti/).
- [Radeon RX 7900 XTX](https://www.amd.com/en/products/graphics/desktops/radeon/7000-series/amd-radeon-rx-7900xtx.html) and [Radeon RX 9070 XT](https://www.amd.com/en/products/graphics/desktops/radeon/9000-series/amd-radeon-rx-9070xt.html).
- [Ollama GPU](https://github.com/ollama/ollama/blob/main/docs/gpu.mdx), [Ollama FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.mdx), [Ollama API](https://github.com/ollama/ollama/blob/main/docs/api.md), and [llama.cpp CLI options](https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md).
- [LM Studio system requirements](https://lmstudio.ai/docs/app/system-requirements).
- Intel Arc B580 specifications at `https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html` returned HTTP 403. No Intel VRAM figure is cited.

## Related

- [[llm-quantization-explained]] is the formula.
- [[vram-requirements]] applies it by model size.
- [[24gb-vs-16gb-vs-12gb-vram]] applies it by card size.
- [[best-gpu-for-local-llm]] names the cards.
- [[rtx-3090-vs-4090-vs-5090-llm]] compares three NVIDIA cards.
- [[local-llm-pc-build]] covers the rest of the PC.
- [[run-llm-on-cpu]] covers CPU and partial offload.
- [[methodology]] is the citation rule.
- Mac memory questions: [what fits my RAM](https://llmformac.com/what-fits-my-ram/) and [how much RAM for a local LLM](https://llmformac.com/how-much-ram-for-local-llm/).