LLM Quantization Explained: Q4 vs Q8, GGUF and Memory Math

Overview

LLM quantization stores weights with fewer bits than the training checkpoint, and the size follows from the parameter count and the bits per weight. This page is the formula the rest of the site cites.

What is LLM quantization?

LLM quantization stores a model’s weights with fewer bits (for example 4 or 8 instead of 16), which shrinks the file roughly in proportion to the bit width. The llama.cpp quantize README, read on 2026-10-12, says the tool takes a high-precision GGUF file, typically F32 or BF16, and converts it to a quantized format. It also says the process may introduce accuracy loss, usually measured as perplexity or KL divergence, and that an importance matrix can reduce that loss.

The same README’s Memory/Disk table gives Llama 3.1 file sizes in GB (the table’s label, not GiB):

What is the difference between Q4 and Q8?

In that README’s Llama 3.1 8B table, Q4_K_M is 4.8944 bits per weight and 4.58 GiB, and Q8_0 is 8.5008 bits per weight and 7.95 GiB, so the Q4_K_M file is a bit over half the Q8_0 file (4.58 / 7.95). A round “4-bit” estimate is not the same number as Q4_K_M.

Selected rows from the same table. Speed columns are omitted here because the README does not name the test machine.

The Memory/Disk table’s 4.9 GB for Llama 3.1 8B Q4_K_M and this table’s 4.58 GiB are the same file described two ways. 4.58 GiB is 4.58 × 1024³ bytes, which is about 4.92 decimal GB, and the other table rounds that to 4.9 GB.

How do I calculate a quantized model's size?

Multiply the parameter count by the bits per weight and divide by 8 to get bytes. Divide by 1,000,000,000 for decimal GB. Divide by 1024³ for GiB.

weights_bytes = params × bits_per_weight ÷ 8

weights_GB = weights_bytes ÷ 1e9

The bits per weight below are the README’s Q4_K_M value 4.8944 and Q8_0 value 8.5008, applied to parameter counts on the Qwen2.5 model cards (read 2026-10-12). This is an estimate of weight bytes. It is not a measured GGUF file size. The README’s own file sizes are the rows above, and they are the figures to use for Llama 3.1.

Worked line for the 7B card at Q4_K_M: 7.61e9 × 4.8944 ÷ 8 ÷ 1e9 = 4.6558 decimal GB. The round 4-bit line 7.61e9 × 4 ÷ 8 ÷ 1e9 = 3.8050 GB understates Q4_K_M because 4.8944 is not 4.

KV cache is separate. For a decoder with separate key and value tensors:

kv_bytes = 2 × layers × kv_heads × head_dim × context × bytes_per_value

head_dim = hidden_size ÷ num_attention_heads

Qwen2.5-7B config.json lists hidden_size 3584, num_attention_heads 28, num_key_value_heads 4, num_hidden_layers 28, and torch_dtype bfloat16. Head dimension is 3584 ÷ 28 = 128. This page uses 2 bytes per value because the checkpoint dtype is bfloat16. KV dtype is a runtime choice; quantized KV is not in config.json.

Substituting: 2 × 28 × 4 × 128 × context × 2 = 57,344 × context bytes.

The card’s context length is 131,072 tokens. That full length is not what the table above uses. A runtime overhead of 1 to 2 GB is an assumption on this site, not a measurement from the README. vram-requirements adds these pieces per model size.

A nameplate “70 billion × 4 ÷ 8 = 35 GB” is the round 4-bit estimate using 70e9. The README’s Llama 3.1 70B Q4_K_M file size is 43.1 GB. Plan with 43.1 GB for that file, not with 35 GB.

Does quantization reduce quality?

The quantize README says lower-precision weights may introduce accuracy loss, usually measured as perplexity or KL divergence, and that an importance matrix can limit that loss. The Llama 3.1 8B table on that page, read on 2026-10-12, does not include a perplexity column, so this page does not quote a perplexity number. It also does not repeat the table’s tokens-per-second columns, because the README does not name the machine that produced them.

What are GGUF, safetensors and MLX formats?

GGUF is the file the quantize README converts to and then quantizes: convert_hf_to_gguf.py writes a GGUF file, and llama-quantize writes a quantized GGUF such as Q4_K_M. Hugging Face documents GGUF for the Hub at huggingface.co/docs/hub/gguf. Safetensors is the tensor format documented by Hugging Face at huggingface.co/docs/safetensors; the README’s convert step is how a Hugging Face repo becomes GGUF. MLX is Apple’s array framework at github.com/ml-explore/mlx. How MLX weights compare with GGUF is the Mac site’s page, mlx vs gguf, not a second formula here.

What should I cite for a fit claim?

Cite this page’s formula, the model card for the parameter count, and the quantize README for the bits per weight or the published file size. Say whether the result is decimal GB or GiB.

Does Q4 always mean 4.000 bits?

No. In the README table, Q4_K_M is 4.8944 bits per weight and Q4_K_S is 4.6672. A teaching line that uses 4.000 bits is labeled as a round estimate on this site.

Sources read on 2026-10-12

• llama.cpp quantize README for bits per weight, GiB sizes, and the Memory/Disk table.
• Qwen2.5-7B, 14B, 32B, and 72B model cards, plus each repo’s config.json, for parameter counts, layers, heads, hidden size, and torch_dtype.
• RTX 5090, RTX 4090, RTX 3090 family, and RTX 4060 family.
• Radeon RX 7900 XTX and Radeon RX 9070 XT.
• Ollama GPU, Ollama FAQ, Ollama API, and llama.cpp CLI options.
• LM Studio system requirements.
• Intel Arc B580 specifications at https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html returned HTTP 403. No Intel VRAM figure is cited.

Related

• llm-quantization-explained is the formula.
• vram-requirements applies it by model size.
• 24gb-vs-16gb-vs-12gb-vram applies it by card size.
• best-gpu-for-local-llm names the cards.
• rtx-3090-vs-4090-vs-5090-llm compares three NVIDIA cards.
• local-llm-pc-build covers the rest of the PC.
• run-llm-on-cpu covers CPU and partial offload.
• methodology is the citation rule.
• Mac memory questions: what fits my RAM and how much RAM for a local LLM.