Overview
A model runs on CPU when its weights and KV cache sit in system RAM instead of VRAM. The byte formula does not change. What changes is which memory holds the bytes, and how wide the path to that memory is. This page does not publish a tokens-per-second comparison.
Can you run an LLM on CPU only?
Yes. The quantize README says the models are fully loaded into memory and that you need sufficient RAM to load them. If there is no GPU in the budget, that memory is system RAM. The weight size is still llm-quantization-explained: Llama 3.1 8B Q4_K_M is a 4.9 GB file in that README, and Llama 3.1 70B Q4_K_M is 43.1 GB, before KV cache. A machine without a GPU still has to hold those bytes.
Ollama’s FAQ describes a load as 100% GPU when the model is entirely on the GPU, and as a CPU/GPU percentage when it is split. Forcing NVIDIA GPUs off is documented on Ollama’s GPU page by setting CUDA_VISIBLE_DEVICES to an invalid id such as -1.
How does GPU offload work?
llama.cpp’s CLI README, read on 2026-10-12, defines -ngl / --gpu-layers / --n-gpu-layers as the maximum number of layers to store in VRAM. The value can be an exact number, auto, or all. Layers that are not stored in VRAM stay on the CPU.
Ollama’s generate API lists num_gpu as a runtime option. The example request sets "num_gpu": 1 next to "main_gpu": 0. The API page shows the key and does not define what the integer counts, so this page does not translate num_gpu into a layer count. Use the FAQ’s 100% GPU versus split CPU/GPU description for what a partial load means, and use llama.cpp’s --n-gpu-layers when you need a layer count.
How much system RAM do I need for CPU inference?
At least the weight bytes plus the KV bytes, plus the same 1 to 2 GB overhead assumption used on vram-requirements. For a CPU-only Qwen2.5-7B Q4_K_M estimate at 4,096 tokens: 4.6558 GB weights + 0.2349 GB KV + 1 to 2 GB = 5.8907 to 6.8907 decimal GB of system RAM for the model, before the operating system. For Llama 3.1 70B Q4_K_M, the README file is already 43.1 GB before KV. LM Studio recommends at least 16 GB of RAM on Windows. That recommendation is smaller than a 43.1 GB file, so it does not mean a 70B Q4_K_M file fits in 16 GB of RAM.
Mac RAM is how much RAM for a local LLM.
Why is CPU inference slower than GPU?
The weights have to be read from memory for each token. A discrete GPU’s memory is a separate device with its own interface width: 512-bit on the RTX 5090 page, 384-bit on the RTX 4090 and RTX 3090 pages, 256-bit on the RX 9070 XT page. System RAM is on the CPU’s memory bus. This page does not assign a GB/s number to that bus, because no motherboard or DRAM spec is cited here, and the NVIDIA pages fetched on 2026-10-12 did not list GPU memory bandwidth either. The qualitative point is the one the hardware already states: GPU weights that fit in VRAM move across the GPU’s own memory interface, and CPU weights move across system memory. No tokens-per-second ratio is inferred from that.
Do MoE models help on CPU?
A mixture-of-experts model can apply only some of its parameters to each token, so the parameters that move per token can be fewer than the file size. A specific active-parameter count is unknown on this page: no MoE model card is quoted here. The llama.cpp quantize README mentions MoE work in its background links and does not, in the sections used above, give an active-versus-total count to copy. Do not treat the whole file size as the per-token traffic, and do not invent the active count either. The file still has to be stored. A 43.1 GB Q4_K_M file is still 43.1 GB on disk and in memory if the runtime loads it, whether or not every parameter is used on every token.
Can I set llama.cpp to GPU for some layers only?
Yes. --n-gpu-layers N stores up to N layers in VRAM. all stores every layer it can. Layers past that stay on the CPU. That is the CLI README’s definition, not a speed promise.
What does Ollama’s num_gpu integer mean?
The API example includes it. The sentence that would define the integer is not on that page, so the meaning of a particular integer is unknown here.
Sources read on 2026-10-12
- llama.cpp quantize README for bits per weight, GiB sizes, and the Memory/Disk table.
- Qwen2.5-7B, 14B, 32B, and 72B model cards, plus each repo’s
config.json, for parameter counts, layers, heads, hidden size, andtorch_dtype. - RTX 5090, RTX 4090, RTX 3090 family, and RTX 4060 family.
- Radeon RX 7900 XTX and Radeon RX 9070 XT.
- Ollama GPU, Ollama FAQ, Ollama API, and llama.cpp CLI options.
- LM Studio system requirements.
- Intel Arc B580 specifications at
https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.htmlreturned HTTP 403. No Intel VRAM figure is cited.
Related
- llm-quantization-explained is the formula.
- vram-requirements applies it by model size.
- 24gb-vs-16gb-vs-12gb-vram applies it by card size.
- best-gpu-for-local-llm names the cards.
- rtx-3090-vs-4090-vs-5090-llm compares three NVIDIA cards.
- local-llm-pc-build covers the rest of the PC.
- run-llm-on-cpu covers CPU and partial offload.
- methodology is the citation rule.
- Mac memory questions: what fits my RAM and how much RAM for a local LLM.