Overview
MoE CPU offload means keeping a Mixture of Experts model’s expert weights in system RAM while the GPU holds the rest, so a model whose total size is far over your VRAM can still load; in llama.cpp it is the --cpu-moe and --n-cpu-moe flags. The rule behind it: total parameters decide how much memory the model needs, and active parameters decide how much weight data is read per token. The sizing method is the one on methodology and llm-quantization-explained. No speed figure appears on this page because none of the sources below publishes one with a named machine; tokens per second is unknown.
How does MoE offload to CPU work?
A MoE model stores many expert sub-networks but runs only a few per token. The Qwen3-30B-A3B card describes “30.5B in total and 3.3B activated”, with 128 experts (card). All 30.5B must be stored somewhere. Only 3.3B are used for any one token. Offload puts the stored-but-rarely-touched expert weights in RAM and leaves the always-used layers on the GPU.
Does a MoE model need VRAM for total or active parameters?
Memory capacity follows total parameters; the amount read per token follows active parameters. Using the formula from methodology, weights_GB = params x bits / 8 / 1e9, at the llama.cpp quantize README’s Q4_K_M value of 4.8944 bits per weight:
| Model (card) | Total | Active | Total weights at 4.8944 bits | Active weights at 4.8944 bits | Active share |
|---|---|---|---|---|---|
| gpt-oss-120b | 117B | 5.1B | 71.5806 GB (actual repo: 65.2489 GB) | 3.1202 GB | 4.36% |
| Qwen3.6-35B-A3B | 35B | 3B | 21.4130 GB | 1.8354 GB | 8.57% |
| Qwen3-30B-A3B | 30.5B | 3.3B | 18.6599 GB | 2.0189 GB | 10.82% |
| GLM-4.5-Air | 106B | 12B | 64.8508 GB | 7.3416 GB | 11.32% |
Example: 117e9 x 4.8944 / 8 / 1e9 = 71.5806 GB. These are estimates at one assumed bit width, not real file sizes. gpt-oss-120b ships with “MXFP4 quantization of the MoE weights” per its card, and the quantize README has no MXFP4 line, so the 4.8944-bit estimate overstates it: its repo’s safetensors total 65,248,893,184 bytes, 65.2489 GB (gpt-oss-vram-requirements). Add KV cache and an assumed 1 to 2 GB overhead on top. KV from each config.json at 4,096 tokens and 2 bytes per value: gpt-oss-120b 0.3020 GB (upper bound, every layer treated as full attention), Qwen3.6-35B-A3B 0.0839 GB, Qwen3-30B-A3B 0.4027 GB (48 layers, 4 KV heads, head dimension 128); GLM-4.5-Air’s config was not read, so its KV term is unknown.
The “active” column is why a MoE model can be faster than a dense model of the same total size, but the card sources give no speed number, so by how much is unknown. Do not apply a dense-model bandwidth ceiling to the total size: only the active share is read per token.
How do I offload MoE experts to CPU in llama.cpp?
Use the options in the llama.cpp CLI README:
| Flag | README description | Environment variable |
|---|---|---|
-cmoe, --cpu-moe | ”keep all Mixture of Experts (MoE) weights in the CPU” | LLAMA_ARG_CPU_MOE |
-ncmoe, --n-cpu-moe N | ”keep the Mixture of Experts (MoE) weights of the first N layers in the CPU” | LLAMA_ARG_N_CPU_MOE |
-ngl, --n-gpu-layers N | ”max. number of layers to store in VRAM, either an exact number, ‘auto’, or ‘all’ (default: auto)“ | LLAMA_ARG_N_GPU_LAYERS |
-ot, --override-tensor | ”override tensor buffer type” | LLAMA_ARG_OVERRIDE_TENSOR |
A starting pattern: set -ngl high so all layers are assigned to the GPU, then add --cpu-moe. If VRAM is left over, lower to --n-cpu-moe N and reduce N until the GPU is nearly full, leaving room for KV cache. -ot takes a tensor name pattern for finer control. The README lists these options but gives no tuned values per model, so the best N for your card is something you find by trying it.
Can gpt-oss-120b run on a single 24GB GPU?
The sources do not show it either way, and the weights alone cannot sit in 24 GB. The card says the model can “run on a single 80GB GPU (like NVIDIA H100 or AMD MI300X)“. The repo’s weights are 65.2489 GB (the 4.8944-bit estimate above is 71.5806 GB), over 24 GB either way, so with a 24 GB card most of the experts would have to live in system RAM. The combined VRAM plus RAM must exceed the file size plus KV cache plus overhead. How the 117B splits between expert and non-expert weights is not stated on the card, so how much of the model a 24 GB card would hold is unknown. See gpt-oss-vram-requirements and 24gb-vram-llm.
Should I buy a GPU for dense models or more RAM for MoE models?
Decide by the models you plan to run. A dense model must fit in VRAM to run well, and the full size is read per token. A MoE model can use cheap RAM for capacity, with the GPU doing the shared layers. Check the arithmetic: if your target is gpt-oss-120b at 65.2489 GB of weights, a 24 GB card plus 64 GB of RAM is 88 GB combined before OS and KV cache (24 + 64 = 88), which leaves headroom on paper, but RAM speed and the resulting tokens per second are unknown. See ram-vs-vram and run-llm-on-cpu.
Sources read on 2026-10-13
- gpt-oss-120b card: “117B parameters with 5.1B active parameters”; single 80GB GPU statement; “MXFP4 quantization of the MoE weights”. Experts count and context length not stated on the page.
- Qwen3.6-35B-A3B card: “35B in total and 3B activated”; 256 experts, “8 Routed + 1 Shared”; 40 layers; 262,144 native context.
- Qwen3-30B-A3B card: 30.5B total, 3.3B activated; 128 experts; 48 layers; 32,768 native context.
- GLM-4.5-Air card: 106 billion total, 12 billion active. No hardware or context statement on the page.
- llama.cpp CLI README:
--cpu-moe,--n-cpu-moe,--n-gpu-layers,--override-tensordescriptions and environment variables. - llama.cpp quantize README: Q4_K_M 4.8944 and Q8_0 8.5008 bits per weight; no MXFP4 line.
Questions
How do I offload MoE experts to CPU in llama.cpp?
Add --cpu-moe to keep all MoE weights in system RAM, or --n-cpu-moe N to keep the MoE weights of the first N layers there, per the llama.cpp CLI README. Pair it with -ngl so the other layers go to the GPU, and lower N until VRAM is nearly full.
Should I buy a GPU for dense models or more RAM for MoE models?
Dense models need the whole file in VRAM; MoE models can park experts in RAM. If you target MoE models, RAM capacity matters more, but RAM bandwidth and speed are unknown here, so no speed claim is made. The sizes are on this page, and ram-vs-vram covers the difference.
Does a MoE model need VRAM for total or active parameters?
Capacity follows total parameters, for example 117B for gpt-oss-120b, while per-token reads follow active parameters, 5.1B. At 4.8944 bits that is an estimated 71.5806 GB stored versus 3.1202 GB active; the real MXFP4 repo is 65.2489 GB.
Can gpt-oss-120b run on a single 24GB GPU?
Not by VRAM alone: the card cites a single 80GB GPU and the repo’s weights are 65.2489 GB. With experts offloaded to system RAM it may load if combined memory suffices, but whether it runs well is unknown; no tokens-per-second figure is published in the sources read.
What does n-cpu-moe do?
It keeps the MoE weights of the first N layers in the CPU, per the llama.cpp README. A higher N frees more VRAM and puts more work on the CPU.
Related
- gpt-oss-vram-requirements sizes the gpt-oss family.
- run-llm-on-cpu covers CPU and partial offload.
- ram-vs-vram explains the two memory types.
- qwen-vram-requirements covers Qwen models.
- 24gb-vram-llm covers the 24 GB class.
- llm-quantization-explained and methodology hold the formulas.
- vram-requirements and vram-calculator size dense models.