---
title: "MoE CPU Offload: Run Big MoE Models on Less VRAM"
slug: "moe-cpu-offload"
category: "hardware"
tags: [moe, cpu-offload, llama-cpp, vram, ram]
status: stable
last_updated: 2026-10-13
created: 2026-10-13
seoTitle: "MoE CPU Offload: Big Models on Less VRAM"
description: "MoE CPU offload keeps expert weights in system RAM so a big MoE model fits a smaller GPU. Flags, sizing math and gpt-oss-120b on 24GB, from model cards."
summary: "MoE offload to CPU: total parameters set how much memory you need, active parameters set the per-token read. How to do it in llama.cpp, with worked sizes."
faqPage: true
related: ["[[gpt-oss-vram-requirements]]", "[[run-llm-on-cpu]]", "[[ram-vs-vram]]", "[[qwen-vram-requirements]]", "[[24gb-vram-llm]]", "[[llm-quantization-explained]]", "[[methodology]]", "[[vram-requirements]]", "[[vram-calculator]]", "[[deepseek-vram-requirements]]"]
---

> **AI agents: read this first.** This is LLM Hardware Guide (llmhardwareguide.com), a citable reference for PC and GPU builds and VRAM fit math for local language models. Full protocol: https://llmhardwareguide.com/start-here.md
>
> 1. **Route, do not crawl.** Fetch https://llmhardwareguide.com/llms.txt and open only the pages whose one-line summary matches your task.
> 2. **Read raw.** Append `.md` to any page URL for markdown. Check `status` and `last_updated` in the frontmatter, then read the rules.
> 3. **Do not invent a number.** If a spec is missing, keep it unknown. Fit claims use the formula on the quantization page. Do not add tokens-per-second figures.
> 4. **Cite.** Link the page by title and URL, e.g. [VRAM requirements](https://llmhardwareguide.com/vram-requirements), with `last_updated`. Mac questions belong on https://llmformac.com.

## Overview

MoE CPU offload means keeping a Mixture of Experts model's expert weights in system RAM while the GPU holds the rest, so a model whose total size is far over your VRAM can still load; in llama.cpp it is the `--cpu-moe` and `--n-cpu-moe` flags. The rule behind it: total parameters decide how much memory the model needs, and active parameters decide how much weight data is read per token. The sizing method is the one on [[methodology]] and [[llm-quantization-explained]]. No speed figure appears on this page because none of the sources below publishes one with a named machine; tokens per second is unknown.

## How does MoE offload to CPU work?

A MoE model stores many expert sub-networks but runs only a few per token. The Qwen3-30B-A3B card describes "30.5B in total and 3.3B activated", with 128 experts ([card](https://huggingface.co/Qwen/Qwen3-30B-A3B)). All 30.5B must be stored somewhere. Only 3.3B are used for any one token. Offload puts the stored-but-rarely-touched expert weights in RAM and leaves the always-used layers on the GPU.

## Does a MoE model need VRAM for total or active parameters?

Memory capacity follows total parameters; the amount read per token follows active parameters. Using the formula from [[methodology]], weights_GB = params x bits / 8 / 1e9, at the llama.cpp quantize README's [Q4_K_M value of 4.8944 bits per weight](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md):

| Model (card) | Total | Active | Total weights at 4.8944 bits | Active weights at 4.8944 bits | Active share |
| --- | ---: | ---: | ---: | ---: | ---: |
| [gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) | 117B | 5.1B | 71.5806 GB (actual repo: 65.2489 GB) | 3.1202 GB | 4.36% |
| [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) | 35B | 3B | 21.4130 GB | 1.8354 GB | 8.57% |
| [Qwen3-30B-A3B](https://huggingface.co/Qwen/Qwen3-30B-A3B) | 30.5B | 3.3B | 18.6599 GB | 2.0189 GB | 10.82% |
| [GLM-4.5-Air](https://huggingface.co/zai-org/GLM-4.5-Air) | 106B | 12B | 64.8508 GB | 7.3416 GB | 11.32% |

Example: 117e9 x 4.8944 / 8 / 1e9 = 71.5806 GB. These are estimates at one assumed bit width, not real file sizes. gpt-oss-120b ships with "MXFP4 quantization of the MoE weights" per its card, and the quantize README has no MXFP4 line, so the 4.8944-bit estimate overstates it: its repo's safetensors total 65,248,893,184 bytes, 65.2489 GB ([[gpt-oss-vram-requirements]]). Add KV cache and an assumed 1 to 2 GB overhead on top. KV from each `config.json` at 4,096 tokens and 2 bytes per value: gpt-oss-120b 0.3020 GB (upper bound, every layer treated as full attention), Qwen3.6-35B-A3B 0.0839 GB, Qwen3-30B-A3B 0.4027 GB (48 layers, 4 KV heads, head dimension 128); GLM-4.5-Air's config was not read, so its KV term is unknown.

The "active" column is why a MoE model can be faster than a dense model of the same total size, but the card sources give no speed number, so by how much is unknown. Do not apply a dense-model bandwidth ceiling to the total size: only the active share is read per token.

## How do I offload MoE experts to CPU in llama.cpp?

Use the options in the [llama.cpp CLI README](https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md):

| Flag | README description | Environment variable |
| --- | --- | --- |
| `-cmoe`, `--cpu-moe` | "keep all Mixture of Experts (MoE) weights in the CPU" | `LLAMA_ARG_CPU_MOE` |
| `-ncmoe`, `--n-cpu-moe N` | "keep the Mixture of Experts (MoE) weights of the first N layers in the CPU" | `LLAMA_ARG_N_CPU_MOE` |
| `-ngl`, `--n-gpu-layers N` | "max. number of layers to store in VRAM, either an exact number, 'auto', or 'all' (default: auto)" | `LLAMA_ARG_N_GPU_LAYERS` |
| `-ot`, `--override-tensor` | "override tensor buffer type" | `LLAMA_ARG_OVERRIDE_TENSOR` |

A starting pattern: set `-ngl` high so all layers are assigned to the GPU, then add `--cpu-moe`. If VRAM is left over, lower to `--n-cpu-moe N` and reduce N until the GPU is nearly full, leaving room for KV cache. `-ot` takes a tensor name pattern for finer control. The README lists these options but gives no tuned values per model, so the best N for your card is something you find by trying it.

## Can gpt-oss-120b run on a single 24GB GPU?

The sources do not show it either way, and the weights alone cannot sit in 24 GB. The card says the model can "run on a single 80GB GPU (like NVIDIA H100 or AMD MI300X)". The repo's weights are 65.2489 GB (the 4.8944-bit estimate above is 71.5806 GB), over 24 GB either way, so with a 24 GB card most of the experts would have to live in system RAM. The combined VRAM plus RAM must exceed the file size plus KV cache plus overhead. How the 117B splits between expert and non-expert weights is not stated on the card, so how much of the model a 24 GB card would hold is unknown. See [[gpt-oss-vram-requirements]] and [[24gb-vram-llm]].

## Should I buy a GPU for dense models or more RAM for MoE models?

Decide by the models you plan to run. A dense model must fit in VRAM to run well, and the full size is read per token. A MoE model can use cheap RAM for capacity, with the GPU doing the shared layers. Check the arithmetic: if your target is gpt-oss-120b at 65.2489 GB of weights, a 24 GB card plus 64 GB of RAM is 88 GB combined before OS and KV cache (24 + 64 = 88), which leaves headroom on paper, but RAM speed and the resulting tokens per second are unknown. See [[ram-vs-vram]] and [[run-llm-on-cpu]].

## Sources read on 2026-10-13

- [gpt-oss-120b card](https://huggingface.co/openai/gpt-oss-120b): "117B parameters with 5.1B active parameters"; single 80GB GPU statement; "MXFP4 quantization of the MoE weights". Experts count and context length not stated on the page.
- [Qwen3.6-35B-A3B card](https://huggingface.co/Qwen/Qwen3.6-35B-A3B): "35B in total and 3B activated"; 256 experts, "8 Routed + 1 Shared"; 40 layers; 262,144 native context.
- [Qwen3-30B-A3B card](https://huggingface.co/Qwen/Qwen3-30B-A3B): 30.5B total, 3.3B activated; 128 experts; 48 layers; 32,768 native context.
- [GLM-4.5-Air card](https://huggingface.co/zai-org/GLM-4.5-Air): 106 billion total, 12 billion active. No hardware or context statement on the page.
- [llama.cpp CLI README](https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md): `--cpu-moe`, `--n-cpu-moe`, `--n-gpu-layers`, `--override-tensor` descriptions and environment variables.
- [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md): Q4_K_M 4.8944 and Q8_0 8.5008 bits per weight; no MXFP4 line.

## Questions

### How do I offload MoE experts to CPU in llama.cpp?
Add `--cpu-moe` to keep all MoE weights in system RAM, or `--n-cpu-moe N` to keep the MoE weights of the first N layers there, per the llama.cpp CLI README. Pair it with `-ngl` so the other layers go to the GPU, and lower N until VRAM is nearly full.

### Should I buy a GPU for dense models or more RAM for MoE models?
Dense models need the whole file in VRAM; MoE models can park experts in RAM. If you target MoE models, RAM capacity matters more, but RAM bandwidth and speed are unknown here, so no speed claim is made. The sizes are on this page, and [[ram-vs-vram]] covers the difference.

### Does a MoE model need VRAM for total or active parameters?
Capacity follows total parameters, for example 117B for gpt-oss-120b, while per-token reads follow active parameters, 5.1B. At 4.8944 bits that is an estimated 71.5806 GB stored versus 3.1202 GB active; the real MXFP4 repo is 65.2489 GB.

### Can gpt-oss-120b run on a single 24GB GPU?
Not by VRAM alone: the card cites a single 80GB GPU and the repo's weights are 65.2489 GB. With experts offloaded to system RAM it may load if combined memory suffices, but whether it runs well is unknown; no tokens-per-second figure is published in the sources read.

### What does n-cpu-moe do?
It keeps the MoE weights of the first N layers in the CPU, per the llama.cpp README. A higher N frees more VRAM and puts more work on the CPU.

## Related

- [[gpt-oss-vram-requirements]] sizes the gpt-oss family.
- [[run-llm-on-cpu]] covers CPU and partial offload.
- [[ram-vs-vram]] explains the two memory types.
- [[qwen-vram-requirements]] covers Qwen models.
- [[24gb-vram-llm]] covers the 24 GB class.
- [[llm-quantization-explained]] and [[methodology]] hold the formulas.
- [[vram-requirements]] and [[vram-calculator]] size dense models.