CUDA Out of Memory in llama.cpp, Ollama and vLLM: Causes and Fixes

Overview

CUDA out of memory in llama.cpp means the GPU could not allocate one more block, and the usual cause is that weights plus KV cache plus overhead grew past your VRAM, most often because the context length was raised. The same cause sits behind Ollama’s num_ctx crashes and vLLM’s “No available memory for the cache blocks”. The memory formula is on vram-requirements and llm-quantization-explained; the citation rule is methodology. This page maps each error message to the term that grew and the setting that shrinks it.

Why do I get CUDA out of memory in llama.cpp?

Total memory is weights, plus KV cache, plus overhead. KV cache bytes are 2 x layers x kv_heads x head_dim x context x bytes_per_value, so they grow in a straight line with context. The llama.cpp CLI README says -c, --ctx-size defaults to 0, “0 = loaded from model”, so with no -c the model’s full trained context is used. In llama.cpp issue #7885 a 4 GB Nvidia A2000 laptop GPU running Phi-3 Mini 128k through Ollama 0.1.42 hit “CUDA error: out of memory” on prompts of about 20k tokens. One maintainer suspected the larger default context (-c 0), another said the extra VRAM came from temporary buffers and suggested -fa, and the reporter found FlashAttention reduced memory use somewhat. The issue was marked stale and closed with no fix made. The KV formula is defined on vram-requirements and llm-quantization-explained.

Worked example from Qwen2.5-32B’s config.json: 64 layers, 8 KV heads, hidden size 5120 over 40 attention heads (head_dim 128), bfloat16 (2 bytes). KV = 2 x 64 x 8 x 128 x 2 = 262,144 bytes per token.

The same model at 131,072 (its max_position_embeddings) is 34.3597 GB of cache alone. Weights for this model are on vram-requirements. Overhead is an assumed 1 to 2 GB there, not a measurement.

Why does raising num_ctx in Ollama run out of VRAM?

Because num_ctx is the context term in the formula above. The Ollama FAQ says the default window is 4096 tokens and can be changed with OLLAMA_CONTEXT_LENGTH or the num_ctx option. In Ollama issue #14116, “Tiered context length can exhaust VRAM”, the reporter describes version 0.15.5 choosing a default from detected VRAM (under 24 GiB: 4,096; 24 to 48 GiB: 32,768; 48 GiB or more: 262,144) and says total VRAM used is proportional to OLLAMA_CONTEXT_LENGTH x OLLAMA_NUM_PARALLEL. A repository collaborator in that thread gave OLLAMA_CONTEXT_LENGTH=4096 as the workaround (it restores the pre-0.15.5 behaviour), and a contributor said he expects OLLAMA_NUM_PARALLEL to increase VRAM use. In issue #9957, a 16 GB card with num_ctx 32768 crashed with “ROCm error: out of memory”; a maintainer found Ollama estimated 15.6 GiB needed against 15.8 GiB available and noted that “ollama estimations are sometimes off”; the reporter later closed it after no crashes on recent versions. That one is AMD, but the mechanism is the same.

If Ollama does not crash but gets slow, it may have split the model to system RAM: ollama ps shows 100% GPU or a split such as 48%/52% CPU/GPU (FAQ). See ollama-not-using-gpu and ram-vs-vram.

What does “no available memory for the cache blocks” mean in vLLM?

After loading weights, vLLM pre-allocates a share of GPU memory for KV cache; the message means nothing was left for it. The error text in vLLM issue #2248 is “No available memory for the cache blocks. Try increasing gpu_memory_utilization when initializing the engine.” The vLLM optimization docs say vLLM “pre-allocates GPU cache using this percentage of memory”, and suggest decreasing max_num_seqs or max_num_batched_tokens, or raising tensor_parallel_size, when cache space is short. The thread also links a memory-profiling change (PR #2031) that counts all occupied GPU memory as belonging to the current instance. Thread replies add causes: CUDA graphs on by default from 0.2.6 (try --enforce-eager), another instance already holding GPU memory (start instances in sequence), a very large max_position_embeddings (one user lowered it to 2048) and --max-num-seqs 1. Those are user reports, not vendor guidance; the thread does not show a merged fix.

Cause, check and fix table

How do I fit a longer context in the same VRAM?

Work through these in order and stop when the error clears.

1. Read your card’s total and free VRAM (how-to-check-vram) and close other GPU programs.
2. Compute the KV term for your model and context with the formula above, and add it to the weight size. If the sum is within about 1 GB of the card, treat it as not fitting.
3. Lower the context: -c in llama.cpp, OLLAMA_CONTEXT_LENGTH or num_ctx in Ollama, max_num_seqs or max_num_batched_tokens in vLLM (vLLM docs; a lower max_position_embeddings was a user report in issue #2248).
4. Quantize the KV cache. The llama.cpp README lists -ctk/-ctv types f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 with f16 the default. The Ollama FAQ says q8_0 uses approximately 1/2 the memory of f16 and q4_0 approximately 1/4, with some precision loss (larger at big contexts for q4_0). It says K/V quantization reduces memory when Flash Attention is enabled, and that Ollama enables Flash Attention automatically when the backend supports it. Halving the 32,768 row gives 4.2950 GB (8.5899 / 2), a quarter gives 2.1475 GB (8.5899 / 4); these assume the FAQ’s ratios hold for this model, which is not measured here.
5. Offload fewer layers to the GPU with -ngl (README: default auto) or accept a partial CPU split; speed cost is unknown here because no vendor figure is read. See run-llm-on-cpu.
6. Choose a smaller quant of the weights (llm-quantization-explained), or a smaller model.

The llama.cpp README also lists --fit (default on), which adjusts unset arguments to fit device memory with a default margin of 1024 (-fitt; the flag’s format string is in MiB) and a minimum fit context of 4096. If you set -c or -ngl yourself, those are your values, not fitted ones.

Questions

Why do I get CUDA out of memory in llama.cpp?

Weights plus KV cache plus overhead exceeded VRAM. KV cache grows with context, and -c defaults to 0, meaning the model’s own context size. Set a smaller -c, quantize the cache with -ctk/-ctv, or lower -ngl. Evidence: llama.cpp issue #7885, where the maintainers named context size and temporary buffers and suggested -fa.

Why does raising num_ctx in Ollama run out of VRAM?

num_ctx sets how many tokens of KV cache Ollama allocates, and that memory is proportional to context and, per issue #14116, to OLLAMA_NUM_PARALLEL. Setting OLLAMA_CONTEXT_LENGTH=4096 is the maintainer-given workaround in that thread, and ollama ps shows whether the model spilled to CPU. Issue #14116 also says larger contexts can push layers into system RAM and slow inference.

What does “no available memory for the cache blocks” mean in vLLM?

After weights load, the share of GPU memory vLLM reserved had no room left for KV cache blocks. Raise gpu_memory_utilization, reduce max_num_seqs or max_num_batched_tokens, free memory held by other processes, or add tensor parallelism (vLLM docs and issue #2248).

How do I fit a longer context in the same VRAM?

Quantize the KV cache, lower the other memory terms, and offload fewer layers. Ollama’s FAQ says q8_0 KV uses approximately half the memory of f16. Calculate first: for Qwen2.5-32B, 32,768 tokens at f16 is 8.5899 GB of cache.

How much does 32,768 tokens of context cost in VRAM?

It depends on the model. For Qwen2.5-32B it is 8.5899 decimal GB at f16 (2 x 64 x 8 x 128 x 2 x 32,768 / 1e9). Other models use their own layers and KV heads from config.json.

Is there a tokens-per-second cost to these fixes?

Unknown. No vendor page read for this article publishes a figure for these settings on a named machine. Issue #14116 says CPU spill slows inference, but gives no figure.

Sources read on 2026-10-13

• vLLM issue #2248: error text, reporter setups, replies (CUDA graphs, --enforce-eager, multiple instances); closed Aug 2025 with no merged fix shown.
• Ollama issue #9957: ROCm OOM at num_ctx 32768 on 16 GB; 15.6G needed vs 15.8G available; estimation caveat.
• Ollama issue #14116: tiered default context, VRAM proportional to context x parallel, OLLAMA_CONTEXT_LENGTH=4096 workaround.
• llama.cpp issue #7885: 4 GB A2000, Phi-3 Mini 128k via Ollama 0.1.42, 20k+ token prompts, maintainer replies; stale then closed Aug 2024.
• llama.cpp CLI README: -c, -ngl, -ctk/-ctv, -fa, --no-kv-offload, --fit defaults.
• Ollama FAQ: 4096 default, OLLAMA_CONTEXT_LENGTH, OLLAMA_KV_CACHE_TYPE, Flash Attention, ollama ps.
• vLLM optimization docs: gpu_memory_utilization, max_num_seqs, max_num_batched_tokens, tensor parallel.
• Qwen2.5-32B config.json: layers 64, KV heads 8, heads 40, hidden size 5120, bfloat16, max positions 131072.

Related

• vram-requirements sizes weights, KV cache and overhead.
• llm-quantization-explained explains bits per weight.
• vram-calculator is planned for this arithmetic.
• ollama-not-using-gpu covers CPU fallback.
• ram-vs-vram and run-llm-on-cpu cover offload.
• how-to-check-vram shows how to read free VRAM.
• methodology is the citation rule.