LLM Hardware Guide

Is 24GB VRAM Enough for Local LLMs? Best Models to Run

By LLM Hardware Guide Editors7 min read

View markdown source

AI agents: read this first: llmhardwareguide.com/start-here

This is LLM Hardware Guide (llmhardwareguide.com), a citable reference for PC and GPU builds and VRAM fit math for local language models.

  1. Route, do not crawl. Fetch llmhardwareguide.com/llms.txt and open only the pages whose one-line summary matches your task.
  2. Read raw. Append .md to any page URL for markdown. Check status and last_updated in the frontmatter, then read the rules.
  3. Do not invent a number. If a spec is missing, keep it unknown. Fit claims use the formula on the quantization page. Do not add tokens-per-second figures.
  4. Cite. Link the page by title and URL, e.g. VRAM requirements, with last_updated. Mac questions belong on llmformac.com.

Overview

The best LLM for 24GB VRAM is a 27B to 31B dense model at Q4_K_M with a short-to-medium context: by the arithmetic below, Gemma 4 31B (about 18.8 GB of weights) and Qwen3.6 27B (about 16.5 GB) both fit, and a 70B-class model does not. 24 GB is a byte budget, so “enough” depends on weights plus KV cache plus overhead. Formulas follow methodology and llm-quantization-explained: weights_GB = params x bits / 8 / 1e9, KV bytes = 2 x layers x kv_heads x head_dim x context x bytes_per_value, and an assumed 1 to 2 GB overhead (an assumption, not a measurement). No tokens-per-second figure is given anywhere on this page: speed is unknown here because no vendor page read for these cards publishes one.

Which cards have 24 GB?

NVIDIA lists the GeForce RTX 3090 and 3090 Ti at “24 GB” GDDR6X on a 384-bit interface (RTX 3090 family), and the RTX 4090 at 24 GB GDDR6X. The AMD Radeon RX 7900 XTX page (read 2026-10-13 after a first HTTP 503) lists 24 GB GDDR6, up to 960 GB/s, 355 W typical board power and an 800 W minimum PSU; it lists no interface width. Memory bandwidth is not listed on the NVIDIA pages: unknown here. Prices: check current retailer price. Card comparison: rtx-3090-llm.

What is the best LLM for 24GB VRAM?

Bits per weight come from the llama.cpp quantize README: Q4_K_M 4.8944 and Q8_0 8.5008 (measured on Llama 3.1 8B there, so these are estimates for other models). Parameters come from each model card.

ModelParams (card)Q4_K_M weightsQ8_0 weightsFits 24 GB at Q4_K_M?
Gemma 4 31B-it30.7B (total)18.7823 GB32.6218 GBYes, short-to-medium context
Qwen3.6-27B27B (card text; Safetensors lists 28B)16.5186 GB (17.1304 at 28B)28.6902 GBYes
gpt-oss-20b21B total, 3.6B activenot estimated (MXFP4)not applicableYes: card says it runs “within 16GB of memory”

Example: 30.7e9 x 4.8944 / 8 / 1e9 = 18.7823 GB. That formula runs low for Gemma: Ollama’s gemma4:31b-it-q4_K_M file is 20 GB (QAT 19 GB), see gemma-vram-requirements, so size from the file when you pull that tag. Q8_0 of either dense model is over 24 GB on weights alone. gpt-oss-20b is distributed with MXFP4 MoE weights, so the K-quant formula does not apply; its file size is unknown here. All three cards are Apache 2.0. Which is “best” depends on your task and no benchmark is quoted here; this page sizes memory only. For Gemma sizes see gemma-vram-requirements.

How much KV cache do these models need at 24 GB?

Values from each config.json, 2 bytes per value (bfloat16 assumed for cache):

  • Gemma 4 31B: 60 layers in a 5 sliding : 1 full pattern, so 50 sliding and 10 full layers. Sliding: 16 KV heads, head_dim 256, window 1024. Full: 4 global KV heads, global head_dim 512. If the runtime caps sliding layers at the window: 50 x 2 x 16 x 256 x 1024 x 2 = 0.8389 GB plus full layers 10 x 2 x 4 x 512 x context x 2. That is 1.5099 GB at 8,192 tokens and 3.5232 GB at 32,768. If the runtime keeps full-length cache on every layer, it is 7.3820 GB at 8,192 and 29.5279 GB at 32,768, which cannot fit. The config sets attention_k_eq_v to true, but how much cache that saves in a given runtime is unknown here, so the figures above do not assume it.
  • Qwen3.6 27B: 64 layers, one full-attention layer in every four, so 16 full layers; 4 KV heads, head_dim 256. 2 x 16 x 4 x 256 x context x 2 = 0.5369 GB at 8,192 and 2.1475 GB at 32,768. The Gated DeltaNet layers keep a recurrent state whose size is unknown here.
  • gpt-oss-20b: 24 layers alternating sliding (128 tokens) and full, taken as 12 each; 8 KV heads, head_dim 64. About 0.2045 GB at 8,192, 0.8085 GB at 32,768 and 3.2244 GB at 131,072.

Totals at 8,192 tokens, Q4_K_M, plus 1 to 2 GB assumed overhead: Gemma 4 31B is 18.7823 + 1.5099 + 1 to 2 = 21.2922 to 22.2922 GB; Qwen3.6 27B is 16.5186 + 0.5369 + 1 to 2 = 18.0555 to 19.0555 GB (excluding DeltaNet state). At 32,768 tokens Gemma is 18.7823 + 3.5232 + 1 to 2 = 23.3055 to 24.3055 GB, which is at or over the line: treat 32K as not a safe fit for Gemma 4 31B on 24 GB. With Ollama’s 20 GB Q4_K_M file instead of the 18.7823 GB estimate, the 8,192-token total is 20 + 1.5099 + 1 to 2 = 22.51 to 23.51 GB.

Is 24GB VRAM enough for AI?

Yes for one 27B to 31B model at Q4 or a 20B MoE with long context, no for 70B-class models or Q8 of anything above about 20B. For image generation and other workloads, VRAM needs are not sized on this page and are unknown here.

Can a 24GB GPU run a 70B model?

Not in VRAM. The README’s Memory/Disk table lists Llama 3.1 70B Q4_K_M at 43.1 GB, so at least 43.1 - 24 = 19.1 GB must sit in system RAM before KV cache and overhead, and offloaded layers run at system-RAM speed (speed unknown here). Full walk-through: vram-for-70b-model. A smaller quant would cut size but its README figure was not read, so it is not stated. MoE models can offload expert weights to RAM more gracefully; see moe-cpu-offload.

Should you buy 24GB or 32GB?

Take 32 GB if you want a 31B model at 32K context or Q8 of a ~27B: Gemma’s 32K case above sits at the 24 GB line. Take 24 GB if 27B at Q4 with moderate context is your target. See 32gb-vram-llm and, below, 16gb-vram-llm for the step down.

Questions

What is the best LLM that runs on a 24 GB VRAM GPU?

A dense 27B to 31B model at Q4_K_M. Gemma 4 31B needs about 21.3 to 22.3 GB at 8,192 tokens and Qwen3.6 27B about 18.1 to 19.1 GB (both with an assumed 1 to 2 GB overhead). No benchmark ranking is quoted here, so “best” means “fits with room”.

What is the best general-purpose model to run locally on 24GB of VRAM in 2026?

Of the three sized here, Qwen3.6 27B leaves the most headroom (about 16.5 GB of Q4_K_M weights) and Gemma 4 31B is the largest that fits. gpt-oss-20b is the lightest option. Quality comparisons are unknown here.

Is 24GB VRAM enough for AI?

For local LLMs up to about 31B at Q4 and moderate context, yes. For 70B-class models, no; they need at least 19.1 GB beyond a 24 GB card on weights alone.

Is 24GB or 32GB better for starters?

24 GB is enough to start with 27B to 31B models at Q4. 32 GB mainly buys longer context or Q8 headroom. Price difference: check current retailer price.

Can a 24GB GPU run a 70B model?

Only with offload to system RAM. Llama 3.1 70B Q4_K_M is a 43.1 GB file, so at least 19.1 GB spills out of VRAM, and speed is unknown here.

Sources read on 2026-10-13

  • Gemma 4 31B-it card and its config.json: 30.7B total parameters, Apache 2.0, 60 layers, sliding window 1024, KV heads 16 (sliding) and 4 (global), head_dim 256 and 512, bfloat16.
  • Qwen3.6-27B card and config.json: 27B (28B in Safetensors), 64 layers, hybrid layout, 4 KV heads, head_dim 256, bfloat16, Apache 2.0.
  • gpt-oss-20b card and config.json: 21B total, 3.6B active, MXFP4, “within 16GB of memory”, 24 layers, 8 KV heads, head_dim 64, sliding window 128.
  • llama.cpp quantize README: Q4_K_M 4.8944 and Q8_0 8.5008 bits per weight, Llama 3.1 70B Q4_K_M 43.1 GB.
  • NVIDIA RTX 3090 family and RTX 4090: 24 GB GDDR6X, 384-bit; no bandwidth listed.
  • AMD Radeon RX 7900 XTX: 24 GB GDDR6, up to 960 GB/s, 355 W, 800 W minimum PSU (read 2026-10-13 after an initial HTTP 503).

Backlinks