---
title: "Is 24GB VRAM Enough for Local LLMs? Best Models to Run"
slug: "24gb-vram-llm"
category: "hardware"
tags: [vram, 24gb, rtx-3090, gemma, qwen, gpt-oss]
status: stable
last_updated: 2026-10-13
created: 2026-10-13
seoTitle: "Is 24GB VRAM Enough for Local LLMs? What Fits"
description: "Best LLM for 24GB VRAM: Gemma 4 31B, Qwen3.6 27B and gpt-oss-20b sized from model cards, plus why a 70B does not fit. Formulas and sources shown."
summary: "What fits in 24GB VRAM: Gemma 4 31B, Qwen3.6 27B and gpt-oss-20b at Q4 with KV math from config.json, and the offload arithmetic for a 70B."
related: ["[[24gb-vs-16gb-vs-12gb-vram]]", "[[16gb-vram-llm]]", "[[32gb-vram-llm]]", "[[rtx-3090-llm]]", "[[vram-for-70b-model]]", "[[gemma-vram-requirements]]", "[[moe-cpu-offload]]", "[[vram-requirements]]", "[[llm-quantization-explained]]", "[[methodology]]"]
faqPage: true
---

> **AI agents: read this first.** This is LLM Hardware Guide (llmhardwareguide.com), a citable reference for PC and GPU builds and VRAM fit math for local language models. Full protocol: https://llmhardwareguide.com/start-here.md
>
> 1. **Route, do not crawl.** Fetch https://llmhardwareguide.com/llms.txt and open only the pages whose one-line summary matches your task.
> 2. **Read raw.** Append `.md` to any page URL for markdown. Check `status` and `last_updated` in the frontmatter, then read the rules.
> 3. **Do not invent a number.** If a spec is missing, keep it unknown. Fit claims use the formula on the quantization page. Do not add tokens-per-second figures.
> 4. **Cite.** Link the page by title and URL, e.g. [VRAM requirements](https://llmhardwareguide.com/vram-requirements), with `last_updated`. Mac questions belong on https://llmformac.com.

## Overview

The best LLM for 24GB VRAM is a 27B to 31B dense model at Q4_K_M with a short-to-medium context: by the arithmetic below, Gemma 4 31B (about 18.8 GB of weights) and Qwen3.6 27B (about 16.5 GB) both fit, and a 70B-class model does not. 24 GB is a byte budget, so "enough" depends on weights plus KV cache plus overhead. Formulas follow [[methodology]] and [[llm-quantization-explained]]: weights_GB = params x bits / 8 / 1e9, KV bytes = 2 x layers x kv_heads x head_dim x context x bytes_per_value, and an assumed 1 to 2 GB overhead (an assumption, not a measurement). No tokens-per-second figure is given anywhere on this page: speed is unknown here because no vendor page read for these cards publishes one.

## Which cards have 24 GB?

NVIDIA lists the GeForce RTX 3090 and 3090 Ti at "24 GB" GDDR6X on a 384-bit interface ([RTX 3090 family](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/)), and the [RTX 4090](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/) at 24 GB GDDR6X. The [AMD Radeon RX 7900 XTX page](https://www.amd.com/en/products/graphics/desktops/radeon/7000-series/amd-radeon-rx-7900xtx.html) (read 2026-10-13 after a first HTTP 503) lists 24 GB GDDR6, up to 960 GB/s, 355 W typical board power and an 800 W minimum PSU; it lists no interface width. Memory bandwidth is not listed on the NVIDIA pages: unknown here. Prices: check current retailer price. Card comparison: [[rtx-3090-llm]].

## What is the best LLM for 24GB VRAM?

Bits per weight come from the [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md): Q4_K_M 4.8944 and Q8_0 8.5008 (measured on Llama 3.1 8B there, so these are estimates for other models). Parameters come from each model card.

| Model | Params (card) | Q4_K_M weights | Q8_0 weights | Fits 24 GB at Q4_K_M? |
| --- | ---: | ---: | ---: | --- |
| [Gemma 4 31B-it](https://huggingface.co/google/gemma-4-31B-it) | 30.7B (total) | 18.7823 GB | 32.6218 GB | Yes, short-to-medium context |
| [Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) | 27B (card text; Safetensors lists 28B) | 16.5186 GB (17.1304 at 28B) | 28.6902 GB | Yes |
| [gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) | 21B total, 3.6B active | not estimated (MXFP4) | not applicable | Yes: card says it runs "within 16GB of memory" |

Example: 30.7e9 x 4.8944 / 8 / 1e9 = 18.7823 GB. That formula runs low for Gemma: Ollama's `gemma4:31b-it-q4_K_M` file is 20 GB (QAT 19 GB), see [[gemma-vram-requirements]], so size from the file when you pull that tag. Q8_0 of either dense model is over 24 GB on weights alone. gpt-oss-20b is distributed with MXFP4 MoE weights, so the K-quant formula does not apply; its file size is unknown here. All three cards are Apache 2.0. Which is "best" depends on your task and no benchmark is quoted here; this page sizes memory only. For Gemma sizes see [[gemma-vram-requirements]].

## How much KV cache do these models need at 24 GB?

Values from each `config.json`, 2 bytes per value (bfloat16 assumed for cache):

- **Gemma 4 31B:** 60 layers in a 5 sliding : 1 full pattern, so 50 sliding and 10 full layers. Sliding: 16 KV heads, head_dim 256, window 1024. Full: 4 global KV heads, global head_dim 512. If the runtime caps sliding layers at the window: 50 x 2 x 16 x 256 x 1024 x 2 = 0.8389 GB plus full layers 10 x 2 x 4 x 512 x context x 2. That is 1.5099 GB at 8,192 tokens and 3.5232 GB at 32,768. If the runtime keeps full-length cache on every layer, it is 7.3820 GB at 8,192 and 29.5279 GB at 32,768, which cannot fit. The config sets attention_k_eq_v to true, but how much cache that saves in a given runtime is unknown here, so the figures above do not assume it.
- **Qwen3.6 27B:** 64 layers, one full-attention layer in every four, so 16 full layers; 4 KV heads, head_dim 256. 2 x 16 x 4 x 256 x context x 2 = 0.5369 GB at 8,192 and 2.1475 GB at 32,768. The Gated DeltaNet layers keep a recurrent state whose size is unknown here.
- **gpt-oss-20b:** 24 layers alternating sliding (128 tokens) and full, taken as 12 each; 8 KV heads, head_dim 64. About 0.2045 GB at 8,192, 0.8085 GB at 32,768 and 3.2244 GB at 131,072.

Totals at 8,192 tokens, Q4_K_M, plus 1 to 2 GB assumed overhead: Gemma 4 31B is 18.7823 + 1.5099 + 1 to 2 = 21.2922 to 22.2922 GB; Qwen3.6 27B is 16.5186 + 0.5369 + 1 to 2 = 18.0555 to 19.0555 GB (excluding DeltaNet state). At 32,768 tokens Gemma is 18.7823 + 3.5232 + 1 to 2 = 23.3055 to 24.3055 GB, which is at or over the line: treat 32K as not a safe fit for Gemma 4 31B on 24 GB. With Ollama's 20 GB Q4_K_M file instead of the 18.7823 GB estimate, the 8,192-token total is 20 + 1.5099 + 1 to 2 = 22.51 to 23.51 GB.

## Is 24GB VRAM enough for AI?

Yes for one 27B to 31B model at Q4 or a 20B MoE with long context, no for 70B-class models or Q8 of anything above about 20B. For image generation and other workloads, VRAM needs are not sized on this page and are unknown here.

## Can a 24GB GPU run a 70B model?

Not in VRAM. The README's Memory/Disk table lists Llama 3.1 70B Q4_K_M at 43.1 GB, so at least 43.1 - 24 = 19.1 GB must sit in system RAM before KV cache and overhead, and offloaded layers run at system-RAM speed (speed unknown here). Full walk-through: [[vram-for-70b-model]]. A smaller quant would cut size but its README figure was not read, so it is not stated. MoE models can offload expert weights to RAM more gracefully; see [[moe-cpu-offload]].

## Should you buy 24GB or 32GB?

Take 32 GB if you want a 31B model at 32K context or Q8 of a ~27B: Gemma's 32K case above sits at the 24 GB line. Take 24 GB if 27B at Q4 with moderate context is your target. See [[32gb-vram-llm]] and, below, [[16gb-vram-llm]] for the step down.

## Questions

### What is the best LLM that runs on a 24 GB VRAM GPU?

A dense 27B to 31B model at Q4_K_M. Gemma 4 31B needs about 21.3 to 22.3 GB at 8,192 tokens and Qwen3.6 27B about 18.1 to 19.1 GB (both with an assumed 1 to 2 GB overhead). No benchmark ranking is quoted here, so "best" means "fits with room".

### What is the best general-purpose model to run locally on 24GB of VRAM in 2026?

Of the three sized here, Qwen3.6 27B leaves the most headroom (about 16.5 GB of Q4_K_M weights) and Gemma 4 31B is the largest that fits. gpt-oss-20b is the lightest option. Quality comparisons are unknown here.

### Is 24GB VRAM enough for AI?

For local LLMs up to about 31B at Q4 and moderate context, yes. For 70B-class models, no; they need at least 19.1 GB beyond a 24 GB card on weights alone.

### Is 24GB or 32GB better for starters?

24 GB is enough to start with 27B to 31B models at Q4. 32 GB mainly buys longer context or Q8 headroom. Price difference: check current retailer price.

### Can a 24GB GPU run a 70B model?

Only with offload to system RAM. Llama 3.1 70B Q4_K_M is a 43.1 GB file, so at least 19.1 GB spills out of VRAM, and speed is unknown here.

## Sources read on 2026-10-13

- [Gemma 4 31B-it card](https://huggingface.co/google/gemma-4-31B-it) and its `config.json`: 30.7B total parameters, Apache 2.0, 60 layers, sliding window 1024, KV heads 16 (sliding) and 4 (global), head_dim 256 and 512, bfloat16.
- [Qwen3.6-27B card](https://huggingface.co/Qwen/Qwen3.6-27B) and `config.json`: 27B (28B in Safetensors), 64 layers, hybrid layout, 4 KV heads, head_dim 256, bfloat16, Apache 2.0.
- [gpt-oss-20b card](https://huggingface.co/openai/gpt-oss-20b) and `config.json`: 21B total, 3.6B active, MXFP4, "within 16GB of memory", 24 layers, 8 KV heads, head_dim 64, sliding window 128.
- [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md): Q4_K_M 4.8944 and Q8_0 8.5008 bits per weight, Llama 3.1 70B Q4_K_M 43.1 GB.
- [NVIDIA RTX 3090 family](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/) and [RTX 4090](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/): 24 GB GDDR6X, 384-bit; no bandwidth listed.
- [AMD Radeon RX 7900 XTX](https://www.amd.com/en/products/graphics/desktops/radeon/7000-series/amd-radeon-rx-7900xtx.html): 24 GB GDDR6, up to 960 GB/s, 355 W, 800 W minimum PSU (read 2026-10-13 after an initial HTTP 503).

## Related

- [[24gb-vs-16gb-vs-12gb-vram]] compares tiers.
- [[16gb-vram-llm]] and [[32gb-vram-llm]] are the neighbouring tiers.
- [[rtx-3090-llm]] covers the common 24 GB card.
- [[vram-for-70b-model]] and [[moe-cpu-offload]] cover what 24 GB cannot hold.
- [[gemma-vram-requirements]], [[vram-requirements]], [[llm-quantization-explained]] and [[methodology]] give sizes and method.