Best LLM for 12GB VRAM: Is 12GB Enough for Local AI?

Overview

The best LLM for 12GB VRAM is a 7B to 14B model at Q4_K_M with a short context: Qwen2.5-14B at Q4_K_M and 4,096 tokens works out to 10.7988 to 11.7988 decimal GB including an assumed 1 to 2 GB of overhead, while a 27B model is not a confirmed fit even at Q2_K and only fits on paper at the 1 to 2.4 bit IQ quants, whose quality is unknown. Every figure below is arithmetic from the formulas on methodology and vram-requirements using numbers from linked model cards and the llama.cpp README. Tokens per second are unknown on this page because none of the sources read publishes a figure for a named 12 GB card.

How do you check whether a model fits in 12GB?

Add three pieces, in decimal GB (bytes divided by 1e9):

• weights_GB = params x bits / 8 / 1e9
• KV cache bytes = 2 x layers x kv_heads x head_dim x context x bytes_per_value
• overhead = an assumed 1 to 2 GB (an assumption, not a measurement)

Bits per weight come from the llama.cpp quantize README table for Llama 3.1 8B: Q2_K 3.1593, Q3_K_M 3.9960, Q4_K_M 4.8944, Q5_K_M 5.7036, Q6_K 6.5633, Q8_0 8.5008. Applying those values to other models is an estimate; a real GGUF file can differ. Because overhead is a range, this page calls a result a fit only when the top of the range is 12 or less, borderline when the range straddles 12, and not a fit when the bottom of the range exceeds 12.

Is 12GB VRAM enough for AI?

Yes for 7B to 14B models; 27B and up needs sub-3-bit quants or offload. Weights only, from the formula above:

The 7B parameter count is the one used on vram-requirements from its model card; the 14B card lists 14.7B parameters, 48 layers and 8 KV heads.

What is the best LLM for an RTX 3060 12GB?

A 14B at Q4_K_M, or a 7B at a higher quant. NVIDIA’s RTX 3060 page lists “12 GB GDDR6 / 8 GB GDDR6” with a “192-bit / 128-bit” interface and 170 W graphics card power, so check which variant you have with how-to-check-vram. Memory bandwidth is unknown: the page does not state it. Totals at 4,096 tokens (KV from each config.json; Qwen2.5-14B uses 48 layers, 8 KV heads, head dimension 128, 2 bytes per value):

Gemma: the Gemma 4 12B QAT Q4_0 GGUF card lists a 6.98 GB file, 11.95B total parameters and a 256K-token context. 6.98 + 1 to 2 GB overhead is 7.98 to 8.98 GB before KV cache. Its config.json gives 48 layers (40 sliding, 8 global); the window-capped KV term on gemma-vram-requirements is 0.4026 GB at 4,096 tokens and 0.8724 GB at 32,768, so the totals are 8.38 to 9.38 GB and 8.85 to 9.85 GB (an upper bound that caches every layer is 1.6106 GB at 4,096, see 8gb-vram-llm). The card states no VRAM figure. See also qwen-vram-requirements.

Can 12GB VRAM run a 27B model?

Not at Q4_K_M or Q2_K on the GPU alone. Qwen3.8-27B (card: “Number of Parameters: 27B”) is 27e9 x 4.8944 / 8 / 1e9 = 16.5186 GB at Q4_K_M, which already exceeds 12 GB. At Q2_K it is 27e9 x 3.1593 / 8 / 1e9 = 10.6626 GB of weights. Its config.json has 64 layers of which 16 are full_attention (every fourth), 4 KV heads, head dimension 256 and bfloat16. If only those 16 layers hold a KV cache (an assumption), KV at 4,096 is 2 x 16 x 4 x 256 x 4096 x 2 = 0.2684 GB, so 10.6626 + 0.2684 + 1 to 2 = 11.9310 to 12.9310 GB: borderline at best. The README table also lists lower-bit types (IQ1_S 2.0042, IQ2_XXS 2.3824 bits per weight): at IQ1_S the weights are 27e9 x 2.0042 / 8 / 1e9 = 6.7642 GB and at IQ2_XXS 8.0406 GB, so the total is 8.0326 to 9.0326 GB and 9.3090 to 10.3090 GB on paper, but output quality at those bits is unknown here and the real GGUF size can differ. The memory the linear-attention layers need is unknown here. The same holds for the Qwen3-30B-A3B mixture-of-experts (MoE) card: “30.5B in total” and “3.3B activated”. The formula uses total parameters because all experts must be stored, so weights are 18.6599 GB at Q4_K_M; only the compute per token is smaller. Partial offload to system RAM is the fallback, see moe-cpu-offload and ram-vs-vram; speed in that mode is unknown.

Which cards have 12GB?

From NVIDIA pages read on 2026-10-13: the RTX 3060 (12 GB GDDR6 variant, 192-bit), the RTX 4070 family (4070 Ti and 4070 SUPER 12 GB GDDR6X, 192-bit; the 4070 is listed “12 GB GDDR6 / 12 GB GDDR6X”), and the RTX 5070 (12 GB GDDR7, 192-bit, 250 W). Memory bandwidth is not listed on any of these pages, so it is unknown. The same pages list 16 GB siblings: RTX 5070 Ti (16 GB GDDR7, 256-bit) and RTX 4070 Ti SUPER (16 GB GDDR6X, 256-bit). Prices: check current retailer price. More cards on gpu-vram-list.

Is 12GB or 16GB VRAM better for local LLMs?

16 GB is better for models and context, 12 GB is enough for a 7B to 14B at a short context. Qwen2.5-14B Q4_K_M with 8,192 tokens totals 11.6041 to 12.6041 GB (borderline at 12 GB), and Q5_K_M at 4,096 totals 12.2857 to 13.2857 GB, which a 16 GB card holds. Qwen3.8-27B at Q4_K_M weights alone (16.5186 GB) exceeds 16 GB too. Compare 16gb-vram-llm, 8gb-vram-llm and 24gb-vs-16gb-vs-12gb-vram.

Does a second 12GB card help?

On paper two 12 GB cards add to 24 GB (2 x 12), which would hold Qwen2.5-32B Q4_K_M weights of 19.8835 GB (32.5B card parameters, 64 layers, 8 KV heads, head dimension 128 as on vram-requirements) plus 1.0737 GB KV at 4,096. Splitting a model across cards adds its own overhead, and that cost, PCIe effects and speed are unknown here. See multi-gpu-llm.

Questions

Is 12GB VRAM enough for AI?

Yes for 7B to 14B models at a short context; 27B and larger is not a confirmed fit. Qwen2.5-14B at Q4_K_M and 4,096 tokens is 10.7988 to 11.7988 GB including an assumed 1 to 2 GB overhead. At 8,192 tokens the range rises to 11.6041 to 12.6041 GB, which is borderline.

Any 12GB VRAM users out there? What do you run?

This page does not use forum posts as a source for numbers. By the arithmetic here, a 12 GB card runs 7B up to Q8_0 and 14B at Q4_K_M with a short context. Speeds are unknown.

What is the best LLM for an RTX 3060 12GB?

A 14B-class model at Q4_K_M (Qwen2.5-14B totals 10.7988 to 11.7988 GB at 4,096 tokens), or a 7B at Q8_0 (9.3213 to 10.3213 GB). Confirm you have the 12 GB variant, because NVIDIA lists an 8 GB RTX 3060 too.

Is 12GB or 16GB VRAM better for local LLMs?

16 GB, if you want longer context or Q5_K_M on a 14B: Q5_K_M at 4,096 tokens is 12.2857 to 13.2857 GB. For 7B models and 14B at Q4_K_M and a short context, 12 GB is enough. See 16gb-vram-llm.

Can 12GB VRAM run a 27B model?

Not at Q4_K_M or Q2_K. Qwen3.8-27B is 16.5186 GB of weights at Q4_K_M and 10.6626 GB at Q2_K, which totals 11.9310 to 12.9310 GB with KV and overhead under the stated assumptions (borderline); sub-3-bit IQ quants fit on paper but their quality is unknown. Offloading to system RAM is possible but its speed is unknown here.

Does a MoE model use less VRAM than a dense model of the same size?

No for storage. Qwen3-30B-A3B has 30.5B total parameters and 3.3B activated, and the weight formula uses the total, giving 18.6599 GB at Q4_K_M. Only compute per token is smaller.

Sources read on 2026-10-13

• NVIDIA RTX 3060 page: 12 GB GDDR6 / 8 GB GDDR6, 192-bit / 128-bit, 170 W; bandwidth not listed.
• NVIDIA RTX 4070 family: memory sizes, types and interface widths for 4070, SUPER, Ti, Ti SUPER; bandwidth not listed.
• NVIDIA RTX 5070 family: 5070 12 GB GDDR7 192-bit 250 W; 5070 Ti 16 GB GDDR7 256-bit; bandwidth not listed.
• Qwen3.8-27B card and its config.json: 27B, 64 layers, full_attention_interval 4, 4 KV heads, head_dim 256, bfloat16.
• Qwen2.5-14B card and config.json: 14.7B, 48 layers, 8 KV heads, bfloat16 (head dimension 128 = hidden 5120 / 40 heads).
• Qwen2.5-7B card: 7.61B (as read on vram-requirements).
• Qwen3-30B-A3B card: 30.5B total, 3.3B activated.
• Gemma 4 12B QAT Q4_0 GGUF card: 6.98 GB file, 11.95B parameters, 256K context.
• llama.cpp quantize README: bits per weight per quant, including IQ1_S 2.0042 and IQ2_XXS 2.3824.
• Qwen2.5-32B card: 32.5B, 64 layers, 8 KV heads (as read on vram-requirements).

Related

• 8gb-vram-llm, 16gb-vram-llm and 24gb-vs-16gb-vs-12gb-vram compare tiers.
• qwen-vram-requirements, vram-requirements and vram-calculator size models.
• gpu-vram-list lists cards.
• multi-gpu-llm and moe-cpu-offload cover splitting and offload.
• methodology is the citation rule.