Overview
The best GPU for a local LLM is the one whose VRAM holds the weight file you will actually run, plus KV cache. A higher shader count does not change the fit arithmetic on llm-quantization-explained.
What is the best GPU for running local LLMs?
Choose by VRAM against the weight size, then check the vendor power figure. There is no single winner, and this page does not rank cards by tokens per second. None of the NVIDIA product pages fetched on 2026-10-12 published an LLM tokens-per-second figure. NVIDIA’s listed “AI TOPS” on the RTX 5090 page (3352 for 5th-generation tensor cores) is that page’s tensor-core spec, not an LLM speed, and it is not converted into one here.
How much VRAM do I need?
Use vram-requirements. Short form, weights only, before KV and before the 1 to 2 GB overhead assumption:
- Qwen2.5-7B Q4_K_M estimate: 4.6558 decimal GB. Llama 3.1 8B Q4_K_M file: 4.9 GB in the quantize README.
- Qwen2.5-14B Q4_K_M estimate: 8.9935 GB.
- Qwen2.5-32B Q4_K_M estimate: 19.8835 GB. Q8_0 estimate: 34.5345 GB.
- Llama 3.1 70B Q4_K_M file: 43.1 GB. Qwen2.5-72B Q4_K_M estimate: 44.4779 GB.
NVIDIA, AMD or Intel for local LLMs?
NVIDIA and AMD figures below are from the vendor pages read on 2026-10-12. Intel Arc is unknown. https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html returned HTTP 403 that day, so this page does not state an Arc VRAM size.
Ollama’s GPU document says NVIDIA GPUs can be limited with CUDA_VISIBLE_DEVICES, lists AMD cards it supports through ROCm (including Radeon RX 7900 XTX and RX 9070 XT), and says further AMD support comes from Vulkan. That is a software-support statement from Ollama, not a speed claim.
RTX 5090
NVIDIA’s RTX 5090 page: 32 GB GDDR7, 512-bit interface, 21760 CUDA cores, boost clock 2.41 GHz, total graphics power 575 W, required system power 1000 W on a PC configured with a Ryzen 9 9950X. The same page’s prepare section also says “850W Minimum” and that a higher rating may be required. NVLink is No. Card length 304 mm, width 137 mm, 2-slot. Memory bandwidth is not listed on the page fetched that day, so bandwidth is unknown. Check current retailer price.
RTX 4090
NVIDIA’s RTX 4090 page: 24 GB GDDR6X, 384-bit interface, 16384 CUDA cores, boost clock 2.52 GHz, total graphics power 450 W, required system power 850 W on a Ryzen 9 5900X. Length 304 mm, width 137 mm, 3-slot. NVLink is No. Average gaming power 315 W is measured across 22 games, not an LLM workload. Memory bandwidth is not on the fetched page, so it is unknown. Check current retailer price.
RTX 3090
NVIDIA’s RTX 3090 family page, GeForce RTX 3090 column (not the 3090 Ti column): 24 GB GDDR6X, 384-bit, 10496 CUDA cores, boost clock 1.70 GHz, graphics card power 350 W, required system power 750 W on a Core i9-10900K. The footnote says a lower rating may work depending on the system. NVLink (SLI-Ready) is Yes. Length 313 mm, 3-slot. Memory bandwidth is not in that table, so it is unknown.
Radeon RX 7900 XTX
AMD’s RX 7900 XTX page: 24 GB GDDR6, memory bandwidth up to 960 GB/s, and a separate line “effective memory bandwidth” up to 3500 GB/s. Infinity Cache 96 MB. 96 compute units. Typical board power 355 W. Minimum PSU recommendation 800 W, based on a Ryzen 9 5900X (AMD note GD-223). Memory interface width is not on the fetched spec list, so it is unknown.
Radeon RX 9070 XT
AMD’s RX 9070 XT page: 16 GB GDDR6, 256-bit interface, memory bandwidth up to 640 GB/s, Infinity Cache 64 MB, 4096 stream processors, memory speed up to 20 Gbps. Typical board power 304 W. Minimum PSU recommendation 750 W.
RTX 4060 Ti and RTX 4060
NVIDIA’s RTX 4060 family page: the 4060 Ti column is 16 GB GDDR6 or 8 GB GDDR6, 128-bit, 4352 CUDA cores, total graphics power 165 W or 160 W. The 4060 column is 8 GB GDDR6, 128-bit, 3072 CUDA cores, total graphics power 115 W. Required system power is 550 W for both, based on a Ryzen 9 5900X. Memory bandwidth is not in that table, so it is unknown.
Is one big GPU better than two smaller ones?
One GPU avoids a split. Ollama’s FAQ says that if a model fits entirely on any single GPU, Ollama loads it on that GPU, and if it does not, Ollama spreads it across the available GPUs. llama.cpp’s CLI README documents --split-mode: none uses one GPU, layer (the default) splits layers and KV across GPUs, and row splits weights across GPUs by rows. --tensor-split takes a comma-separated list of proportions.
Two 24 GB cards are 48 GB of VRAM only if the runtime can use both. Llama 3.1 70B Q4_K_M is a 43.1 GB file, which leaves 48 − 43.1 = 4.9 GB for KV cache and overhead. That margin is real as subtraction and still depends on the split actually working. This page does not claim one layout is faster. The Mac alternative is which Mac for a local LLM.
RTX 5090
32 GB GDDR7. See the vendor section above. Weight fit is on vram-requirements and rtx-3090-vs-4090-vs-5090-llm.
RTX 4090
24 GB GDDR6X.
RTX 3090
24 GB GDDR6X, with NVLink listed as yes on NVIDIA’s family page.
Radeon RX 7900 XTX
24 GB GDDR6, bandwidth up to 960 GB/s on AMD’s page.
Radeon RX 9070 XT
16 GB GDDR6, bandwidth up to 640 GB/s on AMD’s page.
RTX 4060 Ti
16 GB or 8 GB GDDR6.
RTX 4060
8 GB GDDR6.
Sources read on 2026-10-12
- llama.cpp quantize README for bits per weight, GiB sizes, and the Memory/Disk table.
- Qwen2.5-7B, 14B, 32B, and 72B model cards, plus each repo’s
config.json, for parameter counts, layers, heads, hidden size, andtorch_dtype. - RTX 5090, RTX 4090, RTX 3090 family, and RTX 4060 family.
- Radeon RX 7900 XTX and Radeon RX 9070 XT.
- Ollama GPU, Ollama FAQ, Ollama API, and llama.cpp CLI options.
- LM Studio system requirements.
- Intel Arc B580 specifications at
https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.htmlreturned HTTP 403. No Intel VRAM figure is cited.
Related
- llm-quantization-explained is the formula.
- vram-requirements applies it by model size.
- 24gb-vs-16gb-vs-12gb-vram applies it by card size.
- best-gpu-for-local-llm names the cards.
- rtx-3090-vs-4090-vs-5090-llm compares three NVIDIA cards.
- local-llm-pc-build covers the rest of the PC.
- run-llm-on-cpu covers CPU and partial offload.
- methodology is the citation rule.
- Mac memory questions: what fits my RAM and how much RAM for a local LLM.