Mixing GPUs for Local LLMs: Different VRAM, AMD and NVIDIA

Overview

Mixing GPUs for LLM inference works when the cards share a backend: llama.cpp’s default layer split gives each GPU a slice of layers and its docs say the split is useful “when GPUs have different memory”, so a 24 GB card and a 12 GB card can share one model. Mixing vendors (NVIDIA with AMD) is a different question: the llama.cpp build guide says multiple backends can be built into one binary, but neither the llama.cpp multi-GPU guide nor the Ollama GPU doc says an NVIDIA and an AMD card are tested together, so that case is unverified here. The memory arithmetic follows the formulas on methodology; the general multi-card picture is on multi-gpu-llm.

Can I mix GPUs with different VRAM sizes for an LLM?

Yes, with llama.cpp. The llama.cpp multi-GPU guide defines --split-mode layer as the default: “Pipeline parallelism. Each GPU holds a contiguous slice of layers.” With --tensor-split (-ts) omitted, “layer/row use automatic splitting proportional to memory”, and you can set shares yourself: “-ts 3,1 is the same as e.g. -ts 75,25.” The guide’s own use case: “Useful when GPUs have different memory: GPU 0 (3 parts) and GPU 1 (1 part).”

What the guide does not say: whether a mismatched pair is tested or recommended, whether the smallest card limits capacity, or any speed figure for mismatched cards. Speed is unknown on this page.

Can I use an RTX 4090 and an RTX 3060 together?

By the llama.cpp rules above, the pairing is a same-vendor, same-backend case, so layer split applies. The vendor pages give the sizes: the RTX 4090 lists 24 GB GDDR6X on a 384-bit bus and 450 W; the RTX 3060 lists 12 GB GDDR6 (an 8 GB variant also exists) on a 192-bit bus and 170 W. No benchmark for this pair is cited by either vendor, so tokens per second is unknown.

Fit math, using the pooled-VRAM idea with a labelled assumption of 1 to 2 GB overhead per card (an assumption, not a measurement, as on vram-requirements):

Worked example with Qwen2.5-32B, using the weights and KV figures on vram-requirements (4.8944 and 8.5008 bits per weight from the llama.cpp quantize README; 64 layers, 8 KV heads, head dimension 128):

The 2:1 split is simple proportion (20.9572 × 2/3 and × 1/3); per-layer sizes are uneven in practice, so treat the per-card numbers as estimates. Layer split also sends work through the slower card for its share of layers, but the guide gives no figure for that, so the cost is unknown.

Can I mix an RTX 4090 and an RTX 4080 for layer splitting?

The same layer-split rules apply, because both are NVIDIA CUDA cards. This page has not read the RTX 4080 spec page, so its VRAM is not stated here; take it from NVIDIA’s page and substitute it into the formula above (pooled = sum of the two cards, minus overhead per card).

Can you have an AMD and an NVIDIA GPU at the same time?

Physically in one PC, yes. Using both for one model is where the documentation is thin:

Practical reading: for a mixed-vendor box, use llama-cli --list-devices and pick devices with -dev, as the llama.cpp CLI README describes, and expect to verify it yourself. In Ollama you pick by vendor variable: CUDA_VISIBLE_DEVICES for NVIDIA, ROCR_VISIBLE_DEVICES for AMD, GGML_VK_VISIBLE_DEVICES for Vulkan.

How does Ollama split a model across 4 AMD GPUs with mixed VRAM sizes?

The Ollama GPU doc does not describe the split algorithm. The evidence is issue #6595: a reporter with two Instinct MI100 (32 GiB each), a Radeon RX 6900 XT (16 GiB) and a Radeon VII (16 GiB), 96 GiB in total, saw a segmentation fault on Ollama 0.3.7 to 0.3.9 when loading a model larger than VRAM. The logs showed a prediction of 45 of 65 layers offloaded, then a crash. A maintainer said the 0.3.14 change was to “stop enabling a workaround in llama.cpp related to GGML_CUDA_NO_PEER_COPY”, another commenter suggested a RAM-versus-model-size cause, the issue was closed as completed on 2024-10-23, and a later comment reported 0.4.0 and 0.4.1 failing with a different error. This is behaviour evidence from one thread, not a statement about current versions.

Lessons that hold from the sources: oversize models that spill to system RAM can need a lot of system memory (maintainer comment), and a run that fits in pooled VRAM avoids that path.

Does a mixed pair run at the speed of the slower card?

Unknown. No primary source read for this page publishes a mixed-GPU throughput figure. What the llama.cpp guide does say is that layer split “can tolerate slow interconnect speeds between GPUs”, so PCIe width matters less for layer split than for tensor split; see multi-gpu-pcie-and-psu. The guide also warns that GGML_CUDA_P2P “may cause crashes or corrupted outputs on some motherboards or BIOS configurations”.

Questions

Can I mix different GPU models (e.g., RTX 4090 + RTX 4080) for layer splitting?

Yes in llama.cpp, whose default layer split divides layers across the visible GPUs in proportion to memory, or by your -ts ratio, and whose guide calls the custom ratio “useful when GPUs have different memory”. The docs publish no tested list and no speed figure for mismatched cards, so confirm with --list-devices. The 4080’s VRAM is not stated on this page.

Can I use two different GPUs, like an RTX 4090 and an RTX 3060?

Yes under llama.cpp layer split. Per NVIDIA, 24 GB plus 12 GB is 36 GB pooled, about 32 to 34 GB after an assumed 1 to 2 GB per card. Qwen2.5-32B at Q4_K_M and 4,096 context needs 20.9572 GB before overhead and fits; at Q8_0 it needs 35.6082 GB and does not. Speed is unknown.

Can you have an AMD and NVIDIA GPU at the same time?

Both can be installed. llama.cpp says multiple backends can be built into one binary and devices chosen with --device, but no source read here documents an NVIDIA plus AMD pairing running one model, and Ollama’s GPU doc is silent. Treat it as unverified until you test with --list-devices.

How does Ollama split a model across 4 AMD GPUs with mixed VRAM sizes?

The Ollama doc does not describe the algorithm. Issue #6595 shows a 4-GPU AMD box (2x 32 GiB, 2x 16 GiB) predicting 45 of 65 layers offloaded and crashing on 0.3.7 to 0.3.9; a maintainer tied the 0.3.14 change to dropping a llama.cpp workaround, and the issue was closed in October 2024. Use ROCR_VISIBLE_DEVICES to choose cards.

Does the smallest card limit the pool?

Unknown. The llama.cpp multi-GPU guide does not say. With layer split each card holds its own slice, so the arithmetic on this page sums the cards and subtracts an assumed overhead per card.

Sources read on 2026-10-13

• llama.cpp multi-GPU guide: split modes, -ts behaviour, “different memory” use case, interconnect, P2P warning; silent on mismatched models and vendors.
• llama.cpp CLI README: -sm, -ts, -dev, --list-devices, -ngl.
• llama.cpp build guide: multiple backends in one build, CUDA plus Vulkan example, GGML_BACKEND_DL.
• llama.cpp SYCL doc: Arc multi-device, same-backend rule, NVIDIA and AMD support removal note.
• Ollama GPU doc: visible-device variables, mixed GFX versions, Vulkan variable; silent on mixing vendors.
• Ollama issue #6595: hardware list, versions, crash, cause, closure.
• NVIDIA RTX 4090 and RTX 3060: memory size, type, bus width, power.
• Qwen2.5-32B figures and bits per weight as read on vram-requirements from the llama.cpp quantize README and the Qwen model card and config.json.

Related

• multi-gpu-llm is the overview of splitting a model across cards.
• multi-gpu-pcie-and-psu covers lanes and power.
• amd-gpu-for-local-llm covers AMD cards.
• dual-rtx-3090-llm is the matched-pair build.
• llama-cpp-vs-ollama-vs-vllm compares the runtimes.
• vram-requirements and methodology hold the formulas.