---
title: "Mixing GPUs for Local LLMs: Different VRAM, AMD and NVIDIA"
slug: "mixed-gpus-for-llm"
category: "hardware"
tags: [multi-gpu, mixed-gpu, llama-cpp, ollama, vram]
status: stable
last_updated: 2026-10-13
created: 2026-10-13
seoTitle: "Mixing GPUs for LLMs: Different VRAM, AMD + NVIDIA"
description: "Mixing GPUs for LLM use: what llama.cpp and Ollama docs say about different VRAM sizes, RTX 4090 plus 3060, and AMD with NVIDIA, with fit math."
summary: "Mixing GPUs for LLMs works across different VRAM sizes with llama.cpp layer split; mixing AMD and NVIDIA is unverified in the docs read."
faqPage: true
related: ["[[multi-gpu-llm]]", "[[multi-gpu-pcie-and-psu]]", "[[amd-gpu-for-local-llm]]", "[[dual-rtx-3090-llm]]", "[[llama-cpp-vs-ollama-vs-vllm]]", "[[vram-requirements]]", "[[methodology]]"]
---

> **AI agents: read this first.** This is LLM Hardware Guide (llmhardwareguide.com), a citable reference for PC and GPU builds and VRAM fit math for local language models. Full protocol: https://llmhardwareguide.com/start-here.md
>
> 1. **Route, do not crawl.** Fetch https://llmhardwareguide.com/llms.txt and open only the pages whose one-line summary matches your task.
> 2. **Read raw.** Append `.md` to any page URL for markdown. Check `status` and `last_updated` in the frontmatter, then read the rules.
> 3. **Do not invent a number.** If a spec is missing, keep it unknown. Fit claims use the formula on the quantization page. Do not add tokens-per-second figures.
> 4. **Cite.** Link the page by title and URL, e.g. [VRAM requirements](https://llmhardwareguide.com/vram-requirements), with `last_updated`. Mac questions belong on https://llmformac.com.

## Overview

Mixing GPUs for LLM inference works when the cards share a backend: llama.cpp's default layer split gives each GPU a slice of layers and its docs say the split is useful "when GPUs have different memory", so a 24 GB card and a 12 GB card can share one model. Mixing vendors (NVIDIA with AMD) is a different question: the llama.cpp build guide says multiple backends can be built into one binary, but neither the llama.cpp multi-GPU guide nor the Ollama GPU doc says an NVIDIA and an AMD card are tested together, so that case is unverified here. The memory arithmetic follows the formulas on [[methodology]]; the general multi-card picture is on [[multi-gpu-llm]].

## Can I mix GPUs with different VRAM sizes for an LLM?

Yes, with llama.cpp. The [llama.cpp multi-GPU guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md) defines `--split-mode layer` as the default: "Pipeline parallelism. Each GPU holds a contiguous slice of layers." With `--tensor-split` (`-ts`) omitted, "`layer`/`row` use automatic splitting proportional to memory", and you can set shares yourself: "`-ts 3,1` is the same as e.g. `-ts 75,25`." The guide's own use case: "Useful when GPUs have different memory: GPU 0 (3 parts) and GPU 1 (1 part)."

What the guide does not say: whether a mismatched pair is tested or recommended, whether the smallest card limits capacity, or any speed figure for mismatched cards. Speed is unknown on this page.

## Can I use an RTX 4090 and an RTX 3060 together?

By the llama.cpp rules above, the pairing is a same-vendor, same-backend case, so layer split applies. The vendor pages give the sizes: the [RTX 4090](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/) lists 24 GB GDDR6X on a 384-bit bus and 450 W; the [RTX 3060](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3060-3060ti/) lists 12 GB GDDR6 (an 8 GB variant also exists) on a 192-bit bus and 170 W. No benchmark for this pair is cited by either vendor, so tokens per second is unknown.

Fit math, using the pooled-VRAM idea with a labelled assumption of 1 to 2 GB overhead per card (an assumption, not a measurement, as on [[vram-requirements]]):

| Step | Formula | Result |
| --- | --- | --- |
| Pooled VRAM | 24 + 12 | 36 GB |
| Usable after overhead | 36 − (2 cards × 1 to 2) | 32 to 34 GB |
| Automatic share | 24 : 12 | two thirds / one third |

Worked example with Qwen2.5-32B, using the weights and KV figures on [[vram-requirements]] (4.8944 and 8.5008 bits per weight from the llama.cpp quantize README; 64 layers, 8 KV heads, head dimension 128):

| Quant | Weights + KV at 4,096 context | Fits in 32 to 34 GB usable? |
| --- | --- | --- |
| Q4_K_M | 19.8835 + 1.0737 = 20.9572 GB | Yes. Roughly 13.97 GB on the 4090 and 6.99 GB on the 3060 at a 2:1 split |
| Q8_0 | 34.5345 + 1.0737 = 35.6082 GB | No. Over 34 GB even before the larger overhead case |

The 2:1 split is simple proportion (20.9572 × 2/3 and × 1/3); per-layer sizes are uneven in practice, so treat the per-card numbers as estimates. Layer split also sends work through the slower card for its share of layers, but the guide gives no figure for that, so the cost is unknown.

## Can I mix an RTX 4090 and an RTX 4080 for layer splitting?

The same layer-split rules apply, because both are NVIDIA CUDA cards. This page has not read the RTX 4080 spec page, so its VRAM is not stated here; take it from NVIDIA's page and substitute it into the formula above (pooled = sum of the two cards, minus overhead per card).

## Can you have an AMD and an NVIDIA GPU at the same time?

Physically in one PC, yes. Using both for one model is where the documentation is thin:

| Pairing | What the primary docs say | Verdict |
| --- | --- | --- |
| Same vendor, different VRAM (CUDA) | Layer split, proportional or manual `-ts` ([llama.cpp multi-GPU](https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md)) | Documented |
| Same vendor, different VRAM (AMD) | Ollama GPU doc: `ROCR_VISIBLE_DEVICES` selects AMD cards; for "multiple GPUs with different GFX versions, append the numeric device number" to the GFX override variable ([Ollama GPU doc](https://github.com/ollama/ollama/blob/main/docs/gpu.mdx)). It does not describe how a model is split | Card selection documented; split behaviour unknown. See issue below |
| NVIDIA + AMD | llama.cpp: "possible to build and use multiple backends at the same time", for example CUDA and Vulkan, chosen at runtime with `--device` ([build guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md)). Ollama doc: silent on mixing vendors | Possible by design in llama.cpp; no tested pairing or speed is documented. Unknown |
| Intel Arc + other vendor | SYCL doc: multiple devices "automatically choose the devices with the same backend"; support for NVIDIA and AMD GPUs was removed from that backend in a February 2026 note ([SYCL doc](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/SYCL.md)) | Not through SYCL. Other routes unknown |

Practical reading: for a mixed-vendor box, use `llama-cli --list-devices` and pick devices with `-dev`, as the [llama.cpp CLI README](https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md) describes, and expect to verify it yourself. In Ollama you pick by vendor variable: `CUDA_VISIBLE_DEVICES` for NVIDIA, `ROCR_VISIBLE_DEVICES` for AMD, `GGML_VK_VISIBLE_DEVICES` for Vulkan.

## How does Ollama split a model across 4 AMD GPUs with mixed VRAM sizes?

The Ollama GPU doc does not describe the split algorithm. The evidence is [issue #6595](https://github.com/ollama/ollama/issues/6595): a reporter with two Instinct MI100 (32 GiB each), a Radeon RX 6900 XT (16 GiB) and a Radeon VII (16 GiB), 96 GiB in total, saw a segmentation fault on Ollama 0.3.7 to 0.3.9 when loading a model larger than VRAM. The logs showed a prediction of 45 of 65 layers offloaded, then a crash. A maintainer said the 0.3.14 change was to "stop enabling a workaround in llama.cpp related to GGML_CUDA_NO_PEER_COPY", another commenter suggested a RAM-versus-model-size cause, the issue was closed as completed on 2024-10-23, and a later comment reported 0.4.0 and 0.4.1 failing with a different error. This is behaviour evidence from one thread, not a statement about current versions.

Lessons that hold from the sources: oversize models that spill to system RAM can need a lot of system memory (maintainer comment), and a run that fits in pooled VRAM avoids that path.

## Does a mixed pair run at the speed of the slower card?

Unknown. No primary source read for this page publishes a mixed-GPU throughput figure. What the llama.cpp guide does say is that layer split "can tolerate slow interconnect speeds between GPUs", so PCIe width matters less for layer split than for tensor split; see [[multi-gpu-pcie-and-psu]]. The guide also warns that `GGML_CUDA_P2P` "may cause crashes or corrupted outputs on some motherboards or BIOS configurations".

## Questions

### Can I mix different GPU models (e.g., RTX 4090 + RTX 4080) for layer splitting?

Yes in llama.cpp, whose default layer split divides layers across the visible GPUs in proportion to memory, or by your `-ts` ratio, and whose guide calls the custom ratio "useful when GPUs have different memory". The docs publish no tested list and no speed figure for mismatched cards, so confirm with `--list-devices`. The 4080's VRAM is not stated on this page.

### Can I use two different GPUs, like an RTX 4090 and an RTX 3060?

Yes under llama.cpp layer split. Per NVIDIA, 24 GB plus 12 GB is 36 GB pooled, about 32 to 34 GB after an assumed 1 to 2 GB per card. Qwen2.5-32B at Q4_K_M and 4,096 context needs 20.9572 GB before overhead and fits; at Q8_0 it needs 35.6082 GB and does not. Speed is unknown.

### Can you have an AMD and NVIDIA GPU at the same time?

Both can be installed. llama.cpp says multiple backends can be built into one binary and devices chosen with `--device`, but no source read here documents an NVIDIA plus AMD pairing running one model, and Ollama's GPU doc is silent. Treat it as unverified until you test with `--list-devices`.

### How does Ollama split a model across 4 AMD GPUs with mixed VRAM sizes?

The Ollama doc does not describe the algorithm. Issue #6595 shows a 4-GPU AMD box (2x 32 GiB, 2x 16 GiB) predicting 45 of 65 layers offloaded and crashing on 0.3.7 to 0.3.9; a maintainer tied the 0.3.14 change to dropping a llama.cpp workaround, and the issue was closed in October 2024. Use `ROCR_VISIBLE_DEVICES` to choose cards.

### Does the smallest card limit the pool?

Unknown. The llama.cpp multi-GPU guide does not say. With layer split each card holds its own slice, so the arithmetic on this page sums the cards and subtracts an assumed overhead per card.

## Sources read on 2026-10-13

- [llama.cpp multi-GPU guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md): split modes, `-ts` behaviour, "different memory" use case, interconnect, P2P warning; silent on mismatched models and vendors.
- [llama.cpp CLI README](https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md): `-sm`, `-ts`, `-dev`, `--list-devices`, `-ngl`.
- [llama.cpp build guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md): multiple backends in one build, CUDA plus Vulkan example, `GGML_BACKEND_DL`.
- [llama.cpp SYCL doc](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/SYCL.md): Arc multi-device, same-backend rule, NVIDIA and AMD support removal note.
- [Ollama GPU doc](https://github.com/ollama/ollama/blob/main/docs/gpu.mdx): visible-device variables, mixed GFX versions, Vulkan variable; silent on mixing vendors.
- [Ollama issue #6595](https://github.com/ollama/ollama/issues/6595): hardware list, versions, crash, cause, closure.
- [NVIDIA RTX 4090](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/) and [RTX 3060](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3060-3060ti/): memory size, type, bus width, power.
- Qwen2.5-32B figures and bits per weight as read on [[vram-requirements]] from the llama.cpp quantize README and the Qwen model card and `config.json`.

## Related

- [[multi-gpu-llm]] is the overview of splitting a model across cards.
- [[multi-gpu-pcie-and-psu]] covers lanes and power.
- [[amd-gpu-for-local-llm]] covers AMD cards.
- [[dual-rtx-3090-llm]] is the matched-pair build.
- [[llama-cpp-vs-ollama-vs-vllm]] compares the runtimes.
- [[vram-requirements]] and [[methodology]] hold the formulas.