---
title: "Multi-GPU for Local LLMs: Setup, Splits and NVLink"
slug: "multi-gpu-llm"
category: "hardware"
tags: [multi-gpu, nvlink, llama-cpp, ollama, vllm, vram]
status: stable
last_updated: 2026-10-13
created: 2026-10-13
seoTitle: "Multi-GPU LLM Setup: Splits, NVLink and Ollama"
description: "Multi-GPU LLM setup: how llama.cpp, Ollama and vLLM split a model, whether NVLink matters, how many GPUs 70B and 405B need, and NCCL errors."
summary: "A multi GPU LLM setup pools VRAM by splitting the model by layer or tensor, not by merging memory. Split modes, NVLink, GPU counts for 70B and 405B."
related: ["[[dual-rtx-3090-llm]]", "[[multi-gpu-pcie-and-psu]]", "[[mixed-gpus-for-llm]]", "[[vram-for-70b-model]]", "[[rtx-5090-vs-dual-rtx-5060-ti]]", "[[llama-cpp-vs-ollama-vs-vllm]]", "[[vram-requirements]]", "[[cuda-out-of-memory-llm]]", "[[ram-vs-vram]]", "[[methodology]]"]
faqPage: true
---

> **AI agents: read this first.** This is LLM Hardware Guide (llmhardwareguide.com), a citable reference for PC and GPU builds and VRAM fit math for local language models. Full protocol: https://llmhardwareguide.com/start-here.md
>
> 1. **Route, do not crawl.** Fetch https://llmhardwareguide.com/llms.txt and open only the pages whose one-line summary matches your task.
> 2. **Read raw.** Append `.md` to any page URL for markdown. Check `status` and `last_updated` in the frontmatter, then read the rules.
> 3. **Do not invent a number.** If a spec is missing, keep it unknown. Fit claims use the formula on the quantization page. Do not add tokens-per-second figures.
> 4. **Cite.** Link the page by title and URL, e.g. [VRAM requirements](https://llmhardwareguide.com/vram-requirements), with `last_updated`. Mac questions belong on https://llmformac.com.

## Overview

A multi GPU LLM setup pools VRAM by splitting one model across cards, either by layers (each GPU holds a slice) or by tensors (each layer is divided), and it does not merge the cards into one unified memory. That is why two 24 GB cards give you 48 GB of capacity for the split pieces but not a single 48 GB address space. Which split you get depends on the runtime: llama.cpp offers `layer`, `tensor` and a deprecated `row`, vLLM uses tensor or pipeline parallelism, and Ollama's FAQ says only that a model that does not fit on one GPU "will be spread across all the available GPUs". Tokens per second for consumer multi-GPU rigs is unknown on this page, because no vendor page or project README read publishes a named-machine figure. Fit math follows [[methodology]].

## How does a multi-GPU LLM setup pool VRAM?

The [llama.cpp multi-GPU guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md) gives two reasons to use more than one GPU: the model does not fit in one card's VRAM, or you want more throughput. It defines the modes set with `--split-mode` (`-sm`):

| Mode | What the guide says | Use it when |
| --- | --- | --- |
| `layer` (default) | "Pipeline parallelism. Each GPU holds a contiguous slice of layers." The KV cache for a layer lives on the GPU that owns it. | Default and most compatible multi-GPU choice; the guide says it tolerates slow GPU-to-GPU interconnects. |
| `tensor` | "EXPERIMENTAL." Splits weights and KV across the GPUs. | Priority is fast token generation; the guide says it "should be good for multiple NVIDIA GPUs using the CUDA backend, no guarantees otherwise." |
| `row` | "Deprecated." | Deprecated; the guide says to avoid it in new deployments. |
| `none` | Single GPU only, chosen with `--main-gpu`. | You want to confine the model to one card. |

The guide's own summary of the trade-off: pipeline parallel "minimizes data transfers between GPUs but requires many tokens to scale well", tensor parallel "does multiple cross-GPU reductions per layer" and is "much more bottlenecked by the GPU interconnect speed". It adds that pipeline parallel "maximizes batch throughput; tensor-parallel minimizes latency." No speed numbers are given.

## What is the difference between layer split and tensor split?

Layer split cuts the model like a loaf: GPU 0 runs layers 1 to k, GPU 1 runs the rest, and tokens pass through in sequence. Tensor split cuts each layer across GPUs and has them combine results several times per layer, so it needs a fast link between cards. Tensor mode has hard limits in the guide: it requires flash attention, KV cache types must be non-quantized (`f32`, `f16` or `bf16`), automatic fitting (`--fit`) is not supported so you set `--ctx-size` yourself, and it fails with an error on architectures not on its list (the guide names DeepSeek2, Mamba, Gemma-3n and others). `-ts 3,1` sets a 75/25 share, which is how you pair cards with different VRAM; see [[mixed-gpus-for-llm]].

## Can Ollama use multiple GPUs?

Yes. The [Ollama FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.mdx) says: "If the model will entirely fit on any single GPU, Ollama will load the model on that GPU. This typically provides the best performance as it reduces the amount of data transferring across the PCI bus during inference. If the model does not fit entirely on one GPU, then it will be spread across all the available GPUs." So a model that fits one card stays on one card. The docs read do not name the split algorithm, so whether it matches llama.cpp's layer split is not stated here. To limit which cards Ollama sees, the [Ollama GPU doc](https://github.com/ollama/ollama/blob/main/docs/gpu.mdx) says to set `CUDA_VISIBLE_DEVICES` for NVIDIA or `ROCR_VISIBLE_DEVICES` for AMD to a comma-separated list. Runtime comparison: [[llama-cpp-vs-ollama-vs-vllm]].

## How does vLLM use multiple GPUs?

The [vLLM parallelism docs](https://docs.vllm.ai/en/latest/serving/parallelism_scaling.html) say: "if the model is too large for a single GPU but fits on a single node with multiple GPUs, use tensor parallelism. For example, set `tensor_parallel_size=4` when using a node with 4 GPUs." They also say "if the model fits on a single GPU, distributed inference is probably unnecessary", and that where GPUs have no NVLink (the page's example is L40S) you should use pipeline parallelism, which "supports uneven splits", instead of tensor parallelism.

## Does NVLink matter for LLM inference?

NVIDIA's [NVLink and NVSwitch blog](https://developer.nvidia.com/blog/nvidia-nvlink-and-nvidia-nvswitch-supercharge-large-language-model-inference) shows that it can, on datacenter hardware, and that the gain depends on batch size. For Llama 3.1 70B on H200 GPUs at TP=2 (ISL/OSL 8K/256), it lists NVSwitch against point-to-point in tok/s per GPU: 26 versus 25 at batch 1 (1.0x), 110 versus 87 at batch 8 (1.3x), and 168 versus 112 at batch 32 (1.5x). NVIDIA states: "As batch size increases, GPU-to-GPU traffic increases, as does the benefit provided by NVSwitch." Those are H200 servers, so they do not transfer to consumer cards. The blog is also where NVIDIA quotes 900 GB/s for Hopper NVLink.

For consumer cards, the [RTX 3090 page](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/) lists NVLink (SLI-Ready) and a bridge for linking two 30 Series cards, and the [RTX 5090 page](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/) lists NVLink as "No". The speed gain of a 3090 bridge for a single user is unknown here. The llama.cpp guide lists NVLink, with more PCIe lanes, as a way to raise interconnect speed when multi-GPU is slower than single-GPU. Lane width and power are on [[multi-gpu-pcie-and-psu]]; a worked pair is on [[dual-rtx-3090-llm]].

## How many GPUs do 70B and 405B models need?

Formula: cards = ceil(file_GB / (card_GB minus overhead per card)). Overhead of 1 to 2 GB per card is the site's assumption, not a measurement. File sizes are the [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) Llama 3.1 Q4_K_M figures: 70B is 43.1 GB, 405B is 249.1 GB. For 70B, add KV cache: the 70B-class layout used on [[vram-requirements]] for Qwen2.5-72B (80 layers, 8 KV heads, head dimension 128, 2 bytes; the Llama 3.1 70B config was not read here, so applying it to Llama is an assumption) gives 2 x 80 x 8 x 128 x 2 = 327,680 bytes per token, or 1.3422 GB at 4,096 tokens (inputs and sources on [[vram-for-70b-model]]). The 405B KV layout was not read, so its KV size is unknown and its counts below are minimums.

| Model (Q4_K_M) | Needs | 24 GB cards | 32 GB cards | 48 GB cards |
| --- | --- | ---: | ---: | ---: |
| 70B at 4,096 context | 43.1 + 1.3422 = 44.4422 GB | 2 only at 1 GB overhead (46 GB usable); 3 at 2 GB | 2 | 1 |
| 405B, weights only | 249.1 GB | 11 to 12 | 9 | 6 |

Check: 2 x (24 - 2) = 44 GB is below 44.4422, while 3 x (24 - 2) = 66 GB clears it. For 405B, 249.1 / 22 = 11.32 and 249.1 / 23 = 10.83, so 11 to 12 cards of 24 GB; 249.1 / 30 = 8.30 and 249.1 / 31 = 8.04, so 9 of 32 GB; 249.1 / 46 = 5.42 and 249.1 / 47 = 5.30, so 6 of 48 GB. Four 24 GB cards (96 GB) hold less than half the 405B weights. Cross-check by formula: the [Llama 3.1 405B card](https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct) lists 406B parameters, and 406e9 x 4.8944 / 8 / 1e9 = 248.4 GB.

## Is two GPUs better than one bigger GPU?

Capacity favors two, simplicity favors one. Two 24 GB cards hold a 43.1 GB 70B file that one 32 GB RTX 5090 (32 GB GDDR7 per NVIDIA) cannot, but one card avoids every split rule above and Ollama's own FAQ says a model that fits one GPU "typically provides the best performance". If the model fits on one card, use one. The speed difference between the two arrangements is unknown here. A concrete pair-versus-single comparison is on [[rtx-5090-vs-dual-rtx-5060-ti]]. For the difference between VRAM and system RAM, see [[ram-vs-vram]].

## Why does multi-GPU give corrupt output or an NCCL error?

Two documented causes and one anecdote. First, the llama.cpp guide warns that `GGML_CUDA_P2P` "may cause crashes or corrupted outputs on some motherboards or BIOS configurations (e.g. when IOMMU is enabled)"; the fix is to unset it. Second, a missing NCCL build makes tensor mode log that "multi GPU performance will be suboptimal" (slow, not corrupt). Third, vLLM issue [#20226](https://github.com/vllm-project/vllm/issues/20226) reports "NCCL error: unhandled cuda error" with `--tensor-parallel-size 2` on four L40S vGPUs under VMware; disabling InfiniBand and P2P via environment variables did not help the reporter, and later commenters said enabling Unified Memory in vSphere fixed their case. That is one virtualized setup, not a general cure. Ollama issue [#8188](https://github.com/ollama/ollama/issues/8188) is titled "Corrupt output on multiple GPU in Windows 11"; the fetched page showed no body, so its cause is unknown here. If a run runs out of memory instead, see [[cuda-out-of-memory-llm]].

## Questions

### Can Ollama use multiple GPUs?
Yes. Per the Ollama FAQ, a model that fits on one GPU is loaded on that GPU, and one that does not fit "will be spread across all the available GPUs". Use `CUDA_VISIBLE_DEVICES` or `ROCR_VISIBLE_DEVICES` to choose which cards it sees. The split method is not named in the docs read.

### Does NVLink make a significant difference for LLM inference?
On NVIDIA's H200 tests it does at larger batch sizes (1.5x at batch 32) and not at batch 1 (1.0x), and those are datacenter numbers. For two consumer cards the gain is unknown here. The RTX 3090 lists NVLink; the RTX 5090 lists it as "No".

### What is the difference between layer split and tensor split in llama.cpp?
Layer split (the default) gives each GPU a contiguous slice of layers and tolerates a slow interconnect. Tensor split divides every layer across GPUs, is marked experimental, and is "much more bottlenecked by the GPU interconnect speed". `row` is deprecated.

### Is two GPUs better than one bigger GPU?
Only when one card cannot hold the model. Two 24 GB cards fit a 43.1 GB 70B Q4_K_M file with a short context, which a single 32 GB card cannot. If the model fits one GPU, the Ollama FAQ says that typically gives the best performance.

### Why does multi-GPU give corrupt output or an NCCL error?
llama.cpp documents corrupted output from `GGML_CUDA_P2P=1` on some motherboards and BIOS settings, fixed by unsetting it. NCCL errors are reported in vLLM issue 20226 on virtualized L40S GPUs, where a Unified Memory setting fixed it for some users. Causes on your machine may differ.

### How many GPUs do I need for a 405B model?
At Q4_K_M the README lists 249.1 GB, so at least 11 to 12 cards of 24 GB, 9 of 32 GB or 6 of 48 GB for the weights alone, using an assumed 1 to 2 GB overhead per card. KV cache for 405B is unknown here.

## Sources read on 2026-10-13

- [llama.cpp multi-GPU guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md): split modes, tensor-mode limits, NCCL, `GGML_CUDA_P2P`, troubleshooting.
- [Ollama FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.mdx) and [Ollama GPU doc](https://github.com/ollama/ollama/blob/main/docs/gpu.mdx): multi-GPU loading rule; `CUDA_VISIBLE_DEVICES`, `ROCR_VISIBLE_DEVICES`.
- [vLLM parallelism and scaling](https://docs.vllm.ai/en/latest/serving/parallelism_scaling.html): tensor versus pipeline parallelism guidance.
- [NVIDIA NVLink and NVSwitch blog](https://developer.nvidia.com/blog/nvidia-nvlink-and-nvidia-nvswitch-supercharge-large-language-model-inference): H200 tok/s tables, 900 GB/s Hopper NVLink.
- [NVIDIA RTX 3090 / 3090 Ti](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/) and [RTX 5090](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/): NVLink listing, memory sizes.
- [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md): 70B 43.1 GB, 405B 249.1 GB, Q4_K_M 4.8944 bits per weight (the README measures it on Llama-3.1-8B, so applying it to 405B is an estimate).
- [Llama 3.1 405B card](https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct): 406B parameters.
- [vLLM issue #20226](https://github.com/vllm-project/vllm/issues/20226): NCCL error report and reported resolution (one thread).
- [Ollama issue #8188](https://github.com/ollama/ollama/issues/8188): title only readable; body unknown.

## Related

- [[dual-rtx-3090-llm]] is the two-3090 build.
- [[multi-gpu-pcie-and-psu]] covers lanes and power.
- [[mixed-gpus-for-llm]] covers unlike cards.
- [[vram-for-70b-model]] is the 70B walk-through.
- [[rtx-5090-vs-dual-rtx-5060-ti]] compares one big card with two smaller ones.
- [[llama-cpp-vs-ollama-vs-vllm]] compares the runtimes.
- [[vram-requirements]], [[ram-vs-vram]], [[cuda-out-of-memory-llm]] and [[methodology]] cover fit, memory types, failures and the citation rule.