llama.cpp vs Ollama vs vLLM: Which Runtime for Your GPU?

Overview

llama.cpp vs Ollama vs vLLM comes down to hardware and workload: use Ollama for the easiest single-user setup, llama.cpp when you need control over backends, GGUF files and GPU splits, and vLLM when you have a Linux NVIDIA or AMD box and serve several requests at once. Every claim below is from each project’s own README or docs, read on 2026-10-13. Tokens per second are not compared: none of the pages read here publish a figure for a named machine, so speed is unknown. Mac readers should use LM Studio vs Ollama on Mac.

Which one should I use?

llama.cpp vs Ollama: what is actually different?

Ollama lists llama.cpp under its supported backends, so the two are related rather than rivals. The Ollama README links the llama.cpp project there (Ollama README). What Ollama adds is packaging: an installer, a model library and a REST API (/api/chat). What llama.cpp adds is exposure. Its README lists many backends and says it provides “CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity” (llama.cpp README). Whether Ollama gives you the same flags as the llama.cpp command line is not stated in the pages read, so treat it as unknown and check the Ollama docs for each setting.

Which runtime supports my GPU?

Sources: llama.cpp README, Ollama GPU docs, vLLM GPU install docs, vLLM README. AMD specifics are on amd-gpu-for-local-llm, Intel on intel-arc-for-local-llm, and CPU-only on run-llm-on-cpu.

Does Ollama support Vulkan?

Yes, per its current GPU doc. The page says “Additional AMD GPU support is provided by the Vulkan Library,” describes Vulkan as additional GPU support on Windows and Linux, and says it is enabled by default when the backend is installed. You turn it off with OLLAMA_VULKAN=0 or GGML_VK_VISIBLE_DEVICES=-1 (Ollama GPU docs). Older advice that Ollama lacks Vulkan is out of date against this page. If Ollama still ignores your card, see ollama-not-using-gpu.

Does vLLM work on Windows?

Not natively. The vLLM docs say “vLLM does not support Windows natively” and point to WSL with a compatible Linux distribution or a community fork (vLLM install docs). A GitHub thread on RTX 5080/5090 builds has a commenter saying the Docker image builds and runs on Windows through WSL2, but that is one user’s report, not project documentation, and another reported a failed run on an RTX PRO 6000 (vLLM issue 14452). On the Ollama side, its Windows doc lists Windows 10 22H2 or newer and NVIDIA driver 551.61 or newer (Ollama Windows docs). For vLLM in Docker, the docs name the official NVIDIA image vllm/vllm-openai, run with GPU access and --ipc=host, and vllm/vllm-openai-rocm for AMD (vLLM install docs).

Which handles quantized formats better, GGUF or safetensors?

llama.cpp lists “1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization” (llama.cpp README). vLLM lists FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ and GGUF (vLLM README), but its GGUF page calls that support “highly experimental and under-optimized” and says it has moved to the out-of-tree vllm-gguf-plugin (vLLM GGUF docs). In practice: GGUF belongs with llama.cpp, and AWQ/GPTQ/FP8 checkpoints belong with vLLM. Sizing either way follows the same formula, weights_GB = params x bits / 8 / 1e9 plus KV cache plus a labelled overhead (methodology, llm-quantization-explained, worked table on vram-requirements). The file size is the same arithmetic whichever runtime loads it.

How do they handle multiple GPUs?

llama.cpp’s multi-GPU doc: --split-mode layer is the default (each GPU holds a contiguous slice of layers), row is marked deprecated, an older path the doc says has comparatively poor performance, and tensor is experimental and splits weights and KV cache across GPUs. --tensor-split 3,1 gives GPU 0 75% and GPU 1 25%, which the doc calls useful when GPUs have different memory (llama.cpp multi-GPU doc). vLLM lists tensor, pipeline, data, expert and context parallelism (vLLM README). Ollama selects devices with CUDA_VISIBLE_DEVICES, ROCR_VISIBLE_DEVICES or GGML_VK_VISIBLE_DEVICES; its GPU page does not describe split controls (Ollama GPU docs). Hardware side: multi-gpu-llm and mixed-gpus-for-llm.

Is vLLM worth it if you are not serving other people?

For one user chatting with one model, the evidence read here does not show a clear benefit. The features vLLM’s README advertises are batching-oriented: continuous batching of incoming requests, chunked prefill, prefix caching and PagedAttention for KV memory. Those matter most with concurrent requests. Against that, vLLM is Linux-only natively, its GGUF path is experimental, and its pip install targets Python 3.11 to 3.14 and CUDA compute capability 7.5 or higher. Any single-user speed gap is unknown here: no vendor page read publishes a comparison. The thread that prompted this question is Reddit r/LocalLLaMA, which is opinion, not a source for numbers.

vLLM vs SGLang: which for a single GPU box?

Unknown from primary sources. SGLang’s README calls it “an open-source inference framework for large language, vision-language, and diffusion models” and lists NVIDIA, AMD Instinct, Intel, TPU and Apple Silicon platforms. It names no operating systems and publishes no single-GPU comparison with vLLM (SGLang README). TensorRT-LLM was not read for this page, so tensorrt-llm vs vllm is also unknown.

Questions

llama.cpp vs Ollama vs vLLM: which should I use?

Use Ollama for the simplest single-user setup, llama.cpp for backend choice (CUDA, HIP, SYCL, Vulkan), GGUF files and GPU split control, and vLLM for concurrent serving on Linux. All three expose an HTTP API: Ollama a REST API, llama.cpp an OpenAI-compatible server, vLLM an OpenAI-compatible server. Start with the table at the top and your operating system.

Is using vLLM actually worth it if you aren’t serving the model to other people?

Probably not on its documented strengths. Its advertised gains are continuous batching, chunked prefill and prefix caching, which help with many concurrent requests. For one user on a Windows or GGUF-based setup it adds friction: no native Windows support and experimental GGUF. A measured single-user speed difference is unknown on this page.

Does vLLM work on Windows?

Not natively, according to the vLLM docs. They recommend WSL with a Linux distribution or a community fork. A GitHub commenter reports the Docker image runs through WSL2, which is an unofficial report.

Why doesn’t Ollama support Vulkan when llama.cpp does?

It does now, per Ollama’s GPU doc: Vulkan is enabled by default when the backend is installed (the doc covers Windows and Linux), and can be turned off with OLLAMA_VULKAN=0. If you read otherwise, the source predates that page. Intel GPUs appear in that doc only through Vulkan.

vLLM vs SGLang: which for a single GPU box?

Unknown. Neither README published for this page gives a single-GPU comparison, and SGLang’s names no operating systems. Pick by your hardware list and test both with the model you plan to run.

Do I need a different file for each runtime?

Often yes. llama.cpp loads GGUF; vLLM’s natural formats are GPTQ/AWQ/FP8 checkpoints, with GGUF experimental. Ollama’s README does not name the formats it imports, so check its “Importing models” doc. The memory arithmetic is the same: see vram-requirements.

Sources read on 2026-10-13

• llama.cpp README: backend list, hybrid CPU+GPU inference, quantization bit widths, no throughput figures.
• llama.cpp multi-GPU doc: split modes and --tensor-split.
• Ollama README: platforms, llama.cpp backend link, REST API, Docker image.
• Ollama GPU docs: NVIDIA, AMD ROCm v7, Vulkan, device-selection variables.
• Ollama Windows docs: Windows 10 22H2 or newer; NVIDIA driver 551.61 or newer (used in the Windows paragraph).
• vLLM README: quantization list, parallelism, batching, API server, hardware.
• vLLM GPU install docs: Linux only, Windows not native, Python, CUDA and ROCm requirements, Docker images.
• vLLM GGUF docs: experimental status, plugin.
• vLLM issue 14452: user reports on WSL2 and Blackwell builds (not project documentation).
• SGLang README: self-description and hardware list.

Related

• ollama-not-using-gpu when Ollama falls back to CPU.
• best-gpu-for-ollama and amd-gpu-for-local-llm for card choice.
• multi-gpu-llm and mixed-gpus-for-llm for more than one card.
• llm-quantization-explained, vram-requirements and methodology for the fit math.
• intel-arc-for-local-llm and run-llm-on-cpu for non-NVIDIA and CPU paths.
• Mac: LM Studio vs Ollama on Mac.