How do I install llama-cpp-python with CUDA on a cloud GPU?

Cover: Install llama-cpp-python with CUDA on a cloud GPU, pip, wheels and server

Run CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python on a machine where nvcc --version works, then load your GGUF with Llama(model_path=..., n_gpu_layers=-1); according to the official llama-cpp-python docs, a plain pip install llama-cpp-python builds llama.cpp from source, and GGML_CUDA=on turns on the CUDA backend.

Updated 2026-10-31

This guide follows the official llama-cpp-python README and server docs (version 0.3.36 on PyPI), the llama.cpp build guide and NVIDIA’s CUDA GPUs page, all read on 2026-10-31. It is for Python developers who use llama-cpp-python on their own machine and want the same code on a rented GPU such as the RTX 5090 or the other cards listed on /pricing.

Step 1: Deploy a Raw GPU and connect

On /deploy, pick a GPU and the Raw GPU option (JupyterLab + CUDA pre-installed). Connect over SSH or open a terminal in JupyterLab. VRAM need depends on model, quantization and context size; see How much VRAM do I need to run an LLM? before you choose a card and a GGUF file.

Step 2: Check the GPU, CUDA toolkit and Python first

nvidia-smi                 # GPU name, driver, "CUDA Version" the driver supports
nvcc --version             # CUDA toolkit compiler; the source build needs it
python3 --version          # the pre-built CUDA wheels need 3.10, 3.11 or 3.12
gcc --version
cmake --version

If nvcc is not found, check whether it is installed but not on your PATH (ls /usr/local/cuda*/bin/nvcc) and add that bin folder to PATH. If gcc or cmake is missing, install them with sudo apt-get install -y build-essential cmake (drop sudo if you are root).

Then create a virtual environment so the install does not mix with system packages:

python3 -m venv ~/llama-env
source ~/llama-env/bin/activate
pip install --upgrade pip

Step 3: Install llama-cpp-python with CUDA

You have two routes. Both come from the official README.

Option A: build from source with CUDA (works on any CUDA GPU your toolkit supports)

CMAKE_ARGS="-DGGML_CUDA=on" pip install --verbose llama-cpp-python

According to the official llama-cpp-python docs, every llama.cpp cmake build option can be set through CMAKE_ARGS or -C / --config-settings, and --verbose shows the full cmake build log if the install fails. The llama.cpp build guide adds that the CUDA build needs the CUDA toolkit installed, which is why Step 2 checks nvcc.

On an RTX 5090, NVIDIA’s CUDA GPUs page (read 2026-10-31) lists compute capability 12.0. If the build log shows Cannot find valid GPU for '-arch=native', name the architecture yourself, as the llama.cpp build guide describes for CMAKE_CUDA_ARCHITECTURES:

CMAKE_ARGS="-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=120" \
  pip install --verbose --upgrade --force-reinstall --no-cache-dir llama-cpp-python

A CUDA toolkit that predates Blackwell cannot target 12.0, so if this fails on an RTX 5090, check nvcc --version again.

Option B: a pre-built CUDA wheel (no compile)

The README lists these requirements for the pre-built CUDA wheels:

  • CUDA version 11.8, 12.1, 12.2, 12.3, 12.4, 12.5, 13.0 or 13.2
  • GPU compute capability 6.0 through 8.9 for CUDA 11.8 wheels, 6.0 or newer for CUDA 12 wheels, or 7.5 or newer for CUDA 13 wheels
  • Python 3.10, 3.11 or 3.12

Pick the index that matches your CUDA version (cu118, cu121 to cu125, cu130 or cu132):

pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu125

On 2026-10-31 the Linux files on these indexes were tagged manylinux_2_35_x86_64, so check that ldd --version reports glibc 2.35 or newer. If pip falls back to building from source instead of downloading a wheel, use Option A. On an RTX 5090 (compute capability 12.0), check it yourself: after installing the wheel, run the Step 4 check and the Step 5 script; if Step 4 does not return True or Step 5 fails with no kernel image is available, use Option A with CMAKE_CUDA_ARCHITECTURES=120, the documented path for that card.

Rebuilding after a CPU-only install

If you first installed without CMAKE_ARGS, pip may reuse the cached CPU build. The README says to add --upgrade --force-reinstall --no-cache-dir so the package is rebuilt from source:

CMAKE_ARGS="-DGGML_CUDA=on" pip install --upgrade --force-reinstall --no-cache-dir llama-cpp-python

Step 4: Check that the GPU build is active

python -c "import llama_cpp; print(llama_cpp.__version__, llama_cpp.llama_supports_gpu_offload())"

llama_supports_gpu_offload() is part of the package’s low-level bindings. True means the installed library can offload layers to a GPU; False usually means you got a CPU-only build, so rebuild as shown above.

Step 5: Load a GGUF model on the GPU

pip install huggingface-hub

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="<user>/<model>-GGUF",
    filename="*Q4_K_M.gguf",
    n_gpu_layers=-1,   # put every layer on the GPU
    n_ctx=8192,        # context window
    verbose=True,      # prints the load log and the selected chat format
)

out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Say hello in one sentence."}]
)
print(out["choices"][0]["message"]["content"])

According to the official llama-cpp-python docs:

  • Llama.from_pretrained needs the huggingface-hub package and downloads the GGUF to the Hugging Face cache directory. Use Llama(model_path="/path/model.gguf", ...) for a file you already have.
  • n_gpu_layers=-1 moves all layers to the GPU. Lower it if the model does not fit in VRAM.
  • Chat formatting uses, in order: a chat_handler you pass, a chat_format you pass, the tokenizer.chat_template in the GGUF metadata, and otherwise the llama-2 format. verbose=True shows which one was picked.

While the script runs, nvidia-smi in a second terminal should list the Python process on the GPU.

Step 6: Run the OpenAI-compatible server

CMAKE_ARGS="-DGGML_CUDA=on" pip install 'llama-cpp-python[server]'
python3 -m llama_cpp.server \
  --model /path/model.gguf \
  --n_gpu_layers -1 \
  --n_ctx 8192 \
  --host 127.0.0.1 --port 8000 \
  --api_key "${LLAMA_KEY}"

The server settings file sets n_gpu_layers to 0 by default, so pass --n_gpu_layers -1 or the server runs on the CPU. It also sets host to localhost, port to 8000 and n_ctx to 2048, and “if set all requests need to be authenticated” for --api_key. The server docs say every option can also be set as an environment variable and python3 -m llama_cpp.server --help lists them all. To load straight from Hugging Face, the README shows --hf_model_repo_id <user>/<model>-GGUF --model '*Q4_K_M.gguf'.

Keep the server on 127.0.0.1 and reach it from your own computer through an SSH tunnel, so it is not open to the internet:

ssh -N -L 8000:127.0.0.1:8000 <user>@<instance-host> -p <ssh-port>

Then point any OpenAI SDK client at it, as the server docs show:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="<your LLAMA_KEY>")
r = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Say hello in one sentence."}],
)
print(r.choices[0].message.content)

The interactive API docs are at http://127.0.0.1:8000/docs. To keep the server running after you close SSH, start it inside tmux or screen.

Which GPU should I pick?

Any NVIDIA card on /pricing works with the CUDA build, including the RTX 5090 32G, RTX 4090 24G and 48G, RTX PRO 6000 96G and A100 40G and 80G. Fit is about VRAM: the GGUF file plus the context cache has to fit if you use n_gpu_layers=-1. Use How much VRAM do I need to run an LLM? to size it.

Measurements on CloudGPU coming soon; until then, verify with the nvidia-smi check in Step 5 (the python process listed with GPU memory in use).

Billing and cleanup

Billing is per minute, rounded up to the next whole minute with a 1-minute minimum, and stops when you destroy the instance; see /terms. Copy any files you want to keep (scripts, outputs) to your own machine before you destroy it.

FAQ

How do I install llama-cpp-python with CUDA support?

Run CMAKE_ARGS=“-DGGML_CUDA=on” pip install llama-cpp-python on a machine where nvcc --version works. The official README says pip builds llama.cpp from source, and GGML_CUDA=on turns on the CUDA backend. Add --verbose to see the cmake log if it fails.

Is there a pre-built CUDA wheel for llama-cpp-python?

Yes. The README lists CUDA wheels for CUDA 11.8, 12.1 to 12.5, 13.0 and 13.2 on Python 3.10, 3.11 or 3.12, installed with pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/, for example cu125.

Why does llama-cpp-python not use my GPU?

Usually you have a CPU-only build or n_gpu_layers is 0, which is the server default. Rebuild with CMAKE_ARGS=“-DGGML_CUDA=on” pip install --upgrade --force-reinstall --no-cache-dir llama-cpp-python, check llama_cpp.llama_supports_gpu_offload() returns True, and pass n_gpu_layers=-1.

How do I build llama-cpp-python for an RTX 5090?

NVIDIA lists compute capability 12.0 for the GeForce RTX 5090. Build from source with CMAKE_ARGS=“-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=120” and a CUDA toolkit new enough to target that architecture; check it with nvcc --version first.

What port does the llama-cpp-python server use?

The server defaults to port 8000 on localhost. Start it with python3 -m llama_cpp.server --model /path/model.gguf --n_gpu_layers -1 --port 8000, set --api_key, and reach it from your computer through an SSH tunnel at http://127.0.0.1:8000/v1.