How do I build llama.cpp with CUDA on a cloud GPU?

Clone the repository, run cmake -B build -DGGML_CUDA=ON and cmake --build build --config Release -j 8, then start ./build/bin/llama-server -hf <user>/<model>-GGUF --host 127.0.0.1 --port 8080; the official llama.cpp build guide says the CUDA build needs the CUDA toolkit (nvcc) installed and, by default, targets the GPU connected at build time.

Updated 2026-10-28

This guide follows the official llama.cpp build guide and llama-server README and NVIDIA’s CUDA GPUs page, all read on 2026-10-28. It is for people who already run GGUF models with llama.cpp on their own machine and want the same setup on a rented GPU such as the RTX 5090 or the other cards listed on /pricing.

Step 1: Deploy a Raw GPU and connect

On /deploy, pick a GPU and the Raw GPU option (JupyterLab + CUDA pre-installed). Connect over SSH or open a terminal in JupyterLab. VRAM need depends on model, precision and workflow; see How much VRAM do I need to run an LLM? before you choose a card and a quantization.

Step 2: Check the environment first

nvidia-smi                 # GPU name, driver, "CUDA Version" the driver supports
nvcc --version             # CUDA toolkit compiler; the CUDA build needs it
cmake --version
git --version

If nvcc is not found, check whether it is installed but not on your PATH (ls /usr/local/cuda*/bin/nvcc) and add that bin folder to PATH. If cmake is missing, install it with sudo apt-get install -y cmake build-essential (drop sudo if you are root).

Step 3: Install OpenSSL headers (needed for -hf downloads)

According to the official llama.cpp build guide, without the OpenSSL development libraries “the project will build and run without SSL support”. Downloading a model with -hf uses HTTPS, so install them before you build:

sudo apt-get update && sudo apt-get install -y libssl-dev

Step 4: Clone and build with CUDA

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j 8

The -j 8 runs eight compile jobs in parallel, as the build guide suggests; set it to the number of CPU cores you have (nproc). To build only the server, add --target llama-server to the second command. The binaries land in build/bin/.

If nvcc cannot detect the GPU

By default the build targets the GPU connected at build time. If you see Cannot find valid GPU for '-arch=native', the build guide offers two fixes:

# Option A: a build for all CUDA GPUs (larger binary, longer build)
cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=OFF

# Option B: name the compute capability yourself
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="120"   # RTX 5090

NVIDIA’s CUDA GPUs page (read 2026-10-28) lists compute capability 12.0 for the GeForce RTX 5090, 8.9 for the GeForce RTX 4090 and 8.0 for the A100, which you write as 120, 89 and 80. On an RTX 5090, a CUDA toolkit that predates Blackwell cannot target 12.0, so check nvcc --version again if the build fails there.

Step 5: Start llama-server with a GGUF model

./build/bin/llama-server \
  -hf <user>/<model>-GGUF:Q4_K_M \
  --host 127.0.0.1 --port 8080 \
  --api-key "${LLAMA_KEY}" \
  -c 8192

According to the official llama-server README:

• -hf downloads a GGUF from a Hugging Face repo; the quant tag is optional and defaults to Q4_K_M, or the first file in the repo if there is no Q4_K_M. Use -hff <file>.gguf to pick an exact file, or -m /path/model.gguf for a file you already have.
• -ngl / --n-gpu-layers sets how many layers go to VRAM (default auto), and --fit (default on) adjusts unset arguments to fit device memory. Pass -ngl all if you want every layer on the GPU.
• -c sets the context size (0 loads it from the model).
• --host defaults to 127.0.0.1. The current README lists port 9931 as the default, so this guide always passes --port 8080.
• --api-key makes requests send Authorization: Bearer <key>.

Keep the server on 127.0.0.1 and reach it through an SSH tunnel from your own computer, so it is not open to the internet:

ssh -N -L 8080:127.0.0.1:8080 <user>@<instance-host> -p <ssh-port>

To keep the server running after you close the SSH session, start it inside tmux or screen.

Step 6: Check health and call the OpenAI-compatible API

curl http://127.0.0.1:8080/health

The README says /health returns HTTP 503 with “Loading model” while the model loads, then HTTP 200 with {"status": "ok"}. Then send a chat request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${LLAMA_KEY}" \
  -d '{"messages":[{"role":"user","content":"Say hello in one sentence."}]}'

Any OpenAI SDK client works the same way with base_url="http://127.0.0.1:8080/v1" and your --api-key value.

Step 7: Confirm the model is on the GPU

While a request runs, open a second terminal and run nvidia-smi. The llama-server process should be listed with GPU memory in use. The server log also prints how many layers were offloaded to the GPU. If the process uses little or no GPU memory, the build did not include CUDA: rebuild with -DGGML_CUDA=ON in a fresh build folder.

Step 8: Save what you need, then destroy the instance

Data on the instance is deleted when you destroy it. Copy anything you want to keep (your own GGUF files, configs, logs) to your computer first. Billing stops when you destroy the instance; see /terms.

Troubleshooting

• HTTPS is not supported when using -hf: the build has no SSL support. Install libssl-dev, delete the build folder and build again.
• nvcc: command not found during cmake: the CUDA toolkit is missing or not on PATH; see Step 2.
• no kernel image is available for execution on the device: the binary was built for a different compute capability. Rebuild with -DCMAKE_CUDA_ARCHITECTURES set for your card, or with -DGGML_NATIVE=OFF.
• CUDA out of memory at load: lower -c, pick a smaller quantization, or leave --fit on and -ngl on auto. The build guide also documents GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 on Linux, which lets llama.cpp swap to system RAM instead of crashing when VRAM runs out (it is slower).
• curl hangs on port 9931 or 8080: check which --port you passed and that your SSH tunnel uses the same port.

Related guides

• How do I install llama-cpp-python with CUDA on a cloud GPU?
• How do I run text-generation-webui (TextGen) on a cloud GPU?
• How do I fix “CUDA error: no kernel image is available” on an RTX 5090?
• How do I use SSH port forwarding on a cloud GPU?
• How do I download a Hugging Face model on a cloud GPU?

FAQ

Do I need the CUDA toolkit to build llama.cpp with GPU support?

Yes. The official llama.cpp build guide says the CUDA build needs the CUDA toolkit installed. Check it with nvcc --version before you run cmake -B build -DGGML_CUDA=ON.

Which CUDA architecture should I use for an RTX 5090?

NVIDIA lists compute capability 12.0 for the GeForce RTX 5090, so pass -DCMAKE_CUDA_ARCHITECTURES=“120” if nvcc cannot detect the GPU. By default llama.cpp builds for the GPU connected at build time, so on the rented machine itself you usually do not need the flag.

What port does llama-server use?

The current llama-server README lists 9931 as the default port and 127.0.0.1 as the default host. Pass --port explicitly, for example --port 8080, so your tunnel and client settings match.

Is llama-server OpenAI-compatible?

Yes. llama-server serves POST /v1/chat/completions, which the README describes as an OpenAI-compatible Chat Completions API, so you can point an OpenAI SDK client at http://127.0.0.1:8080/v1 with your --api-key value.

How do I keep llama-server private on a cloud GPU?

Leave --host at 127.0.0.1, set --api-key, and reach the server from your own computer through an SSH tunnel such as ssh -N -L 8080:127.0.0.1:8080 to the instance.