How do I serve an OpenAI-compatible API with vLLM on a cloud GPU?

Short answer: Deploy a Raw GPU, install vLLM in a Python virtual environment with uv pip install vllm --torch-backend=auto, then run vllm serve <model> --api-key <key>. According to the vLLM quickstart, the server starts at http://localhost:8000 by default and serves OpenAI endpoints such as /v1/models and /v1/chat/completions (vLLM quickstart, read 2026-11-03). Forward port 8000 over SSH and point any OpenAI client at http://localhost:8000/v1.

Step 1: Pick a card and deploy a Raw GPU

The weights plus the KV cache must fit on one GPU. Size the model with how much VRAM an LLM needs; VRAM need depends on model, precision and workflow. vLLM needs an NVIDIA GPU with compute capability 7.5 or higher; the cards CloudGPU lists for vLLM are on /rent-gpu-for-vllm.

On /deploy, choose Raw GPU (JupyterLab and CUDA, root SSH) and a card. This guide installs vLLM yourself so you control the version and the flags; check /deploy for the current template list. To have the install run on its own when the instance is handed over, paste Step 2 into the optional startup script (how to run a setup script when a cloud GPU starts).

Step 2: Install vLLM

nvidia-smi                       # GPU, driver, and the CUDA version the driver supports
pip install -U uv
uv venv --python 3.12 --seed ~/vllm-env
source ~/vllm-env/bin/activate
uv pip install vllm --torch-backend=auto
vllm --version

The quickstart lists Python 3.11 to 3.14 and recommends uv; --torch-backend=auto picks a PyTorch build that matches your driver. On an RTX 5090, check the “no kernel image” fix if PyTorch reports no kernel for sm_120.

Step 3: Download the model (optional but useful)

vllm serve downloads from Hugging Face on first start, but a separate download is easier to watch and resume:

pip install -U "huggingface_hub"
cloudgpu-net on                  # download acceleration for this shell
hf download Qwen/Qwen2.5-7B-Instruct --local-dir ~/models/qwen2.5-7b-instruct
cloudgpu-net off

For gated models such as Llama, accept the licence on Hugging Face and export HF_TOKEN first. See how to speed up model downloads.

Step 4: Start the server

Run it inside tmux so it keeps running when SSH drops:

tmux new -s vllm
source ~/vllm-env/bin/activate
export VLLM_API_KEY="$(openssl rand -hex 16)"; echo "$VLLM_API_KEY"   # note the key
vllm serve ~/models/qwen2.5-7b-instruct \
  --host 127.0.0.1 --port 8000 \
  --served-model-name qwen2.5-7b \
  --max-model-len 16384

What the flags do, per the vLLM engine arguments page (read 2026-11-03):

Flag Meaning
--api-key or VLLM_API_KEY requests must send this key as a Bearer token (quickstart)
--served-model-name “The model name(s) used in the API”; without it, the name is the model path
--max-model-len “Model context length (prompt and output)”; lower it if the KV cache does not fit
--gpu-memory-utilization “The fraction of GPU memory to be used for the model executor”
--generation-config defaults to auto, which loads sampling defaults from the model’s generation_config.json; vllm uses vLLM’s own defaults
--dtype auto uses FP16 for FP32/FP16 models and BF16 for BF16 models

Binding to 127.0.0.1 keeps the server off the public internet; you reach it through SSH in the next step. Wait until the log shows the server is listening on port 8000.

Step 5: Call it from your computer

Open a tunnel with the user, host and port from your instance row:

ssh -N -L 8000:127.0.0.1:8000 -p <ssh-port> <user>@<host>

Then, on your computer:

export VLLM_API_KEY=<the key from Step 4>
curl http://localhost:8000/v1/models -H "Authorization: Bearer $VLLM_API_KEY"
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" -H "Authorization: Bearer $VLLM_API_KEY" \
  -d '{"model": "qwen2.5-7b", "messages": [{"role": "user", "content": "Say hello in one sentence."}]}'

With the OpenAI Python SDK (pip install openai), streaming included:

import os
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key=os.environ["VLLM_API_KEY"])
stream = client.chat.completions.create(
    model="qwen2.5-7b",
    messages=[{"role": "user", "content": "Explain the KV cache in two sentences."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Tools that take an OpenAI base URL (Open WebUI, LangChain, Cursor-style clients) work the same way with http://localhost:8000/v1. Details on tunnels: SSH port forwarding on a cloud GPU.

Troubleshooting

  • CUDA out of memory at start: lower --max-model-len, choose a quantised model, or use a card with more VRAM. See fixing CUDA out of memory.
  • Chat requests fail: the vLLM serving docs say “vLLM requires the model to include a chat template in its tokenizer configuration”; for a model without one, pass --chat-template <file.jinja> (online serving docs, read 2026-11-03).
  • Answers differ from what you expect: vLLM applies the model’s generation_config.json by default; start with --generation-config vllm to use vLLM’s defaults.
  • 401 errors: the client must send the same key as Authorization: Bearer <key>.
  • curl cannot connect: check that the tunnel is open and the server log shows it is listening on 8000.

Stop paying when you stop serving

The GPU is billed per minute, rounded up, with a 1-minute minimum, including idle time while the server waits for requests. Billing stops when you destroy the instance, and data on the instance is deleted then, so copy anything you want to keep first. If you start instances through the REST API, idle_minutes destroys an instance after its GPU has been idle that long (API docs). If you would rather not run a GPU at all, CloudGPU’s own OpenAI-compatible model API is at https://cloudgpu.app/v1.

Frequently asked questions

Which port does vllm serve use?

The vLLM quickstart says the server starts at http://localhost:8000 by default. Pass --host 127.0.0.1 --port 8000 to make it explicit and reach it through an SSH tunnel.

How do I add an API key to vLLM?

Pass --api-key or set the VLLM_API_KEY environment variable, per the vLLM quickstart. Clients then send the key as Authorization: Bearer .

How do I change the model name clients use?

Use --served-model-name. Per the vLLM engine arguments, without it the model name is the same as the model argument, for example the local path.

Which endpoints does the vLLM server provide?

The vLLM serving docs list the Completions, Chat Completions, Embeddings, Transcriptions and Translation APIs, plus the Responses API; /v1/models lists the served model (read 2026-11-03).

How is a vLLM server billed on CloudGPU?

Per minute of GPU time, rounded up, with a 1-minute minimum, including idle time. Billing stops when you destroy the instance. Rates are on https://cloudgpu.app/pricing.

Sources (read 2026-11-03, Asia/Shanghai)

The Deploy page lists the cards you can start. Prices are on https://cloudgpu.app/pricing and are not repeated here.

Last updated: 2026-11-03 (Asia/Shanghai).