How to run GPUStack on a cloud GPU

Rent a GPU on Deploy, then open a terminal in JupyterLab or over SSH. Install GPUStack with the Docker command the 2.3 quickstart prints, add a worker on that same machine, and deploy the catalog model that page names. Call /v1/chat/completions on that server, and copy the reply before you destroy the instance, because destroying the instance deletes its disk and stops the server.

An agent can rent the GPU and call this OpenAI-compatible gateway through an SSH tunnel. The agent pages are GPU compute for AI agents and the agent integration guide. Those pages are how an agent discovers a GPU and runs work on it. This page is the GPUStack install. A voice framework whose LLM page sets base_url on OpenAILLMService is How to run Pipecat voice agents with a self-hosted model on a cloud GPU. An agent session whose OpenAI-compatible page sets base_url and baseURL on openai.LLM is How to run LiveKit Agents with a self-hosted model on a cloud GPU. A general vLLM install is How to serve an OpenAI-compatible API with vLLM on a cloud GPU. A general SGLang install is How to run SGLang on a cloud GPU. Ollama’s install and port 11434 are How to run Ollama on a cloud GPU. A CUDA container whose quickstart curls http://localhost:8080/v1/chat/completions is How to run LocalAI on a cloud GPU. A gateway whose README prints uv tool install 'litellm[proxy]' is How to run LiteLLM Proxy in front of a self-hosted model on a cloud GPU. A library whose vLLM chat page sets ChatOpenAI base_url to http://localhost:8000/v1 is How to build LangChain agents with a self-hosted model on a cloud GPU. A browser agent whose models page sets base_url to http://localhost:11434/v1 is How to run browser-use with a self-hosted model on a cloud GPU. CloudGPU’s hosted gateway is a different base URL: How do I use an OpenAI-compatible API for open models?. How published weight sizes are read: How much VRAM do I need to run an LLM?. The tunnel is How to use SSH port forwarding on a cloud GPU. If the process runs out of memory: How to fix CUDA out of memory on a cloud GPU. If the server raises no kernel image on an RTX 5090: How to fix CUDA no kernel image on an RTX 5090.

The upstream LICENSE is the Apache License, Version 2.0.

The steps below are copied from the docs pages named in each step, fetched 2026-11-14. They were not run on a CloudGPU instance for this page. The quickstart points at other GPU types. This page does not copy that list. The Inference APIs page documents other API families. This page does not copy those sections. The pages this guide read do not print hf download or huggingface-cli. Weight download for a model server you start yourself is a different guide: How to download a Hugging Face model.

What the docs print about memory

The 2.3 quickstart and the 2.3 Inference APIs page print no GPU GB figure. The quickstart tells you to select Qwen3.5-0.8B in the catalog. It does not print a Hugging Face repository path. A Hub lookup of that name on 2026-11-14 returned Qwen/Qwen3.5-0.8B, with gated false and card license apache-2.0. That README printed no GPU GB figure in the text this page searched. This page does not mark a catalog row as enough, and it does not claim the GPUStack catalog entry is that repository. How published weight sizes are read: How much VRAM do I need to run an LLM?.

GET /v1/public/compute-capabilities does not list GPUs. It returns feature flags. The memory column is vram_gb from GET /v1/public/gpu-catalog (generated_at 2026-11-14T20:06:07.058Z). The catalog response also includes an availability field and a price field. This page does not copy them. Every catalog row below has min_gpu_count 1. Rows are sorted by vram_gb, then by name. Which listing you can start is on Deploy.

The catalog template list had no GPUStack template. It did have Ollama templates (jupyterlab-ollama, ollama-qwen3, ollama). Those templates are a different install. This page does not treat them as GPUStack.

Catalog name vram_gb
RTX 2080 8
RTX 3080 10
RTX 2080 Ti 11
RTX 3080 Ti 12
V100 16G 16
A30 24
RTX 3090 24
RTX 4090 24
RTX 4090D 24
RTX 4080 SUPER 32G 32
RTX 5090 32
RTX 5090D 32
V100 32G 32
A100 40G 40
A800 40G 40
A40 48
RTX 4090 48G 48
RTX 4090D 48G 48
RTX A6000 48
Ascend 910B 64
NVIDIA RTX PRO 5000 72G 72
RTX PRO 5000 72G 72
A100 80G 80
A800 80G 80
H800 80
RTX 6000D 83
H20 96G 96
RTX PRO 6000 96G 96
RTX PRO 6000 Max-Q 96
H20-3e 141G 141

Install the server

Repository HEAD was aba5c665aa6ed7f2b38a7d08c666d6a594e90b95, committed 2026-11-13T01:38:32Z. pyproject.toml at that commit sets version to v0.0.0 and requires-python to 3.11 or newer and below 3.13. PyPI on 2026-11-14 reported gpustack 2.3.1. The Docker command below does not print an image tag. This page does not add one.

The quickstart says the worker node needs an NVIDIA GPU, the NVIDIA driver, Docker, and the NVIDIA Container Toolkit. It says a dedicated CPU node for the server is optional. If no dedicated CPU node is available, the server can be installed on the same machine as a GPU worker. The server command, copied:

sudo docker run -d --name gpustack \
    --restart unless-stopped \
    -p 80:80 \
    --volume gpustack-data:/var/lib/gpustack \
    gpustack/gpustack

The same page’s registry-mirror alternative, copied:

sudo docker run -d --name gpustack \
    --restart unless-stopped \
    -p 80:80 \
    --volume gpustack-data:/var/lib/gpustack \
    quay.io/gpustack/gpustack \
    --system-default-container-registry quay.io

The log command, copied:

sudo docker logs -f gpustack

The password command, copied:

sudo docker exec gpustack cat /var/lib/gpustack/initial_admin_password

This page does not print a password value. The quickstart says the username is admin and the UI is http://your_host_ip.

/latest/quickstart/ on 2026-11-14 printed the same server command, the same password command, the same worker flags, the same model name, and the same chat curl as the 2.3 quickstart. This page follows those commands.

Worker on the same machine

The quickstart says the UI’s worker command looks similar to the following. Copied:

sudo docker run -d --name gpustack-worker \
      --restart=unless-stopped \
      --privileged \
      --network=host \
      --volume /var/run/docker.sock:/var/run/docker.sock \
      --volume gpustack-data:/var/lib/gpustack \
      --runtime nvidia \
      gpustack/gpustack \
      --server-url http://your_gpustack_server_url \
      --token your_worker_token \
      --advertise-address 192.168.1.2

--runtime nvidia is the runtime flag that block prints. http://your_gpustack_server_url, your_worker_token, and 192.168.1.2 are the placeholders that block prints. The quickstart says the UI generates the command you run. This page does not replace those placeholders, and it did not run the command. The image name again has no tag.

Deploy Qwen3.5-0.8B and call chat completions

The quickstart’s deploy steps say to open the Catalog page, select Qwen3.5-0.8B, and click Save after the compatibility checks pass. It says the deployment status Running means the model has been deployed. It says to open Playground - Chat and check that qwen3.5-0.8b is selected.

The API key steps say to open Access Control, then API Keys, click New API Key, and save the key the UI shows once. The chat request that follows, copied:

# Replace `your_api_key` and `your_gpustack_server_url`
# with your actual API key and GPUStack server URL.
export GPUSTACK_API_KEY=your_api_key
curl http://your_gpustack_server_url/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GPUSTACK_API_KEY" \
  -d '{
    "model": "qwen3.5-0.8b",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Tell me a joke."
      }
    ],
    "stream": true
  }'

The Inference APIs page says GPUStack provides OpenAI-compatible APIs at /v1. Its chat example uses the same path and sets model to qwen3, with the user message Hello!. The endpoint names that page lists are List Models, Create Completion, Create Chat Completion, Create Response, Create Embeddings, Create Image, Create Image Edit, Create Speech, and Create Transcription. This page copies the quickstart request because that is the one whose model name matches the catalog selection above.

The container publishes port 80. None of these pages print an ssh command. How to use SSH port forwarding on a cloud GPU is how a browser on your computer reaches a port on the instance. This page does not add an ssh command. After a forward that lands on port 80 of your computer, the UI host the quickstart names is the forwarded host, and the chat path it already prints is /v1/chat/completions.

When it fails

CUDA out of memory. The quickstart and the Inference APIs page print no GPU GB figure. If the model server raises CUDA out of memory, use how to fix CUDA out of memory on a cloud GPU. This page does not pick a different model id for you.

“CUDA error: no kernel image is available for execution on the device” on an RTX 5090. The pages this guide read do not pin a PyTorch build and do not name compute capability 12.0. The worker command passes --runtime nvidia. The RTX 5090 guide says that card is compute capability 12.0 (sm_120). This page did not re-test a model server on an RTX 5090. Catalog RTX 5090 and RTX 5090D are 32. That is the catalog number only.

The quickstart says the worker node needs the NVIDIA driver, Docker, and the NVIDIA Container Toolkit. This page does not invent an error string for a missing toolkit.

What to copy off, then destroy

The cleanup section tells you to delete the model on the Deployments page. The chat reply is in the playground and in the curl response. Copy that text before you destroy the instance. The Docker volume gpustack-data is on the instance. The model server stops with the instance. Save your work before you destroy a cloud GPU is the checklist. The terms say data on local instance storage is permanently deleted when the instance is destroyed.

The billing page states the billing rule in these words:

Per-minute options have a 1-minute minimum; unused prepaid minutes are refunded when you destroy the instance. Some configurations use whole-hour billing with a 1-hour minimum; unused time within a charged hour is not refunded on voluntary stop. The billing period, minimum charge and initial prepayment are shown before confirmation. No subscription.

Amounts are not copied here.

Frequently asked questions

What memory figure does the GPUStack quickstart print?

The 2.3 quickstart and the 2.3 Inference APIs page print no GPU GB figure. The Qwen/Qwen3.5-0.8B README fetched on 2026-11-14 printed no GPU GB figure. This page does not mark a catalog row as enough. Which listing you can start is on https://cloudgpu.app/deploy.

Which command starts the server?

The quickstart runs sudo docker run -d --name gpustack --restart unless-stopped -p 80:80 --volume gpustack-data:/var/lib/gpustack gpustack/gpustack. The image name has no tag. A worker on the same machine is allowed when no dedicated CPU node is available, and the sample worker command passes --runtime nvidia. This page did not run the commands.

How does the quickstart read the initial admin password?

The quickstart prints sudo docker exec gpustack cat /var/lib/gpustack/initial_admin_password and says the username is admin. This page does not print a password value. The UI URL in that page is http://your_host_ip. The container publishes port 80. Those pages do not print an SSH command.

What license sentence does the file print?

The upstream LICENSE is the Apache License, Version 2.0. pyproject.toml sets version to v0.0.0 and requires-python to 3.11 or newer and below 3.13. PyPI on 2026-11-14 reported gpustack 2.3.1. Repository HEAD was aba5c665aa6ed7f2b38a7d08c666d6a594e90b95, committed 2026-11-13T01:38:32Z. The Docker command does not print an image tag. That license text is not a CloudGPU rate.

Sources (read 2026-11-14, Asia/Shanghai)

  • gpustack commit aba5c665aa6ed7f2b38a7d08c666d6a594e90b95, committed 2026-11-13T01:38:32Z. The upstream LICENSE is the Apache License, Version 2.0. pyproject.toml version v0.0.0. requires-python 3.11 or newer and below 3.13. PyPI gpustack 2.3.1 the same day
  • Quickstart https://docs.gpustack.ai/2.3/quickstart/ , fetched 2026-11-14. Server image gpustack/gpustack with no tag. Port 80:80. Volume gpustack-data. Password command sudo docker exec gpustack cat /var/lib/gpustack/initial_admin_password. Username admin. Worker flag --runtime nvidia. Catalog model Qwen3.5-0.8B. Chat model qwen3.5-0.8b. /latest/quickstart/ printed the same commands that day. Other GPU types named on the page are not copied
  • Inference APIs https://docs.gpustack.ai/2.3/integrations/inference-apis/ , fetched 2026-11-14. OpenAI-compatible APIs at /v1. The chat example sets model to qwen3. Other API families on that page are not copied
  • Qwen/Qwen3.5-0.8B Hub model, fetched 2026-11-14. gated false. Card license apache-2.0. The README search found no GPU GB figure. The quickstart does not print this repository path
  • The pages above do not print hf download or an ssh command
  • Public GPU memory: https://cloudgpu.app/v1/public/gpu-catalog (generated_at 2026-11-14T20:06:07.058Z). GET /v1/public/compute-capabilities returned feature flags and no GPU list. The catalog had no GPUStack template. It listed Ollama template ids jupyterlab-ollama, ollama-qwen3, and ollama

Cards you can start are on the Deploy page. Amounts are not copied here.

Last updated: 2026-11-14 (Asia/Shanghai).