How to run LocalAI on a cloud GPU

Rent a GPU on Deploy, then open a terminal in JupyterLab or over SSH. Start the CUDA container image the current README prints for the CUDA major version on that machine, load the gallery model qwen3-4b with the command that README prints, and call http://localhost:8080/v1/chat/completions. An agent points its OpenAI-compatible base URL at that server through an SSH tunnel. Copy any model files the container stored before you destroy the instance, because destroying the instance deletes its disk.

An agent can rent the GPU and call this server. The agent pages are GPU compute for AI agents and the agent integration guide. Those pages are how an agent discovers a GPU and runs work on it. This page is the LocalAI install. A general vLLM install is How to serve an OpenAI-compatible API with vLLM on a cloud GPU. A general SGLang install is How to run SGLang on a cloud GPU. Ollama’s install and port 11434 are How to run Ollama on a cloud GPU. A gateway whose README prints uv tool install 'litellm[proxy]' is How to run LiteLLM Proxy in front of a self-hosted model on a cloud GPU. CloudGPU’s hosted gateway is a different base URL: How do I use an OpenAI-compatible API for open models?. A CUDA build of another engine is How to build llama.cpp with CUDA on a cloud GPU. How published weight sizes are read: How much VRAM do I need to run an LLM?. The tunnel is How to use SSH port forwarding on a cloud GPU. If the process runs out of memory: How to fix CUDA out of memory on a cloud GPU. If the process raises no kernel image on an RTX 5090: How to fix CUDA no kernel image on an RTX 5090. A voice framework whose LLM page sets base_url on OpenAILLMService is How to run Pipecat voice agents with a self-hosted model on a cloud GPU. An agent session whose OpenAI-compatible page sets base_url and baseURL on openai.LLM is How to run LiveKit Agents with a self-hosted model on a cloud GPU. A Docker server whose 2.3 quickstart publishes port 80 and whose chat example posts to /v1/chat/completions is How to run GPUStack on a cloud GPU. A framework whose 2.0.11 quickstart pins agentscope==2.0.11, and whose OpenAICredential field base_url is passed to the chat client, is How to run AgentScope agents with a self-hosted model on a cloud GPU. A framework whose README comment sets model_server to http://localhost:8000/v1 and api_key to EMPTY is How to run Qwen-Agent with a self-hosted model on a cloud GPU. A library whose local-LLM page sets OpenAIGPTConfig chat_model and api_base for a vLLM-style endpoint is How to run Langroid agents with a self-hosted model on a cloud GPU. An agent whose Ollama page sets llm to an ollama/ tag, and whose default base URL text is http://localhost:11434/v1, is How to run PraisonAI agents with a self-hosted model on a cloud GPU. An agent whose OllamaPromptDriver sets model to llama3.1, and whose OpenAiChatPromptDriver example sets base_url to http://127.0.0.1:1234/v1, is How to run Griptape agents with a self-hosted model on a cloud GPU. An agent whose local branch sets base_url to http://localhost:11434/v1 and passes that client into AgentConfig is How to run Atomic Agents with a self-hosted model on a cloud GPU.

The commands below are copied from the upstream files named in each step. They were not run on a CloudGPU instance for this page. The README also prints other gallery examples. This page does not copy those names.

What the docs print about memory

The current README, the containers page, and the quickstart print no GPU GB figure for qwen3-4b. The gallery entry qwen3-4b in gallery/index.yaml on master prints “Number of Parameters: 4.0B” and “Context Length: 32,768 natively and 131,072 tokens with YaRN.” Those are parameter and context figures. This page does not treat them as vram_gb. The GPU acceleration page prints a VRAM line in a log for a different model file. This page does not copy that number and does not treat it as the requirement for qwen3-4b. The same page’s out-of-memory list says to lower context_size and to use a quantized model such as Q4_K_M. The gallery file for qwen3-4b is Qwen3-4B.Q4_K_M.gguf. This page does not mark a catalog row as enough.

GET /v1/public/compute-capabilities does not list GPUs. It returns feature flags. The memory column is vram_gb from GET /v1/public/gpu-catalog (generated_at 2026-11-14T19:13:01.996Z). The catalog response also includes an availability field. This page does not copy it. Every catalog row below has min_gpu_count 1. Which listing you can start is on Deploy.

The catalog template list had no LocalAI template. It did have Ollama templates (jupyterlab-ollama, ollama-qwen3, ollama). Those templates are a different install. This page does not treat them as LocalAI.

Catalog name	vram_gb
RTX 2080	8
RTX 3080	10
RTX 2080 Ti	11
RTX 3080 Ti	12
V100 16G	16
A30	24
RTX 3090	24
RTX 4090	24
RTX 4090D	24
RTX 4080 SUPER 32G	32
RTX 5090	32
RTX 5090D	32
V100 32G	32
A100 40G	40
A800 40G	40
A40	48
RTX 4090 48G	48
RTX 4090D 48G	48
RTX A6000	48
Ascend 910B	64
NVIDIA RTX PRO 5000 72G	72
RTX PRO 5000 72G	72
A100 80G	80
A800 80G	80
H800	80
RTX 6000D	83
H20 96G	96
RTX PRO 6000 96G	96
RTX PRO 6000 Max-Q	96
H20-3e 141G	141

Install the CUDA image

Master on 2026-11-14 was commit 6febd4381a7f6623e97429578fdd3d960cf3a204, committed 2026-11-14T18:13:59Z. pushed_at was 2026-11-14T18:14:02Z. Tag v4.12.1 is commit ec9274027eadd552d56f2c6c1fe86cd9216a991a, committed 2026-10-23T10:03:52Z, published 2026-10-23T10:27:00Z. Those commits are not the same. The master README says the script, the app, and the latest container image give version 4.12.1. The start index, which refreshes to /docs/start/, says that section needs a build newer than 4.12.1. This page does not resolve that difference. It copies the container tags both READMEs print, and the model command the master README and the live quickstart print.

The NVIDIA CUDA lines on the master README, also printed on the v4.12.1 README and on the containers page. The issue URL https://localai.io/docs/installation/containers/ refreshes to that containers page. Copied:

# NVIDIA, CUDA 13
docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-13

# NVIDIA, CUDA 12
docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-12

Use the tag whose CUDA major version matches the CUDA version on the instance. This page does not pick a CUDA major version for a catalog row. The containers page also prints podman lines and other device tags. This page does not copy those lines. It also prints a versioned tag pattern. This page does not fill in a version that pattern does not print as a finished tag. The README says LocalAI is then at http://localhost:8080.

The containers page’s GPU check, copied:

docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi

That check uses a CUDA 12 base image. It is not the LocalAI image.

Load qwen3-4b and call the API

The quickstart prints two install paths. The install-script path, copied:

curl -sSL https://localai.io/install.sh | sh
local-ai run qwen3-4b

The container path is the docker run above. That command does not install a local-ai binary on the host. The README’s later local-ai run qwen3-4b and local-ai chat --model qwen3-4b lines belong to the host CLI, not to a shell outside the container. This page does not run those two lines on the host after docker run.

After the container is at http://localhost:8080, the quickstart’s web steps are the documented install. Open Models, then Explore, search for qwen3-4b, and click Install if you did not start with local-ai run qwen3-4b. Then open Chats and select qwen3-4b. The quickstart also says a Docker or CLI start can take a gallery name as an argument. The example it prints for that sentence is still local-ai run qwen3-4b. It does not print a docker run line that appends qwen3-4b. This page does not invent one.

The v4.12.1 README prints a different gallery example. This page does not copy it. The name qwen3-4b is an entry in gallery/index.yaml on master and on tag v4.12.1. On master the entry says license apache-2.0, lists https://huggingface.co/Qwen/Qwen3-4B and https://huggingface.co/MaziyarPanahi/Qwen3-4B-GGUF, and downloads huggingface://MaziyarPanahi/Qwen3-4B-GGUF/Qwen3-4B.Q4_K_M.gguf with sha256 a37931937683a723ae737a0c6fc67dab7782fd8a1b9dea2ca445b7a1dbd5ca3a.

On 2026-11-14, GET https://huggingface.co/api/models/Qwen/Qwen3-4B reported gated false, private false, cardData.license apache-2.0, pipeline_tag text-generation, lastModified 2025-07-26T03:46:39.000Z, and sha 1cfa9a7208912126459214e8b04321603b3df60c. GET https://huggingface.co/api/models/MaziyarPanahi/Qwen3-4B-GGUF reported gated false, private false, pipeline_tag text-generation, cardData.base_model Qwen/Qwen3-4B, cardData.quantized_by MaziyarPanahi, lastModified 2025-04-28T21:28:16.000Z, and sha b507d8f2622a6be905066595a5b3cd00e11ab379. cardData.license was empty. The README search of Qwen/Qwen3-4B found no GPU GB figure.

The quickstart says qwen3-4b is a small Qwen3 model and that it supports tool calling. The API section says LocalAI exposes an OpenAI-compatible API and to point a client at http://localhost:8080. The sentence also names an SDK. This page does not copy that SDK name. The request it prints, copied:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-4b",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

It says the reply text is in choices[0].message.content. The same page also names other API formats. This page does not copy those names. It says the server listens on port 8080 on all interfaces and that authentication is off unless you set LOCALAI_API_KEY or LOCALAI_AUTH=true. The files this page read do not print an SSH command. For port 8080, the tunnel is How to use SSH port forwarding on a cloud GPU. After the tunnel, the base URL an OpenAI-compatible client uses is the http://localhost:8080 the quickstart prints, and the path it curls is /v1/chat/completions.

When it fails

CUDA out of memory. The GPU acceleration page’s out-of-memory list says to reduce gpu_layers, lower context_size, and use a quantized model such as Q4_K_M. The gallery file named above is Q4_K_M. If the process raises CUDA out of memory, use how to fix CUDA out of memory on a cloud GPU. This page does not pick a smaller model id for you.

“CUDA error: no kernel image is available for execution on the device” on an RTX 5090. The README and the containers page do not name compute capability 12.0. The RTX 5090 guide says that card is compute capability 12.0 (sm_120) and that PyTorch 2.7 and later, on a build for CUDA 12.8 or newer, is the line that announced support. This page did not install a torch wheel and did not re-test this image on an RTX 5090. Catalog RTX 5090 and RTX 5090D are 32. That is the catalog number only.

What to copy off, then destroy

The quickstart Install step downloads the gallery file into the running container. The docker run lines above do not print a volume. The containers page’s Compose example mounts ./models:/models:cached. That example’s command selects a gallery model this page does not copy. Copy the mounted directory, if you used that mount, before you destroy the instance. Save your work before you destroy a cloud GPU is the checklist. The terms say data on local instance storage is permanently deleted when the instance is destroyed.

The pricing page FAQ states the billing rule in these words:

Per-minute options have a 1-minute minimum; unused prepaid minutes are refunded when you destroy the instance. Some configurations use whole-hour billing with a 1-hour minimum; unused time within a charged hour is not refunded on voluntary stop. The billing period, minimum charge and initial prepayment are shown before confirmation. No subscription.

Prices are not copied here.

Frequently asked questions

What memory figure do the LocalAI docs print for qwen3-4b?

The current README, the containers page, and the quickstart print no GPU GB figure for qwen3-4b. The gallery entry prints 4.0B parameters and a context length of 32,768 tokens, or 131,072 with YaRN. This page does not treat those figures as vram_gb. The GPU acceleration page out-of-memory list names context_size and Q4_K_M. The gallery file is Qwen3-4B.Q4_K_M.gguf. This page does not mark a catalog row as enough. Which listing you can start is on https://cloudgpu.app/deploy.

Which container tag and model command do the docs print?

The master README and the v4.12.1 README both print docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-12 and the same command with latest-gpu-nvidia-cuda-13. The master README says the latest container image gives version 4.12.1. Tag v4.12.1 is commit ec9274027eadd552d56f2c6c1fe86cd9216a991a. Master on 2026-11-14 was commit 6febd4381a7f6623e97429578fdd3d960cf3a204. The quickstart install-script block prints local-ai run qwen3-4b after the install script. That binary is not installed by the container command. After docker run, the quickstart says to open Models, then Explore, search for qwen3-4b, and click Install. The v4.12.1 README prints a different gallery example. This page does not copy it. The start index says that section needs a build newer than 4.12.1. This page did not run the container.

How does a client call the server?

The quickstart curls http://localhost:8080/v1/chat/completions with model qwen3-4b. It says to point a client at http://localhost:8080. The server listens on port 8080. The files do not print an SSH command. The gallery entry license field is apache-2.0. The model API on 2026-11-14 reported Qwen/Qwen3-4B gated false and license apache-2.0. The GGUF repo MaziyarPanahi/Qwen3-4B-GGUF was gated false, and its card license field was empty.

What license does the file print?

The LICENSE file is the MIT License, and its copyright line names Ettore Di Giacinto for 2023-2025. The GitHub license field on 2026-11-14 was MIT.

Sources (read 2026-11-14, Asia/Shanghai)

mudler/LocalAI master commit 6febd4381a7f6623e97429578fdd3d960cf3a204, committed 2026-11-14T18:13:59Z, pushed_at 2026-11-14T18:14:02Z. README: localai/localai:latest-gpu-nvidia-cuda-12, localai/localai:latest-gpu-nvidia-cuda-13, local-ai run qwen3-4b, local-ai chat --model qwen3-4b, http://localhost:8080, latest image gives version 4.12.1. Other gallery examples are not copied. GitHub license field MIT. LICENSE copyright Copyright © 2023-2025 Ettore Di Giacinto (mudler@localai.io)
Tag v4.12.1, commit ec9274027eadd552d56f2c6c1fe86cd9216a991a, committed 2026-10-23T10:03:52Z, published 2026-10-23T10:27:00Z. The CUDA 12 and CUDA 13 docker lines match. The gallery example on that README is not copied. gallery/index.yaml on that tag contains name: qwen3-4b
Containers page https://localai.io/docs/start/install/containers/ and docs/content/start/install/containers.md on master. https://localai.io/docs/installation/containers/ refreshes to that page. The versioned tag pattern is not filled in. GPU check docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi
Quickstart https://localai.io/docs/start/quickstart/: install-script block curl -sSL https://localai.io/install.sh | sh then local-ai run qwen3-4b. The container block is a separate docker run. After that container, open Models, then Explore, and Install qwen3-4b. Curl http://localhost:8080/v1/chat/completions, model qwen3-4b, port 8080, LOCALAI_API_KEY, LOCALAI_AUTH=true. Other API format names are not copied. https://localai.io/docs/getting-started/ refreshes to /docs/start/, which says the section needs a build newer than 4.12.1
gallery/index.yaml on master, entry qwen3-4b: license apache-2.0, parameters 4.0B, context 32768 and 131072 with YaRN, file huggingface://MaziyarPanahi/Qwen3-4B-GGUF/Qwen3-4B.Q4_K_M.gguf, sha256 a37931937683a723ae737a0c6fc67dab7782fd8a1b9dea2ca445b7a1dbd5ca3a
GPU acceleration page, out-of-memory list: reduce gpu_layers, lower context_size, Q4_K_M. A VRAM log for a different model file is not copied
Hugging Face model API Qwen/Qwen3-4B: gated false, cardData.license apache-2.0, pipeline_tag text-generation, lastModified 2025-07-26T03:46:39.000Z, sha 1cfa9a7208912126459214e8b04321603b3df60c. No GPU GB figure in the README search
Hugging Face model API MaziyarPanahi/Qwen3-4B-GGUF: gated false, cardData.license empty, cardData.base_model Qwen/Qwen3-4B, sha b507d8f2622a6be905066595a5b3cd00e11ab379, lastModified 2025-04-28T21:28:16.000Z
Public GPU memory: https://cloudgpu.app/v1/public/gpu-catalog (generated_at 2026-11-14T19:13:01.996Z). GET /v1/public/compute-capabilities returned feature flags and no GPU list. The catalog had no LocalAI template

Cards you can start are on the Deploy page. Prices are on https://cloudgpu.app/pricing and are not repeated here.

Last updated: 2026-11-14 (Asia/Shanghai).