How to run LiveKit Agents with a self-hosted model on a cloud GPU

Rent a GPU on Deploy, then open a terminal in JupyterLab or over SSH. Install the LiveKit Agents OpenAI plugin from the official docs, serve an open-weight model on that machine, and pass that server’s URL as base_url on openai.LLM. Start the local development server the self-host page prints, and copy any reply text before you destroy the instance, because destroying the instance deletes its disk and stops the server.

An agent can rent the GPU, serve the model, and run this agent framework against it. The agent pages are GPU compute for AI agents and the agent integration guide. Those pages are how an agent discovers a GPU and runs work on it. This page is the LiveKit Agents install. A voice framework whose LLM page sets base_url on OpenAILLMService is How to run Pipecat voice agents with a self-hosted model on a cloud GPU. A Docker server whose 2.3 quickstart publishes port 80 and whose chat example posts to /v1/chat/completions is How to run GPUStack on a cloud GPU. A general vLLM install is How to serve an OpenAI-compatible API with vLLM on a cloud GPU. A general SGLang install is How to run SGLang on a cloud GPU. Ollama’s install and port 11434 are How to run Ollama on a cloud GPU. A CUDA container whose quickstart curls http://localhost:8080/v1/chat/completions is How to run LocalAI on a cloud GPU. A gateway whose README prints uv tool install 'litellm[proxy]' is How to run LiteLLM Proxy in front of a self-hosted model on a cloud GPU. A library whose vLLM chat page sets ChatOpenAI base_url to http://localhost:8000/v1 is How to build LangChain agents with a self-hosted model on a cloud GPU. A browser agent whose models page sets base_url to http://localhost:11434/v1 is How to run browser-use with a self-hosted model on a cloud GPU. CloudGPU’s hosted gateway is a different base URL: How do I use an OpenAI-compatible API for open models?. How published weight sizes are read: How much VRAM do I need to run an LLM?. The tunnel is How to use SSH port forwarding on a cloud GPU. If the process runs out of memory: How to fix CUDA out of memory on a cloud GPU. If the server raises no kernel image on an RTX 5090: How to fix CUDA no kernel image on an RTX 5090.

The upstream LICENSE is the Apache License, Version 2.0.

The steps below are copied from the docs pages named in each step, fetched 2026-11-14. They were not run on a CloudGPU instance for this page. The OpenAI-compatible page prints a table of convenience methods. This page does not copy that table. The models overview shows other sessions. This page does not copy those sessions. The pages this guide read do not print hf download or huggingface-cli. Weight download for the model server is a different guide: How to download a Hugging Face model.

What the docs print about memory

The OpenAI-compatible LLMs page prints no GPU GB figure. The self-hosted deployments page recommends 4 cores and 8GB per agent server. Its load test prints ~2.8GB memory used and ~3.8 cores on one machine. Those figures are agent-server memory and CPU, not GPU VRAM. This page does not mark a catalog row as enough. The model server’s memory is a different page: How much VRAM do I need to run an LLM?.

GET /v1/public/compute-capabilities does not list GPUs. It returns feature flags. The memory column is vram_gb from GET /v1/public/gpu-catalog (generated_at 2026-11-14T20:06:07.058Z). The catalog response also includes an availability field and a price field. This page does not copy them. Every catalog row below has min_gpu_count 1. Rows are sorted by vram_gb, then by name. Which listing you can start is on Deploy.

The catalog template list had no LiveKit template. It did have Ollama templates (jupyterlab-ollama, ollama-qwen3, ollama). Those templates are a different install. This page does not treat them as LiveKit Agents.

Catalog name vram_gb
RTX 2080 8
RTX 3080 10
RTX 2080 Ti 11
RTX 3080 Ti 12
V100 16G 16
A30 24
RTX 3090 24
RTX 4090 24
RTX 4090D 24
RTX 4080 SUPER 32G 32
RTX 5090 32
RTX 5090D 32
V100 32G 32
A100 40G 40
A800 40G 40
A40 48
RTX 4090 48G 48
RTX 4090D 48G 48
RTX A6000 48
Ascend 910B 64
NVIDIA RTX PRO 5000 72G 72
RTX PRO 5000 72G 72
A100 80G 80
A800 80G 80
H800 80
RTX 6000D 83
H20 96G 96
RTX PRO 6000 96G 96
RTX PRO 6000 Max-Q 96
H20-3e 141G 141

Install

Repository HEAD was cbb5770c716c70009969cd7fa88e3c3662222ce9, committed 2026-11-14T01:28:23Z. livekit-agents/pyproject.toml at that commit sets license to Apache-2.0 and requires-python to 3.10 or newer and below 3.15. livekit/agents/version.py sets __version__ to 1.8.12. PyPI on 2026-11-14 reported livekit-agents 1.8.12.

The OpenAI-compatible LLMs page’s Python install, copied:

uv add "livekit-agents[openai]~=1.8"

The Node.js install on that page, copied:

pnpm add @livekit/agents-plugin-openai@1.x

Point openai.LLM at the server

The OpenAI-compatible LLMs page says to pass the endpoint and API key to the openai.LLM constructor. The Python example, copied:

from livekit.plugins import openai

session = AgentSession(
    llm=openai.LLM(
        model="your-model-name",
        base_url="https://your-provider.com/v1",
        api_key="your-api-key",
    ),
    # ... tts, stt, vad, turn_detection, etc.
)

The Node.js example, copied:

import * as openai from '@livekit/agents-plugin-openai';

const session = new voice.AgentSession({
    llm: new openai.LLM({
        model: 'your-model-name',
        baseURL: 'https://your-provider.com/v1',
        apiKey: 'your-api-key',
    }),
    // ... tts, stt, vad, turn_detection, etc.
});

your-model-name, https://your-provider.com/v1, and your-api-key are the placeholders that page prints. This page does not replace them with unstated strings, and it did not run the constructor. A vLLM server’s listen address is a different document: How to serve an OpenAI-compatible API with vLLM on a cloud GPU. That guide’s server starts at http://localhost:8000. Clients on that guide use http://localhost:8000/v1. Passing that URL as base_url or baseURL is this page’s use of the documented parameter. It was not run here.

The same page prints a table of convenience methods. This page does not copy that table and does not link those products. The comment in the session leaves tts, stt, and vad unnamed. The agents docs index lists a local Kokoro TTS page and a Silero VAD page, and it does not list a local speech-to-text page, so this page does not assemble a speech stack.

Local development server

The running-locally page’s Linux install and dev command, copied:

curl -sSL https://get.livekit.io | bash

livekit-server --dev

That page prints this API key pair for that dev process:

API key: devkey
API secret: secret

It says the signal server binds to 127.0.0.1:7880 by default. It also prints --bind 0.0.0.0 for access from other devices on the network. This page does not add that flag. The page also prints a macOS install line. The rented machine in this guide is Linux, so the block above is the Linux one.

The self-hosted deployments page says agent servers register with LiveKit server over a WebSocket and do not need a public inbound port. It says the default health check server listens on http://0.0.0.0:8081/. Its environment example names LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET. The values in that example are placeholders this page does not copy. The same example names other keys this page does not copy. That page says to use a local server for development. This page does not merge the dev key pair with those variable names into one file, and it did not run that combination.

None of these pages print an ssh command. How to use SSH port forwarding on a cloud GPU is how a program on your computer reaches a port on the instance. The signal server page says the bind address is 127.0.0.1:7880. This page does not add an ssh command.

When it fails

CUDA out of memory. The OpenAI-compatible LLMs page prints no GPU GB figure. If the model server raises CUDA out of memory, use how to fix CUDA out of memory on a cloud GPU. This page does not pick a model id for you.

“CUDA error: no kernel image is available for execution on the device” on an RTX 5090. The pages this guide read do not pin a PyTorch build and do not name compute capability 12.0. The RTX 5090 guide says that card is compute capability 12.0 (sm_120). This page did not re-test a model server on an RTX 5090. Catalog RTX 5090 and RTX 5090D are 32. That is the catalog number only.

What to copy off, then destroy

The session examples do not name an output file. Copy the reply text before you destroy the instance. The model server and the dev server stop with the instance. Save your work before you destroy a cloud GPU is the checklist. The terms say data on local instance storage is permanently deleted when the instance is destroyed.

The billing page states the billing rule in these words:

Per-minute options have a 1-minute minimum; unused prepaid minutes are refunded when you destroy the instance. Some configurations use whole-hour billing with a 1-hour minimum; unused time within a charged hour is not refunded on voluntary stop. The billing period, minimum charge and initial prepayment are shown before confirmation. No subscription.

Amounts are not copied here.

Frequently asked questions

What memory figure do the LiveKit pages this guide read print?

The OpenAI-compatible LLMs page prints no GPU GB figure. The self-hosted deployments page recommends 4 cores and 8GB per agent server and its load test prints ~2.8GB memory used. Those figures are agent-server memory, not GPU VRAM. This page does not mark a catalog row as enough. Which listing you can start is on https://cloudgpu.app/deploy.

Which constructor sets a custom base URL?

The OpenAI-compatible LLMs page constructs openai.LLM with model your-model-name, base_url https://your-provider.com/v1, and api_key your-api-key. The Node.js example uses baseURL and apiKey. The same page prints a table of convenience methods this page does not copy. This page did not run the constructor.

What does the local server page print?

The Linux install is curl -sSL https://get.livekit.io | bash. The dev command is livekit-server --dev. That page prints API key devkey and API secret secret, and says the signal server binds to 127.0.0.1:7880. The deployments page names LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET and says to use a local server for development. This page does not merge those files into one command.

What license sentence does the file print?

The upstream LICENSE is the Apache License, Version 2.0. livekit-agents/pyproject.toml sets license to Apache-2.0 and requires-python to 3.10 or newer and below 3.15. livekit/agents/version.py sets version to 1.8.12. The OpenAI-compatible page installs livekit-agents[openai]~=1.8. PyPI on 2026-11-14 reported livekit-agents 1.8.12. Repository HEAD was cbb5770c716c70009969cd7fa88e3c3662222ce9. That license text is not a CloudGPU rate.

Sources (read 2026-11-14, Asia/Shanghai)

  • livekit/agents commit cbb5770c716c70009969cd7fa88e3c3662222ce9, committed 2026-11-14T01:28:23Z. The upstream LICENSE is the Apache License, Version 2.0. livekit-agents/pyproject.toml license Apache-2.0. requires-python 3.10 or newer and below 3.15. __version__ 1.8.12. PyPI livekit-agents 1.8.12 the same day
  • OpenAI-compatible LLMs https://docs.livekit.io/agents/models/llm/openai-compatible-llms/ , fetched 2026-11-14. uv add "livekit-agents[openai]~=1.8". pnpm add @livekit/agents-plugin-openai@1.x. openai.LLM model="your-model-name" base_url="https://your-provider.com/v1" api_key="your-api-key". Node.js uses baseURL and apiKey. The convenience-method table is not copied
  • Running locally https://docs.livekit.io/transport/self-hosting/local , fetched 2026-11-14. Linux install curl -sSL https://get.livekit.io | bash. livekit-server --dev. API key devkey. API secret secret. Bind 127.0.0.1:7880
  • Self-hosted deployments https://docs.livekit.io/deploy/custom/deployments , fetched 2026-11-14. Agent servers do not need a public inbound port. Health check http://0.0.0.0:8081/. Environment names LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET. 4 cores and 8GB per agent server. Load test ~2.8GB memory used. Other keys in that example are not copied
  • Agents docs index https://docs.livekit.io/agents/llms.txt , fetched 2026-11-14. It lists a local Kokoro TTS page and a Silero VAD page. It does not list a local speech-to-text page. This page does not assemble a speech stack
  • The pages above do not print hf download or an ssh command
  • Public GPU memory: https://cloudgpu.app/v1/public/gpu-catalog (generated_at 2026-11-14T20:06:07.058Z). GET /v1/public/compute-capabilities returned feature flags and no GPU list. The catalog had no LiveKit template. It listed Ollama template ids jupyterlab-ollama, ollama-qwen3, and ollama

Cards you can start are on the Deploy page. Amounts are not copied here.

Last updated: 2026-11-14 (Asia/Shanghai).