How to run Langroid agents with a self-hosted model on a cloud GPU

Rent a GPU on Deploy, then open a terminal in JupyterLab or over SSH. Install Langroid with the pip install langroid line the README prints, serve an ungated open-weight model on that machine, and set OpenAIGPTConfig chat_model and api_base to that server. Copy the task output before you destroy the instance, because destroying the instance deletes its disk.

An agent can rent the GPU and run this library against a model served on it. The agent pages are GPU compute for AI agents and the agent integration guide. Those pages are how an agent discovers a GPU and runs work on it. This page is the Langroid install. A general vLLM install is How to serve an OpenAI-compatible API with vLLM on a cloud GPU. A general SGLang install is How to run SGLang on a cloud GPU. Ollama’s install and port 11434 are How to run Ollama on a cloud GPU. An OpenAI-compatible engine whose CUDA 12 container tag is localai/localai:latest-gpu-nvidia-cuda-12 is How to run LocalAI on a cloud GPU. A gateway whose README prints uv tool install 'litellm[proxy]' is How to run LiteLLM Proxy in front of a self-hosted model on a cloud GPU. CloudGPU’s hosted gateway is a different base URL: How do I use an OpenAI-compatible API for open models?. How published weight sizes are read: How much VRAM do I need to run an LLM?. A library whose README prints uv add langchain is How to build LangChain agents with a self-hosted model on a cloud GPU. An AgentChat library whose OpenAI-compatible example sets base_url to http://localhost:11434/v1 is How to run AutoGen agents with a self-hosted model on a cloud GPU. A typed agent whose vLLM example sets base_url to http://localhost:8000/v1 is How to run Pydantic AI agents with a self-hosted model on a cloud GPU. A Python agent library whose OpenAIModel takes api_base is How to run smolagents with a self-hosted model on a cloud GPU. A framework whose 2.0.11 quickstart prints uv pip install "agentscope==2.0.11" is How to run AgentScope agents with a self-hosted model on a cloud GPU. A framework whose README comment sets model_server to http://localhost:8000/v1 is How to run Qwen-Agent with a self-hosted model on a cloud GPU. The tunnel is How to use SSH port forwarding on a cloud GPU. If the process runs out of memory: How to fix CUDA out of memory on a cloud GPU. If the process raises no kernel image on an RTX 5090: How to fix CUDA no kernel image on an RTX 5090.

The commands below are copied from the local-LLM page and from the README at commit be56d05c129ec169d9f2160489438300de8ae30d, except the download lines marked as added. They were not run on a CloudGPU instance for this page. The local-LLM page also documents hosted chat_model prefixes. This page does not copy those sections.

What the docs print about memory

The local-LLM page prints no GPU GB figure. The Qwen/Qwen2.5-Coder-32B README fetched on 2026-11-14 does not print a GPU GB number. It points at a benchmark page. This page does not copy a number from that other page. This page does not mark a catalog row as enough.

GET /v1/public/compute-capabilities does not list GPUs. It returns feature flags. The memory column is vram_gb from GET /v1/public/gpu-catalog (generated_at 2026-11-14T21:12:05.060Z). The catalog response also includes an availability field. This page does not copy it. Every catalog row below has min_gpu_count 1. Which listing you can start is on Deploy.

The catalog template list had no Langroid template. It did have Ollama templates (jupyterlab-ollama, ollama-qwen3, ollama). Those templates are a different install. This page does not treat them as Langroid.

Catalog name	vram_gb
RTX 2080	8
RTX 3080	10
RTX 2080 Ti	11
RTX 3080 Ti	12
V100 16G	16
A30	24
RTX 3090	24
RTX 4090	24
RTX 4090D	24
RTX 4080 SUPER 32G	32
RTX 5090	32
RTX 5090D	32
V100 32G	32
A100 40G	40
A800 40G	40
A40	48
RTX 4090 48G	48
RTX 4090D 48G	48
RTX A6000	48
Ascend 910B	64
NVIDIA RTX PRO 5000 72G	72
RTX PRO 5000 72G	72
A100 80G	80
A800 80G	80
H800	80
RTX 6000D	83
H20 96G	96
RTX PRO 6000 96G	96
RTX PRO 6000 Max-Q	96
H20-3e 141G	141

Install

The README installation section, on commit be56d05c129ec169d9f2160489438300de8ae30d (committed 2026-11-14T13:57:21Z), says Langroid requires Python 3.11 or higher. The install line, copied:

pip install langroid

pyproject.toml on that commit says version = "0.69.23", license = {text = "MIT"}, and requires-python = "<3.14,>=3.10". The README sentence and the requires-python line are not the same bound. PyPI langroid info.version on 2026-11-14 was 0.69.23, requires_python <3.14,>=3.10, license MIT, uploaded 2026-11-14T13:57:34.483662Z. The GitHub license field on 2026-11-14 was MIT. pushed_at was 2026-11-14T13:57:25Z. This page did not install the package.

Point OpenAIGPTConfig at the GPU server

The local-LLM page’s “Other local LLM scenarios” section says to set both chat_model and api_base when the shorthand does not fit. Its example, copied:

import langroid.language_models as lm
llm_config = lm.OpenAIGPTConfig(
    chat_model="Mistral-7B-Instruct-v0.2",
    api_base="http://192.168.0.5:5078/v1",
)

That api_base is the address in the page’s curl example. It is not a command this page adds. On the same commit, OpenAIGPTConfig in langroid/language_models/openai_gpt.py sets api_base: str | None = None and the comment says it is used for local or other non-OpenAI models. The class docstring says the class covers locally served models behind an OpenAI-compatible API. chat_model is described as any model name served at an OpenAI-compatible API endpoint.

The same page’s vLLM section prints this serve line:

vllm serve Qwen/Qwen2.5-Coder-32B

It says to set chat_model to vllm/Qwen/Qwen2.5-Coder-32B, and, if a port other than the default 8000 was used, set api_base to localhost:{port}. That api_base string is not http://192.168.0.5:5078/v1. The page says to set HF_TOKEN when the model is not publicly available. On 2026-11-14, GET https://huggingface.co/api/models/Qwen/Qwen2.5-Coder-32B reported gated false, cardData.license apache-2.0, pipeline_tag text-generation, lastModified 2024-11-18T12:54:58.000Z, and sha 2e12b5f7bc878d424d222e224ed40aee564ec45f. The card README does not print a GPU GB number. It points at a benchmark page. This page does not treat 32B in that id as vram_gb.

The constructor on that commit treats a chat_model that starts with ollama/ or vllm/ as local. It also lists other prefixes. This page does not copy those other prefixes.

The Ollama section says that as of version 0.1.24, Ollama provides an OpenAI-compatible API. The pull line and the config, copied:

ollama pull mistral:7b-instruct-v0.2-q8_0

import langroid.language_models as lm
import langroid as lr

llm_config = lm.OpenAIGPTConfig(
    chat_model="ollama/mistral:7b-instruct-v0.2-q8_0",
    chat_context_length=16_000, # adjust based on model
)

mistral:7b-instruct-v0.2-q8_0 is not Mistral-7B-Instruct-v0.2, and neither string is Qwen/Qwen2.5-Coder-32B. chat_context_length is a token setting in that example. It is not a GPU GB figure. How to run Ollama on a cloud GPU is the Ollama install. That server stays on the rented GPU. How to serve an OpenAI-compatible API with vLLM on a cloud GPU is the vLLM install.

The Langroid files this page read do not print hf download. How to download a Hugging Face model prints pip install -U "huggingface_hub" and hf download. The block below is added on this page. This page did not download the repository.

# Added on this page. The vLLM section passes this id to vllm serve.
pip install -U "huggingface_hub"
hf download Qwen/Qwen2.5-Coder-32B

For a server you start on the instance, the SSH tunnel is How to use SSH port forwarding on a cloud GPU. The local-LLM page does not print an SSH command.

ChatAgent

The Ollama section builds a ChatAgent from that llm_config. Copied:

agent_config = lr.ChatAgentConfig(
    llm=llm_config,
    system_message="You are helpful but concise",
)
agent = lr.ChatAgent(agent_config)
task = lr.Task(agent, interactive=True)
task.run()

langroid/__init__.py on that commit exports ChatAgent and ChatAgentConfig. The explicit api_base block earlier on the page is a different OpenAIGPTConfig. This page does not merge the two snippets. The vLLM chat_model string is a third setting. This page did not run task.run.

When it fails

CUDA out of memory. The local-LLM page prints no GPU GB figure. The Qwen/Qwen2.5-Coder-32B card README does not print a GPU GB number. If the process raises CUDA out of memory, use how to fix CUDA out of memory on a cloud GPU. This page does not pick a smaller model id for you.

“CUDA error: no kernel image is available for execution on the device” on an RTX 5090. The README and pyproject.toml do not name compute capability 12.0. The RTX 5090 guide says that card is compute capability 12.0 (sm_120) and that PyTorch 2.7 and later, on a build for CUDA 12.8 or newer, is the line that announced support. This page did not install a torch wheel and did not re-test this library on an RTX 5090. Catalog RTX 5090 and RTX 5090D are 32. That is the catalog number only.

What to copy off, then destroy

task.run is the interactive loop in the Ollama section. Copy the printed output before you destroy the instance. Save your work before you destroy a cloud GPU is the checklist. The terms say data on local instance storage is permanently deleted when the instance is destroyed.

The pricing page FAQ states the billing rule in these words:

Per-minute options have a 1-minute minimum; unused prepaid minutes are refunded when you destroy the instance. Some configurations use whole-hour billing with a 1-hour minimum; unused time within a charged hour is not refunded on voluntary stop. The billing period, minimum charge and initial prepayment are shown before confirmation. No subscription.

Prices are not copied here.

Frequently asked questions

What memory figure do the Langroid docs print?

The local-LLM page prints no GPU GB figure. The Qwen/Qwen2.5-Coder-32B README fetched on 2026-11-14 does not print a GPU GB number. It points at a benchmark page. This page does not copy a number from that other page. This page does not mark a catalog row as enough. Which listing you can start is on https://cloudgpu.app/deploy.

Which install line does the README print?

The README prints pip install langroid and says Langroid requires Python 3.11 or higher. pyproject.toml on commit be56d05c129ec169d9f2160489438300de8ae30d says version 0.69.23 and requires-python 3.10 or newer and below 3.14. PyPI info.version on 2026-11-14 was 0.69.23, requires_python that same bound, license MIT, uploaded 2026-11-14T13:57:34.483662Z. The README sentence and the requires-python line are not the same bound. This page did not install the package.

How does OpenAIGPTConfig point at a server on the GPU?

The local-LLM page sets chat_model to Mistral-7B-Instruct-v0.2 and api_base to http://192.168.0.5:5078/v1 in the explicit example. The vLLM section prints vllm serve Qwen/Qwen2.5-Coder-32B, sets chat_model to vllm/Qwen/Qwen2.5-Coder-32B, and sets api_base to localhost:{port} when the port is not the default 8000. Those api_base strings are not the same. The Ollama section sets chat_model to ollama/mistral:7b-instruct-v0.2-q8_0 and builds a ChatAgent. The source comment says api_base is used for local or other non-OpenAI models. Those pages do not print an SSH command. This page does not copy the hosted-provider prefix sections.

What license does the file print?

Sources (read 2026-11-14)

Langroid local-LLM page, read 2026-11-14: https://langroid.github.io/langroid/tutorials/local-llm-setup/ . Ollama pull and OpenAIGPTConfig with chat_model="ollama/mistral:7b-instruct-v0.2-q8_0", ChatAgent, Task. vLLM line vllm serve Qwen/Qwen2.5-Coder-32B, chat_model vllm/Qwen/Qwen2.5-Coder-32B, api_base localhost:{port} when the port is not 8000. Explicit chat_model="Mistral-7B-Instruct-v0.2" and api_base="http://192.168.0.5:5078/v1". The hosted-provider prefix sections are not copied. The page says “As of version 0.1.24” for the Ollama API. This page did not find 0.69.23 on that page
langroid/langroid commit be56d05c129ec169d9f2160489438300de8ae30d, committed 2026-11-14T13:57:21Z. README pip install langroid and “Python 3.11+”. pyproject.toml version 0.69.23, requires-python <3.14,>=3.10, license MIT. OpenAIGPTConfig.api_base comment and the ollama/ and vllm/ prefixes. __init__.py exports ChatAgent and ChatAgentConfig. GitHub license field MIT. pushed_at 2026-11-14T13:57:25Z. PyPI langroid 0.69.23, uploaded 2026-11-14T13:57:34.483662Z, requires_python <3.14,>=3.10, license MIT
Hugging Face model API Qwen/Qwen2.5-Coder-32B: gated false, cardData.license apache-2.0, pipeline_tag text-generation, lastModified 2024-11-18T12:54:58.000Z, sha 2e12b5f7bc878d424d222e224ed40aee564ec45f. The card README does not print a GPU GB number. hf download is added on this page and was not run
Public GPU memory: https://cloudgpu.app/v1/public/gpu-catalog (generated_at 2026-11-14T21:12:05.060Z). GET /v1/public/compute-capabilities returned feature flags and no GPU list. The catalog had no Langroid template

Cards you can start are on the Deploy page. Prices are on https://cloudgpu.app/pricing and are not repeated here.

Last updated: 2026-11-14.