How to run AutoGen agents with a self-hosted model on a cloud GPU

Rent a GPU on Deploy, then open a terminal in JupyterLab or over SSH. Install AgentChat the way the official README prints, serve a model on that machine, and pass OllamaChatCompletionClient or OpenAIChatCompletionClient into AssistantAgent. Copy the printed reply before you destroy the instance, because destroying the instance deletes its disk.

An agent can rent the GPU and run this library on it. The agent pages are GPU compute for AI agents and the agent integration guide. Those pages are how an agent discovers a GPU and runs work on it. This page is the AutoGen install. A Python agent library whose CodeAgent can take TransformersModel or an OpenAI-compatible server is How to run smolagents with a self-hosted model on a cloud GPU. A local agent UI whose Docker command publishes port 3000 is How to run OpenHands with a self-hosted model on a cloud GPU. A typed agent whose Ollama page sets base_url to http://localhost:11434/v1 is How to run Pydantic AI agents with a self-hosted model on a cloud GPU. A framework whose README sets Ollama and HuggingFaceEmbedding is How to build LlamaIndex agents with a self-hosted model on a cloud GPU. A general vLLM install is How to serve an OpenAI-compatible API with vLLM on a cloud GPU. A general SGLang install is How to run SGLang on a cloud GPU. Ollama’s install and port 11434 are How to run Ollama on a cloud GPU. A TurboMind server whose lmdeploy serve api_server listens on port 23333 is How to serve LLMs with LMDeploy on a cloud GPU. A coding-agent server whose card prints --tool-call-parser qwen3_coder is How to serve Qwen3-Coder for coding agents on a cloud GPU. CloudGPU’s hosted gateway is a different base URL: How do I use an OpenAI-compatible API for open models?. How published weight sizes are read: How much VRAM do I need to run an LLM?. The tunnel is How to use SSH port forwarding on a cloud GPU. If the process runs out of memory: How to fix CUDA out of memory on a cloud GPU. If the process raises no kernel image on an RTX 5090: How to fix CUDA no kernel image on an RTX 5090. A crew whose LLM class takes base_url for Ollama or an OpenAI-compatible server is How to run CrewAI with a self-hosted model on a cloud GPU. A graph runtime whose overview example does not name a model is How to run LangGraph agents with a self-hosted model on a cloud GPU. A browser agent whose models page sets base_url to http://localhost:11434/v1 is How to run browser-use with a self-hosted model on a cloud GPU. A pipeline whose Ollama page sets url to http://localhost:11434, and whose vLLM page defaults api_base_url to http://localhost:8000/v1, is How to run Haystack agents with a self-hosted model on a cloud GPU. A program whose language-models page sets dspy.LM to ollama_chat/llama3.2:1b, and whose GPU-server example sets api_base to http://localhost:7501/v1, is How to run DSPy with a self-hosted model on a cloud GPU. An agent whose Ollama example sets id to llama3.1:8b, and whose VLLM class defaults base_url to http://localhost:8000/v1/, is How to run Agno agents with a self-hosted model on a cloud GPU.

The root LICENSE file begins with “Attribution 4.0 International”, LICENSE-CODE begins with “MIT License” and its copyright line names the repository owner, and the GitHub license field on 2026-11-14 was CC-BY-4.0.

The commands below are copied from the upstream files named in each step. They were not run on a CloudGPU instance for this page. Anything this page adds around them is marked. The README and the models notebook also name other providers. This page does not list those names. The README on main has an archived badge and a caution that says the project is archived and is no longer maintained. That caution links a successor repository. This page does not copy that name. Tag python-v0.7.5 does not contain that caution. The install line below is on both READMEs.

What the docs print about memory

The README, the models notebook, the Ollama client docstring, and the local-LLM cookbook print no GPU GB figure. The models notebook says small local models are typically not as capable as larger models on the cloud. That note prints no GPU GB figure. This page does not mark a catalog row as enough.

GET /v1/public/compute-capabilities does not list GPUs. It returns feature flags. The memory column is vram_gb from gpu-prices.json, updated_at 2026-11-14T06:12:25.272Z. GET /v1/public/gpu-catalog the same day (generated_at 2026-11-14T06:12:26.381Z) listed the same names and the same vram_gb values, plus Ascend 910B at 64, which is not in gpu-prices.json and is not listed here. The catalog response also includes an availability field. This page does not copy it. Every prices row has gpu_count 1. Every catalog row has min_gpu_count 1. Which listing you can start is on Deploy.

The catalog template list had no AutoGen template. It did have Ollama templates (jupyterlab-ollama, ollama-qwen3, ollama). Those templates are a different install. This page does not treat them as AutoGen.

Catalog name vram_gb
RTX 2080 8
RTX 3080 10
RTX 2080 Ti 11
RTX 3080 Ti 12
V100 16G 16
A30 24
RTX 3090 24
RTX 4090 24
RTX 4090D 24
RTX 4080 SUPER 32G 32
RTX 5090 32
RTX 5090D 32
V100 32G 32
A100 40G 40
A800 40G 40
A40 48
RTX 4090 48G 48
RTX 4090D 48G 48
RTX A6000 48
NVIDIA RTX PRO 5000 72G 72
RTX PRO 5000 72G 72
A100 80G 80
A800 80G 80
H800 80
RTX 6000D 83
H20 96G 96
RTX PRO 6000 96G 96
RTX PRO 6000 Max-Q 96
H20-3e 141G 141

Install

The README says AutoGen requires Python 3.10 or later. The AgentChat install, copied:

pip install -U "autogen-agentchat" "autogen-ext[openai]"

python/packages/autogen-agentchat/pyproject.toml at both commits below says version = "0.7.5" and requires-python = ">=3.10". PyPI info.version for autogen-agentchat on 2026-11-14 was 0.7.5, requires_python >=3.10, uploaded 2025-09-30T06:16:12.895879Z. autogen-ext the same day was also 0.7.5, uploaded 2025-09-30T06:16:22.832378Z. Tag python-v0.7.5 is commit 83afbf5857aac683340d4c692194e548b1e8edda, committed 2025-09-30T06:07:16Z. Main is commit 8390985108cb1e3881f5f7297765758d8ad2540c, committed 2026-11-12T10:19:33Z. Those are not the same commit. The pyproject.toml blob, the model-client files, and the models notebook are the same blob on both commits. The README blob is not. This page did not install either checkout.

The README also prints pip install -U "autogenstudio" and autogenstudio ui --port 8080 --appdir ./my-app, and says that UI is on http://localhost:8080. It says that UI is not meant to be a production-ready app. This page does not treat that UI as the agent example.

AssistantAgent

The README hello world imports AssistantAgent and OpenAIChatCompletionClient, then runs a task. The constructor on that snippet sets model to a closed-model id. The heading above it names a different closed-model id. This page does not copy either id. The agent lines, copied:

agent = AssistantAgent("assistant", model_client=model_client)
print(await agent.run(task="Say 'Hello World!'"))
await model_client.close()

The snippet is inside async def main and asyncio.run(main()). This page did not run it.

Ollama and an OpenAI-compatible server

The models notebook says you can use OpenAIChatCompletionClient with models hosted on OpenAI-compatible endpoints, and that this functionality has not been tested. The Ollama section of that notebook, copied:

pip install -U "autogen-ext[ollama]"

from autogen_core.models import UserMessage
from autogen_ext.models.ollama import OllamaChatCompletionClient

# Assuming your Ollama server is running locally on port 11434.
ollama_model_client = OllamaChatCompletionClient(model="llama3.2")

The same class docstring’s own example sets model to llama3, not llama3.2. Those tags are not the same. The docstring says host is an optional model host url. The notebook example does not pass host. The docstring says Ollama must be installed and the appropriate model pulled. It does not print an ollama pull line for llama3 or llama3.2. This page does not add one.

The client source docstring says you can use OpenAIChatCompletionClient for OpenAI-compatible ChatCompletion endpoints, and that this use is not tested or guaranteed. It says base_url is required when the model is not one the class hosts itself, and model_info is required when the model name is not one the class already lists. The example for that case, copied:

from autogen_ext.models.openai import OpenAIChatCompletionClient
from autogen_core.models import ModelFamily

custom_model_client = OpenAIChatCompletionClient(
    model="deepseek-r1:1.5b",
    base_url="http://localhost:11434/v1",
    api_key="placeholder",
    model_info={
        "vision": False,
        "function_calling": False,
        "json_output": False,
        "family": ModelFamily.R1,
        "structured_output": True,
    },
)

That example’s model string is an Ollama tag. The files this page read do not print a Hugging Face repository id for it, and they do not print hf download. This page does not add one. They also do not print a vLLM serve command or a vLLM port. How to serve an OpenAI-compatible API with vLLM on a cloud GPU is the server install. How to run Ollama on a cloud GPU is the Ollama install.

The local-LLM cookbook prints a different path. The commands in that notebook, copied:

curl -fsSL https://ollama.com/install.sh | sh

ollama pull llama3.2:1b

pip install 'litellm[proxy]'
litellm --model ollama/llama3.2:1b

The notebook says the proxy will be available at http://0.0.0.0:4000/. The litellm command does not pass a port. The client in that notebook, copied:

return OpenAIChatCompletionClient(
    model="llama3.2:1b",
    api_key="NotRequiredSinceWeAreLocal",
    base_url="http://0.0.0.0:4000",
    model_capabilities={
        "json_output": False,
        "vision": False,
        "function_calling": True,
    },
)

llama3.2:1b, llama3.2, llama3, and deepseek-r1:1.5b are not the same tag. _openai_client.py at this commit raises if both model_capabilities and model_info are set, and warns that model_capabilities is deprecated, use model_info instead. The cookbook still passes model_capabilities. This page did not run the proxy.

For a server you start on the instance, the SSH tunnel is How to use SSH port forwarding on a cloud GPU. The models notebook names port 11434. The OpenAI-compatible example prints http://localhost:11434/v1. The cookbook prints http://0.0.0.0:4000. The README’s studio command prints port 8080.

When it fails

CUDA out of memory. The AutoGen files this page read print no GPU GB figure. If the process raises CUDA out of memory, use how to fix CUDA out of memory on a cloud GPU. This page does not pick a smaller model id for you.

“CUDA error: no kernel image is available for execution on the device” on an RTX 5090. The README and pyproject.toml do not name compute capability 12.0. The RTX 5090 guide says that card is compute capability 12.0 (sm_120) and that PyTorch 2.7 and later, on a build for CUDA 12.8 or newer, is the line that announced support. This page did not install a torch wheel and did not re-test this library on an RTX 5090. Catalog RTX 5090 and RTX 5090D are 32. That is the catalog number only.

What to copy off, then destroy

agent.run returns the result in the process that called it. The hello world prints that result. Copy the printed reply before you destroy the instance. Save your work before you destroy a cloud GPU is the checklist. The terms say data on local instance storage is permanently deleted when the instance is destroyed.

The pricing page FAQ states the billing rule in these words:

Per-minute options have a 1-minute minimum; unused prepaid minutes are refunded when you destroy the instance. Some configurations use whole-hour billing with a 1-hour minimum; unused time within a charged hour is not refunded on voluntary stop. The billing period, minimum charge and initial prepayment are shown before confirmation. No subscription.

Prices are not copied here.

Frequently asked questions

What memory figure do the AutoGen docs print?

The README, the models notebook, the Ollama client docstring, and the local-LLM cookbook print no GPU GB figure. The models notebook says small local models are typically not as capable as larger models on the cloud. That note prints no GPU GB figure. This page does not mark a catalog row as enough. Which listing you can start is on https://cloudgpu.app/deploy.

Which install lines and classes do the docs print?

The README requires Python 3.10 or later and prints pip install -U autogen-agentchat and autogen-ext[openai]. pyproject.toml says version 0.7.5 and requires-python >=3.10. PyPI info.version for autogen-agentchat on 2026-11-14 was 0.7.5. Tag python-v0.7.5 and main are not the same commit. The pyproject.toml blob is the same on both. The README blob is not. The hello world constructs AssistantAgent with name assistant and model_client, and the task string says Hello World. The model id on that constructor is not copied on this page. The models notebook constructs OllamaChatCompletionClient with model llama3.2. The class docstring example uses llama3. Those tags are not the same. This page did not install the package.

How do the docs point the client at a local or OpenAI-compatible server?

The models notebook assumes the Ollama server is on port 11434 and does not pass host. The OpenAIChatCompletionClient docstring example sets model to deepseek-r1:1.5b, base_url to http://localhost:11434/v1, api_key to placeholder, and a model_info dict. It says use of that client for a non-hosted model is not tested or guaranteed. The local-LLM cookbook pulls llama3.2:1b, runs a proxy, and sets base_url to http://0.0.0.0:4000 with model_capabilities. The client source says model_capabilities is deprecated and to use model_info. Those model tags are not the same. The files do not print a vLLM port. The README studio command prints port 8080.

What license do the files print?

The root LICENSE file begins with Attribution 4.0 International. LICENSE-CODE begins with MIT License and its copyright line names the repository owner. The GitHub license field on 2026-11-14 was CC-BY-4.0. That license text is not a CloudGPU price.

Sources (read 2026-11-14, Asia/Shanghai)

  • AutoGen README (Python 3.10 or later, pip install -U "autogen-agentchat" "autogen-ext[openai]", AssistantAgent, studio command on port 8080, archived badge on this commit only): commit 8390985108cb1e3881f5f7297765758d8ad2540c, committed 2026-11-12T10:19:33Z, https://github.com/microsoft/autogen/blob/8390985108cb1e3881f5f7297765758d8ad2540c/README.md. GitHub license field CC-BY-4.0. pushed_at 2026-11-12T10:19:36Z. Root LICENSE begins with Attribution 4.0 International. LICENSE-CODE begins with MIT License. The copyright line names the repository owner. This page does not print that name. The README also names other providers and a successor repository. This page does not list those names
  • Tag python-v0.7.5, commit 83afbf5857aac683340d4c692194e548b1e8edda, committed 2025-09-30T06:07:16Z. pyproject.toml version 0.7.5. The README on this tag does not contain the archived caution. The install line matches. PyPI autogen-agentchat info.version 0.7.5, uploaded 2025-09-30T06:16:12.895879Z, requires_python >=3.10. PyPI autogen-ext info.version 0.7.5, uploaded 2025-09-30T06:16:22.832378Z
  • python/docs/src/user-guide/agentchat-user-guide/tutorial/models.ipynb (pip install -U "autogen-ext[ollama]", OllamaChatCompletionClient(model="llama3.2"), port 11434, OpenAI-compatible note that the path is not tested): same blob on both commits
  • python/packages/autogen-ext/src/autogen_ext/models/openai/_openai_client.py (base_url, model_info, example deepseek-r1:1.5b at http://localhost:11434/v1, model_capabilities deprecation warning): same blob on both commits. The hosted example model id is not copied
  • python/packages/autogen-ext/src/autogen_ext/models/ollama/_ollama_client.py (model="llama3", optional host, pull is required and no pull command is printed): same blob on both commits
  • python/docs/src/user-guide/core-user-guide/cookbook/local-llms-ollama-litellm.ipynb (ollama pull llama3.2:1b, litellm --model ollama/llama3.2:1b, http://0.0.0.0:4000, model_capabilities): same blob on both commits
  • Public GPU memory: https://cloudgpu.app/gpu-prices.json (updated_at 2026-11-14T06:12:25.272Z) and https://cloudgpu.app/v1/public/gpu-catalog (generated_at 2026-11-14T06:12:26.381Z). GET /v1/public/compute-capabilities returned feature flags and no GPU list. The catalog had no AutoGen template

Cards you can start are on the Deploy page. Prices are on https://cloudgpu.app/pricing and are not repeated here.

Last updated: 2026-11-14 (Asia/Shanghai).