How to run Agno agents with a self-hosted model on a cloud GPU

Q: Which install lines and Agent examples do the docs print?

The Ollama cookbook prints pip install -U ollama agno and ollama pull llama3.1:8b. The chat basic example sets Ollama id to llama3.1:8b. The Ollama class default id is llama3.1. Those strings are not the same. The vLLM cookbook prints uv pip install vllm and a vllm serve command for Qwen/Qwen2.5-7B-Instruct. basic.py sets VLLM id to that same id. pyproject.toml prints version 3.2.0 and requires-python >=3.9,<4. PyPI info.version on 2026-11-14 was 3.2.0. Tag v3.2.0 and main are not the same commit. The files this page cites are the same blob on both. This page did not install the packages.

Q: How do the docs point the agent at a local or OpenAI-compatible server?

VLLM subclasses OpenAILike. The class sets base_url from VLLM_BASE_URL, and the default in that line is http://localhost:8000/v1/. The serve command does not pass --port. basic.py does not pass api_key. The class reads VLLM_API_KEY and raises ModelAuthenticationError when that variable is empty. The cookbook does not print a value. This page adds export VLLM_API_KEY before vllm serve and before basic.py, in the same shell. The vLLM guide on this site fills that variable with openssl rand -hex 16. The vLLM quickstart says that variable, or the api-key flag, makes the server require the key as a Bearer token. The Ollama basic example does not pass host. The class sets host to https://ollama.com only when an API key is set and host is empty. This page does not use that URL as the server on the rented GPU. The basic example does not print a port.

Q: What license does the file print?

The LICENSE file begins with Apache License Version 2.0, January 2004. Its appendix copyright line names 2025-2026 Agno Inc. The GitHub license field on 2026-11-14 was Apache-2.0. That license text is not a CloudGPU price.

Rent a GPU on Deploy, then open a terminal in JupyterLab or over SSH. Install Agno the way the Ollama cookbook in the official repository prints, set VLLM_API_KEY in that shell, serve a model on that machine, and point Ollama or VLLM at that server. Copy the text print_response writes before you destroy the instance, because destroying the instance deletes its disk.

An agent can rent the GPU and run this library on it. The agent pages are GPU compute for AI agents and the agent integration guide. Those pages are how an agent discovers a GPU and runs work on it. This page is the Agno install. A Python agent library whose CodeAgent can take TransformersModel or an OpenAI-compatible server is How to run smolagents with a self-hosted model on a cloud GPU. A crew whose LLM class takes base_url for Ollama or an OpenAI-compatible server is How to run CrewAI with a self-hosted model on a cloud GPU. A graph runtime whose overview example does not name a model is How to run LangGraph agents with a self-hosted model on a cloud GPU. An AgentChat library whose Ollama example sets model to llama3.2 is How to run AutoGen agents with a self-hosted model on a cloud GPU. A typed agent whose vLLM example sets base_url to http://localhost:8000/v1 is How to run Pydantic AI agents with a self-hosted model on a cloud GPU. A framework whose README sets Ollama and HuggingFaceEmbedding is How to build LlamaIndex agents with a self-hosted model on a cloud GPU. A browser agent whose models page sets base_url to http://localhost:11434/v1 is How to run browser-use with a self-hosted model on a cloud GPU. A pipeline whose OllamaChatGenerator sets url to http://localhost:11434 is How to run Haystack agents with a self-hosted model on a cloud GPU. A program whose language-models page sets dspy.LM to ollama_chat/llama3.2:1b is How to run DSPy with a self-hosted model on a cloud GPU. A stateful CLI whose connect help sets --base-url to http://localhost:8000/v1 for openai-compatible is How to run Letta agents with a self-hosted model on a cloud GPU. A TypeScript agent whose local-models section sets url on the model object is How to run Mastra agents with a self-hosted model on a cloud GPU. A ChatAgent whose vLLM example sets ModelPlatformType.VLLM is How to run CAMEL-AI agents with a self-hosted model on a cloud GPU. A general vLLM install is How to serve an OpenAI-compatible API with vLLM on a cloud GPU. A general SGLang install is How to run SGLang on a cloud GPU. Ollama’s install and port 11434 are How to run Ollama on a cloud GPU. An embedding server is How to serve embeddings and rerankers with Text Embeddings Inference on a cloud GPU. CloudGPU’s hosted gateway is a different base URL: How do I use an OpenAI-compatible API for open models?. How published weight sizes are read: How much VRAM do I need to run an LLM?. The tunnel is How to use SSH port forwarding on a cloud GPU. If the process runs out of memory: How to fix CUDA out of memory on a cloud GPU. If the process raises no kernel image on an RTX 5090: How to fix CUDA no kernel image on an RTX 5090. A Docker Compose app whose self-host page opens http://localhost/install is How to run Dify with a self-hosted model on a cloud GPU. A canvas whose ChatOllama page names a base URL is How to run Flowise with a self-hosted model on a cloud GPU. A flow editor whose Ollama component defaults the base URL to http://127.0.0.1:11434 is How to run Langflow with a self-hosted model on a cloud GPU.

The LICENSE file begins with “Apache License Version 2.0, January 2004”, its appendix copyright line names 2025-2026 Agno Inc., and the GitHub license field on 2026-11-14 was Apache-2.0.

The commands below are copied from the upstream files named in each step. They were not run on a CloudGPU instance for this page. Anything this page adds around them is marked. The models index README names other providers. This page does not list those names.

What the docs print about memory

The README, the Ollama cookbook, and the vLLM basic example print no GPU GB figure for llama3.1:8b or Qwen/Qwen2.5-7B-Instruct. The Qwen2.5-7B-Instruct card README says to see a benchmark page for GPU memory and throughput. This page does not copy that page and does not treat it as vram_gb. The vLLM cookbook embeddings section prints “~14GB VRAM” for intfloat/e5-mistral-7b-instruct, “~2GB VRAM” for BAAI/bge-large-en-v1.5, and “~500MB VRAM” for sentence-transformers/all-MiniLM-L6-v2. This page does not treat those figures as the chat model’s vram_gb. The serve command prints --gpu-memory-utilization 0.9. That flag is not a GPU GB figure. This page does not mark a catalog row as enough.

GET /v1/public/compute-capabilities does not list GPUs. It returns feature flags. The memory column is vram_gb from gpu-prices.json, updated_at 2026-11-14T07:15:49.882Z. GET /v1/public/gpu-catalog the same day (generated_at 2026-11-14T07:15:50.490Z) listed the same names and the same vram_gb values, plus Ascend 910B at 64, which is not in gpu-prices.json and is not listed here. The catalog response also includes an availability field. This page does not copy it. Every prices row has gpu_count 1. Every catalog row has min_gpu_count 1. Which listing you can start is on Deploy.

The catalog template list had no Agno template. It did have Ollama templates (jupyterlab-ollama, ollama-qwen3, ollama). Those templates are a different install. This page does not treat them as Agno.

Catalog name	vram_gb
RTX 2080	8
RTX 3080	10
RTX 2080 Ti	11
RTX 3080 Ti	12
V100 16G	16
A30	24
RTX 3090	24
RTX 4090	24
RTX 4090D	24
RTX 4080 SUPER 32G	32
RTX 5090	32
RTX 5090D	32
V100 32G	32
A100 40G	40
A800 40G	40
A40	48
RTX 4090 48G	48
RTX 4090D 48G	48
RTX A6000	48
NVIDIA RTX PRO 5000 72G	72
RTX PRO 5000 72G	72
A100 80G	80
A800 80G	80
H800	80
RTX 6000D	83
H20 96G	96
RTX PRO 6000 96G	96
RTX PRO 6000 Max-Q	96
H20-3e 141G	141

Install

The root README does not print pip install agno. The Ollama cookbook does. Copied:

pip install -U ollama agno

The same file prints the pull:

ollama pull llama3.1:8b

libs/agno/pyproject.toml says version = "3.2.0", requires-python = ">=3.9,<4", and license = { file = "LICENSE" }. PyPI info.version for agno on 2026-11-14 was 3.2.0, requires_python >=3.9,<4, uploaded 2026-11-12T17:48:01.469162Z. Tag v3.2.0 is commit 3bfc84113fdd80c1522ef7bafa80721ce79fdf62, committed 2026-11-12T17:44:28Z. Main is commit 439da90695d16d469b3b1963913ab1ad6e6a9648, committed 2026-11-14T01:21:55Z. Those are not the same commit. The README, LICENSE, pyproject.toml, the Ollama cookbook, cookbook/90_models/ollama/chat/basic.py, the vLLM cookbook, cookbook/90_models/vllm/basic.py, Ollama, VLLM, and OpenAILike are the same blob on both. This page did not diff the rest of the tree. This page did not install the package.

The vLLM cookbook install, copied:

uv pip install vllm

That section does not print an agno install line. basic.py imports agno.

Agent

The Ollama chat basic example, copied:

from agno.agent import Agent
from agno.models.ollama import Ollama

agent = Agent(model=Ollama(id="llama3.1:8b"), markdown=True)

The class default in libs/agno/agno/models/ollama/chat.py is id: str = "llama3.1". llama3.1 and llama3.1:8b are not the same string. The file’s __main__ block calls agent.print_response("Share a 2 sentence horror story"). This page did not run it.

The vLLM basic example, copied:

from agno.agent import Agent
from agno.models.vllm import VLLM

agent = Agent(
    model=VLLM(id="Qwen/Qwen2.5-7B-Instruct", top_k=20, enable_thinking=False),
    markdown=True,
)

The file’s __main__ block calls agent.print_response("Share a 2 sentence horror story"). This page did not run it. A quickstart file in cookbook/00_quickstart sets the model to a closed-vendor class. This page does not copy that class or that model id.

Ollama and an OpenAI-compatible server

VLLM subclasses OpenAILike. The OpenAILike docstring says it talks to a provider that uses the OpenAI API schema. VLLM._get_client_params sets base_url from the environment variable VLLM_BASE_URL when base_url is empty. The default in that line is "http://localhost:8000/v1/". If api_key is empty, the same method reads VLLM_API_KEY and raises ModelAuthenticationError with the message “VLLM_API_KEY not set. Please set the VLLM_API_KEY environment variable.” basic.py does not pass api_key or base_url. The cookbook does not print a value for VLLM_API_KEY, and the serve command below does not pass --api-key.

How to serve an OpenAI-compatible API with vLLM on a cloud GPU says the vLLM quickstart uses --api-key or VLLM_API_KEY, and that clients then send the key as a Bearer token. Set the variable in the same shell before vllm serve and before basic.py. The block below is added on this page. It uses the key command that vLLM guide prints. Agno does not print it. This page did not run it.

# Added on this page. basic.py does not set VLLM_API_KEY.
# VLLM._get_client_params raises when the variable is empty.
# Set this before vllm serve and before basic.py.
export VLLM_API_KEY="$(openssl rand -hex 16)"

The vLLM cookbook serve command, copied:

vllm serve Qwen/Qwen2.5-7B-Instruct \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --dtype float16 \
    --max-model-len 2048 \
    --gpu-memory-utilization 0.9

That command does not pass --port. The class default URL uses port 8000. The same README’s remote embedder example sets base_url="http://localhost:8000/v1", without the trailing slash the class default has. Those two strings are not the same. How to serve an OpenAI-compatible API with vLLM on a cloud GPU is the general server install.

The cookbook does not print hf download. How to download a Hugging Face model prints pip install -U "huggingface_hub" and hf download. The block below is added on this page. This page did not download the repository.

# Added on this page. The vLLM example passes this id to VLLM and to vllm serve.
pip install -U "huggingface_hub"
hf download Qwen/Qwen2.5-7B-Instruct

On 2026-11-14, GET https://huggingface.co/api/models/Qwen/Qwen2.5-7B-Instruct reported gated false, cardData.license apache-2.0, pipeline_tag text-generation, lastModified 2025-01-12T02:10:10.000Z, and sha a09a35458c702b33eeacc393d103063234e8bc28. The card README prints no GPU GB figure. It points at a benchmark page. This page does not copy that page. This page does not treat 7B in the id as vram_gb.

The Ollama basic example does not pass host. In Ollama._get_client_params, if api_key is set and host is empty, host becomes "https://ollama.com". The basic example does not set api_key. This page does not use that URL as the server on the rented GPU. The basic example does not print a port. The cookbook says to install Ollama and pull llama3.1:8b. How to run Ollama on a cloud GPU is that install. The files this page read do not print a Hugging Face repository id for llama3.1:8b. This page does not add hf download for that tag.

For a server you start on the instance, the SSH tunnel is How to use SSH port forwarding on a cloud GPU. The VLLM default URL prints port 8000.

When it fails

CUDA out of memory. The Agno chat examples this page read print no GPU GB figure for llama3.1:8b or Qwen/Qwen2.5-7B-Instruct. If the process raises CUDA out of memory, use how to fix CUDA out of memory on a cloud GPU. This page does not pick a smaller model id for you.

“CUDA error: no kernel image is available for execution on the device” on an RTX 5090. pyproject.toml does not name compute capability 12.0. The RTX 5090 guide says that card is compute capability 12.0 (sm_120) and that PyTorch 2.7 and later, on a build for CUDA 12.8 or newer, is the line that announced support. This page did not install a torch wheel and did not re-test this library on an RTX 5090. Catalog RTX 5090 and RTX 5090D are 32. That is the catalog number only.

What to copy off, then destroy

agent.print_response writes the reply in the process that called it. Copy that text before you destroy the instance. Save your work before you destroy a cloud GPU is the checklist. The terms say data on local instance storage is permanently deleted when the instance is destroyed.

The pricing page FAQ states the billing rule in these words:

Per-minute options have a 1-minute minimum; unused prepaid minutes are refunded when you destroy the instance. Some configurations use whole-hour billing with a 1-hour minimum; unused time within a charged hour is not refunded on voluntary stop. The billing period, minimum charge and initial prepayment are shown before confirmation. No subscription.

Prices are not copied here.

Frequently asked questions

What memory figure do the Agno docs print?

The README, the Ollama cookbook, and the vLLM basic example print no GPU GB figure for llama3.1:8b or Qwen/Qwen2.5-7B-Instruct. The Qwen2.5-7B-Instruct card prints no GPU GB figure. It points at a benchmark page. This page does not copy that page. The vLLM cookbook embeddings section prints ~14GB VRAM for intfloat/e5-mistral-7b-instruct, ~2GB VRAM for BAAI/bge-large-en-v1.5, and ~500MB VRAM for sentence-transformers/all-MiniLM-L6-v2. This page does not treat those figures as the chat model vram_gb. The serve command prints the flag gpu-memory-utilization 0.9. That flag is not a GPU GB figure. This page does not mark a catalog row as enough. Which listing you can start is on https://cloudgpu.app/deploy.

Which install lines and Agent examples do the docs print?

The Ollama cookbook prints pip install -U ollama agno and ollama pull llama3.1:8b. The chat basic example sets Ollama id to llama3.1:8b. The Ollama class default id is llama3.1. Those strings are not the same. The vLLM cookbook prints uv pip install vllm and a vllm serve command for Qwen/Qwen2.5-7B-Instruct. basic.py sets VLLM id to that same id. pyproject.toml prints version 3.2.0 and requires-python >=3.9,<4. PyPI info.version on 2026-11-14 was 3.2.0. Tag v3.2.0 and main are not the same commit. The files this page cites are the same blob on both. This page did not install the packages.

How do the docs point the agent at a local or OpenAI-compatible server?

VLLM subclasses OpenAILike. The class sets base_url from VLLM_BASE_URL, and the default in that line is http://localhost:8000/v1/. The serve command does not pass --port. basic.py does not pass api_key. The class reads VLLM_API_KEY and raises ModelAuthenticationError when that variable is empty. The cookbook does not print a value. This page adds export VLLM_API_KEY before vllm serve and before basic.py, in the same shell. The vLLM guide on this site fills that variable with openssl rand -hex 16. The vLLM quickstart says that variable, or the api-key flag, makes the server require the key as a Bearer token. The Ollama basic example does not pass host. The class sets host to https://ollama.com only when an API key is set and host is empty. This page does not use that URL as the server on the rented GPU. The basic example does not print a port.

What license does the file print?

The LICENSE file begins with Apache License Version 2.0, January 2004. Its appendix copyright line names 2025-2026 Agno Inc. The GitHub license field on 2026-11-14 was Apache-2.0. That license text is not a CloudGPU price.

Sources (read 2026-11-14, Asia/Shanghai)

agno-agi/agno README: commit 439da90695d16d469b3b1963913ab1ad6e6a9648, committed 2026-11-14T01:21:55Z. GitHub license field Apache-2.0. pushed_at 2026-11-14T03:23:58Z. The README does not print pip install agno. LICENSE begins with Apache License Version 2.0, January 2004. The appendix copyright line names 2025-2026 Agno Inc.
Tag v3.2.0, commit 3bfc84113fdd80c1522ef7bafa80721ce79fdf62, committed 2026-11-12T17:44:28Z. README, LICENSE, libs/agno/pyproject.toml, the Ollama and vLLM cookbooks, basic.py for both, Ollama, VLLM, and OpenAILike match main. This page did not diff the rest of the tree. PyPI agno info.version 3.2.0, uploaded 2026-11-12T17:48:01.469162Z, requires_python >=3.9,<4
cookbook/90_models/ollama/README.md (pip install -U ollama agno, ollama pull llama3.1:8b): same blob as the tag. The optional OLLAMA_API_KEY export on that page is for the cloud note in the class, not the rented-GPU server
cookbook/90_models/ollama/chat/basic.py (Agent, Ollama(id="llama3.1:8b"), print_response): same blob as the tag. libs/agno/agno/models/ollama/chat.py default id is llama3.1. host becomes https://ollama.com only when api_key is set and host is empty
cookbook/90_models/vllm/README.md (uv pip install vllm, vllm serve Qwen/Qwen2.5-7B-Instruct, --gpu-memory-utilization 0.9, embedding VRAM lines): same blob as the tag. Those VRAM lines are not the chat model’s vram_gb
cookbook/90_models/vllm/basic.py (VLLM(id="Qwen/Qwen2.5-7B-Instruct")): same blob as the tag. It does not pass api_key. libs/agno/agno/models/vllm/vllm.py subclasses OpenAILike, defaults base_url to http://localhost:8000/v1/, and raises when VLLM_API_KEY is empty. export VLLM_API_KEY="$(openssl rand -hex 16)" is added on this page. The command is the one on the vLLM guide. Agno does not print it. libs/agno/agno/models/openai/like.py says the class uses the OpenAI API schema
cookbook/90_models/README.md names other providers. This page does not list them. cookbook/00_quickstart/agent_with_tools.py sets a closed-vendor model class. This page does not copy it
Hugging Face model API Qwen/Qwen2.5-7B-Instruct: gated false, cardData.license apache-2.0, pipeline_tag text-generation, lastModified 2025-01-12T02:10:10.000Z, sha a09a35458c702b33eeacc393d103063234e8bc28. The card README prints no GPU GB figure and points at a benchmark page. This page does not copy that page. hf download is added on this page and was not run
Public GPU memory: https://cloudgpu.app/gpu-prices.json (updated_at 2026-11-14T07:15:49.882Z) and https://cloudgpu.app/v1/public/gpu-catalog (generated_at 2026-11-14T07:15:50.490Z). GET /v1/public/compute-capabilities returned feature flags and no GPU list. The catalog had no Agno template

Cards you can start are on the Deploy page. Prices are on https://cloudgpu.app/pricing and are not repeated here.

Last updated: 2026-11-14 (Asia/Shanghai).