How to run Haystack agents with a self-hosted model on a cloud GPU
Rent a GPU on Deploy, then open a terminal in JupyterLab or over SSH. Install Haystack the way the official README prints, serve a model on that machine, and point OllamaChatGenerator or VLLMChatGenerator at the URL those docs print. Copy last_message text and replies text before you destroy the instance, because destroying the instance deletes its disk.
An agent can rent the GPU and run this library on it. The agent pages are GPU compute for AI agents and the agent integration guide. Those pages are how an agent discovers a GPU and runs work on it. This page is the Haystack install. A Python agent library whose CodeAgent can take TransformersModel or an OpenAI-compatible server is How to run smolagents with a self-hosted model on a cloud GPU. A crew whose LLM class takes base_url for Ollama or an OpenAI-compatible server is How to run CrewAI with a self-hosted model on a cloud GPU. A graph runtime whose overview example does not name a model is How to run LangGraph agents with a self-hosted model on a cloud GPU. An AgentChat library whose Ollama example sets model to llama3.2 is How to run AutoGen agents with a self-hosted model on a cloud GPU. A typed agent whose vLLM example sets base_url to http://localhost:8000/v1 is How to run Pydantic AI agents with a self-hosted model on a cloud GPU. A framework whose README sets Ollama and HuggingFaceEmbedding is How to build LlamaIndex agents with a self-hosted model on a cloud GPU. A browser agent whose models page sets base_url to http://localhost:11434/v1 is How to run browser-use with a self-hosted model on a cloud GPU. A program whose language-models page sets dspy.LM to ollama_chat/llama3.2:1b is How to run DSPy with a self-hosted model on a cloud GPU. An agent whose Ollama example sets id to llama3.1:8b is How to run Agno agents with a self-hosted model on a cloud GPU. A general vLLM install is How to serve an OpenAI-compatible API with vLLM on a cloud GPU. A general SGLang install is How to run SGLang on a cloud GPU. Ollama’s install and port 11434 are How to run Ollama on a cloud GPU. An embedding server is How to serve embeddings and rerankers with Text Embeddings Inference on a cloud GPU. CloudGPU’s hosted gateway is a different base URL: How do I use an OpenAI-compatible API for open models?. How published weight sizes are read: How much VRAM do I need to run an LLM?. The tunnel is How to use SSH port forwarding on a cloud GPU. If the process runs out of memory: How to fix CUDA out of memory on a cloud GPU. If the process raises no kernel image on an RTX 5090: How to fix CUDA no kernel image on an RTX 5090.
The LICENSE file begins with “Apache License Version 2.0, January 2004”, its appendix copyright line names 2021 deepset GmbH, and the GitHub license field on 2026-11-14 was Apache-2.0.
The commands below are copied from the upstream files named in each step. They were not run on a CloudGPU instance for this page. Anything this page adds around them is marked. The README names other providers. This page does not list those names.
What the docs print about memory
The README, the Agent page, the OllamaChatGenerator page, and the VLLMChatGenerator page print no GPU GB figure for the chat examples. The Qwen3-0.6B card and the Qwen3-4B-Instruct-2507 card print no GPU GB figure. The vLLM integration README embeddings section prints “~14GB VRAM” for intfloat/e5-mistral-7b-instruct, “~2GB VRAM” for BAAI/bge-large-en-v1.5, and “~500MB VRAM” for sentence-transformers/all-MiniLM-L6-v2. This page does not treat those embedding figures as the chat model’s vram_gb. This page does not mark a catalog row as enough.
GET /v1/public/compute-capabilities does not list GPUs. It returns feature flags. The memory column is vram_gb from gpu-prices.json, updated_at 2026-11-14T07:15:49.882Z. GET /v1/public/gpu-catalog the same day (generated_at 2026-11-14T07:15:50.490Z) listed the same names and the same vram_gb values, plus Ascend 910B at 64, which is not in gpu-prices.json and is not listed here. The catalog response also includes an availability field. This page does not copy it. Every prices row has gpu_count 1. Every catalog row has min_gpu_count 1. Which listing you can start is on Deploy.
The catalog template list had no Haystack template. It did have Ollama templates (jupyterlab-ollama, ollama-qwen3, ollama). Those templates are a different install. This page does not treat them as Haystack.
| Catalog name | vram_gb |
|---|---|
| RTX 2080 | 8 |
| RTX 3080 | 10 |
| RTX 2080 Ti | 11 |
| RTX 3080 Ti | 12 |
| V100 16G | 16 |
| A30 | 24 |
| RTX 3090 | 24 |
| RTX 4090 | 24 |
| RTX 4090D | 24 |
| RTX 4080 SUPER 32G | 32 |
| RTX 5090 | 32 |
| RTX 5090D | 32 |
| V100 32G | 32 |
| A100 40G | 40 |
| A800 40G | 40 |
| A40 | 48 |
| RTX 4090 48G | 48 |
| RTX 4090D 48G | 48 |
| RTX A6000 | 48 |
| NVIDIA RTX PRO 5000 72G | 72 |
| RTX PRO 5000 72G | 72 |
| A100 80G | 80 |
| A800 80G | 80 |
| H800 | 80 |
| RTX 6000D | 83 |
| H20 96G | 96 |
| RTX PRO 6000 96G | 96 |
| RTX PRO 6000 Max-Q | 96 |
| H20-3e 141G | 141 |
Install
The README installation section, copied:
pip install haystack-ai
pyproject.toml says requires-python = ">=3.11" and license = "Apache-2.0". The version field is dynamic. haystack/version.py sets __version__ from the installed haystack-ai distribution and falls back to "main". PyPI info.version for haystack-ai on 2026-11-14 was 3.5.0, requires_python >=3.11, info.license empty, uploaded 2026-11-10T12:34:32.015851Z. Tag v3.5.0 is commit 831b8cfa95a250cea2134091986f734666cce47d, committed 2026-11-10T12:33:57Z. Main is commit 4b4346b6d798338d9b43e70115006494012b1756, committed 2026-11-13T12:05:35Z. Those are not the same commit. The README, LICENSE, pyproject.toml, the Agent page, the OllamaChatGenerator page, and the VLLMChatGenerator page are the same blob on both. This page did not install the package.
The Ollama page prints a second install. The vLLM page prints a third. Copied:
pip install ollama-haystack
pip install vllm-haystack
PyPI ollama-haystack on 2026-11-14 was 7.0.2, requires_python >=3.11, info.license empty, uploaded 2026-10-16T07:54:06.482739Z. PyPI vllm-haystack the same day was 3.0.1, requires_python >=3.11, info.license empty, uploaded 2026-10-20T09:02:33.283245Z. The vLLM chat-generator source at tag integrations/vllm-v3.0.1 (commit 9b16f0130675fe94fa7871737664f259e6808585, committed 2026-10-20T09:01:26Z) is the same blob as integrations main 8249f304723c78b2d92566b6b3ceaeeb35868edb (committed 2026-11-13T15:09:33Z). This page did not diff the Ollama integration source at tag integrations/ollama-v7.0.2 against the Ollama page.
Agent
The Agent page says chat_generator is the only mandatory init parameter, an instance of a Chat Generator that supports tools. The on-its-own example sets that argument to OpenAIChatGenerator with a closed-model id. This page does not copy that id or that constructor call. The tool and the run call, copied from that example:
from typing import Annotated
from haystack.components.agents import Agent
from haystack.dataclasses import ChatMessage
from haystack.tools import tool
@tool(outputs_to_state={"calc_result": {"source": "result"}})
def calculator(
expression: Annotated[str, "Math expression to evaluate, e.g. '7 * (4 + 2)'"],
) -> dict:
"""Evaluate basic math expressions."""
try:
result = eval(expression, {"__builtins__": {}})
return {"result": result}
except Exception as e:
return {"error": str(e)}
response = agent.run(messages=[ChatMessage.from_user("What is 7 * (4 + 2)?")])
print(response["last_message"].text)
agent in that snippet is the Agent the same example constructs. This page did not run it. The same page’s run-metadata sample prints response["last_message"] is not the only key. It also prints step_count, token_usage, tool_call_counts, and exit_reason.
Ollama and an OpenAI-compatible server
The Ollama page says OllamaChatGenerator needs a model name and a url. By default it uses "qwen3:0.6b" and "http://localhost:11434". The on-its-own example uses a different model string. Copied:
from haystack_integrations.components.generators.ollama import OllamaChatGenerator
from haystack.dataclasses import ChatMessage
generator = OllamaChatGenerator(
model="zephyr",
url="http://localhost:11434",
generation_kwargs={
"num_predict": 100,
"temperature": 0.9,
},
)
messages = [
ChatMessage.from_system("\nYou are a helpful, respectful and honest assistant"),
ChatMessage.from_user("What's Natural Language Processing?"),
]
print(generator.run(messages=messages))
zephyr and qwen3:0.6b are not the same string. The page also prints ollama pull zephyr and this Docker line:
docker run -d -p 11434:11434 --name ollama ollama/ollama:latest
The files this page read do not print a Hugging Face repository id for zephyr or qwen3:0.6b. This page does not add hf download for those tags. A streaming-with-tools example on the same page sets model="llama3.1:8b". That tag is not the default and is not the on-its-own example. How to run Ollama on a cloud GPU is the Ollama install.
The vLLM page says vLLM exposes an OpenAI-compatible HTTP server, which VLLMChatGenerator uses. It expects the server at api_base_url, by default http://localhost:8000/v1. The on-its-own example, copied:
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.vllm import VLLMChatGenerator
generator = VLLMChatGenerator(
model="Qwen/Qwen3-4B-Instruct-2507",
generation_kwargs={"max_tokens": 512, "temperature": 0.7},
)
messages = [ChatMessage.from_user("What's Natural Language Processing?")]
response = generator.run(messages=messages)
print(response["replies"][0].text)
The same page’s serve lines, copied:
vllm serve Qwen/Qwen3-4B-Instruct-2507
vllm serve Qwen/Qwen3-0.6B --enable-auto-tool-choice --tool-call-parser hermes
Those two model ids are not the same. The tool-calling example sets model="Qwen/Qwen3-0.6B". The integrations README also prints a vLLM-metal serve line for macOS. This page does not use that line. The Haystack files this page read do not print hf download. How to download a Hugging Face model prints pip install -U "huggingface_hub" and hf download. The block below is added on this page. This page did not download the repositories.
# Added on this page. The vLLM examples pass these ids to VLLMChatGenerator.
pip install -U "huggingface_hub"
hf download Qwen/Qwen3-4B-Instruct-2507
hf download Qwen/Qwen3-0.6B
On 2026-11-14, GET https://huggingface.co/api/models/Qwen/Qwen3-4B-Instruct-2507 reported gated false, cardData.license apache-2.0, pipeline_tag text-generation, lastModified 2025-09-17T06:56:53.000Z, and sha cdbee75f17c01a7cc42f958dc650907174af0554. The card README this page read prints no GPU GB figure. GET https://huggingface.co/api/models/Qwen/Qwen3-0.6B the same day reported gated false, cardData.license apache-2.0, pipeline_tag text-generation, lastModified 2025-07-26T03:46:27.000Z, and sha c1899de289a04d12100db370d81485cdf75e47ca. That card README prints no GPU GB figure. This page does not treat 0.6B or 4B in those ids as vram_gb.
How to serve an OpenAI-compatible API with vLLM on a cloud GPU is the general server install. The Ollama blocks print port 11434. The vLLM default prints port 8000. For a server you start on the instance, the SSH tunnel is How to use SSH port forwarding on a cloud GPU.
When it fails
CUDA out of memory. The Haystack chat pages this page read print no GPU GB figure for the Ollama and vLLM chat examples. If the process raises CUDA out of memory, use how to fix CUDA out of memory on a cloud GPU. This page does not pick a smaller model id for you.
“CUDA error: no kernel image is available for execution on the device” on an RTX 5090. The README and pyproject.toml do not name compute capability 12.0. The RTX 5090 guide says that card is compute capability 12.0 (sm_120) and that PyTorch 2.7 and later, on a build for CUDA 12.8 or newer, is the line that announced support. This page did not install a torch wheel and did not re-test this library on an RTX 5090. Catalog RTX 5090 and RTX 5090D are 32. That is the catalog number only.
What to copy off, then destroy
agent.run returns last_message. The vLLM example prints response["replies"][0].text. Copy the printed output before you destroy the instance. Save your work before you destroy a cloud GPU is the checklist. The terms say data on local instance storage is permanently deleted when the instance is destroyed.
The pricing page FAQ states the billing rule in these words:
Per-minute options have a 1-minute minimum; unused prepaid minutes are refunded when you destroy the instance. Some configurations use whole-hour billing with a 1-hour minimum; unused time within a charged hour is not refunded on voluntary stop. The billing period, minimum charge and initial prepayment are shown before confirmation. No subscription.
Prices are not copied here.
Frequently asked questions
What memory figure do the Haystack docs print?
The README, the Agent page, the OllamaChatGenerator page, and the VLLMChatGenerator page print no GPU GB figure for the chat examples. The Qwen3-0.6B card and the Qwen3-4B-Instruct-2507 card print no GPU GB figure. The vLLM integration README embeddings section prints ~14GB VRAM for intfloat/e5-mistral-7b-instruct, ~2GB VRAM for BAAI/bge-large-en-v1.5, and ~500MB VRAM for sentence-transformers/all-MiniLM-L6-v2. This page does not treat those embedding figures as the chat model vram_gb. This page does not mark a catalog row as enough. Which listing you can start is on https://cloudgpu.app/deploy.
Which install lines and classes do the docs print?
The README prints pip install haystack-ai. pyproject.toml says requires-python >=3.11 and license Apache-2.0. PyPI info.version for haystack-ai on 2026-11-14 was 3.5.0, requires_python >=3.11, info.license empty. Tag v3.5.0 and main are not the same commit. The README blob is the same on both. The Ollama page prints pip install ollama-haystack. The vLLM page prints pip install vllm-haystack. PyPI ollama-haystack on 2026-11-14 was 7.0.2. PyPI vllm-haystack on 2026-11-14 was 3.0.1. The Agent page says chat_generator is the only mandatory init parameter. Its on-its-own example sets OpenAIChatGenerator model to a closed-model id. This page does not copy that id. This page did not install the packages.
How do the docs point the generator at a local or OpenAI-compatible server?
The Ollama page says OllamaChatGenerator uses model qwen3:0.6b and url http://localhost:11434 by default. The on-its-own example sets model to zephyr and url to http://localhost:11434. Those model strings are not the same. The vLLM page says VLLMChatGenerator uses an OpenAI-compatible HTTP server and defaults api_base_url to http://localhost:8000/v1. The on-its-own example sets model to Qwen/Qwen3-4B-Instruct-2507. The tool-calling serve line is vllm serve Qwen/Qwen3-0.6B and names the flags enable-auto-tool-choice and tool-call-parser hermes. The files do not print an SSH command.
What license does the file print?
The LICENSE file begins with Apache License Version 2.0, January 2004. Its appendix copyright line names 2021 deepset GmbH. The GitHub license field on 2026-11-14 was Apache-2.0. PyPI info.license for haystack-ai was empty on 2026-11-14. That license text is not a CloudGPU price.
Sources (read 2026-11-14, Asia/Shanghai)
- deepset-ai/haystack README (
pip install haystack-ai): commit4b4346b6d798338d9b43e70115006494012b1756, committed 2026-11-13T12:05:35Z. GitHub license fieldApache-2.0.pushed_at2026-11-14T02:11:02Z.LICENSEbegins with Apache License Version 2.0, January 2004. The appendix copyright line names 2021 deepset GmbH. The README names other providers. This page does not list them - Tag
v3.5.0, commit831b8cfa95a250cea2134091986f734666cce47d, committed 2026-11-10T12:33:57Z. README,LICENSE,pyproject.toml, the Agent page, the OllamaChatGenerator page, and the VLLMChatGenerator page match main. PyPIhaystack-aiinfo.version3.5.0, uploaded 2026-11-10T12:34:32.015851Z,requires_python>=3.11,info.licenseempty docs-website/docs/pipeline-components/agents-1/agent.mdx(chat_generator, calculator tool,agent.run,last_message): same blob as the tag. TheOpenAIChatGeneratormodel id on that example is not copieddocs-website/docs/pipeline-components/generators/ollamachatgenerator.mdx(OllamaChatGenerator, defaultqwen3:0.6b,http://localhost:11434,zephyr,pip install ollama-haystack,ollama pull zephyr, Docker publish11434): same blob as the tag. This page did not diff that page against tagintegrations/ollama-v7.0.2. PyPIollama-haystack7.0.2, uploaded 2026-10-16T07:54:06.482739Zdocs-website/docs/pipeline-components/generators/vllmchatgenerator.mdx(VLLMChatGenerator, OpenAI-compatible server, defaulthttp://localhost:8000/v1,Qwen/Qwen3-4B-Instruct-2507,Qwen/Qwen3-0.6B,--enable-auto-tool-choice,--tool-call-parser hermes,pip install vllm-haystack): same blob as the tag. The chat-generator source matches integrations main and tagintegrations/vllm-v3.0.1. PyPIvllm-haystack3.0.1, uploaded 2026-10-20T09:02:33.283245Z. The macOS vLLM-metal serve line is not used on this pageintegrations/vllm/README.mdat integrations main (embedding VRAM lines~14GB,~2GB,~500MB): those lines are not the chat model’svram_gb- Hugging Face model API
Qwen/Qwen3-4B-Instruct-2507:gatedfalse,cardData.licenseapache-2.0,pipeline_tagtext-generation,lastModified2025-09-17T06:56:53.000Z,shacdbee75f17c01a7cc42f958dc650907174af0554. The card README prints no GPU GB figure.Qwen/Qwen3-0.6B:gatedfalse,cardData.licenseapache-2.0,pipeline_tagtext-generation,lastModified2025-07-26T03:46:27.000Z,shac1899de289a04d12100db370d81485cdf75e47ca. That card README prints no GPU GB figure.hf downloadis added on this page and was not run - Public GPU memory: https://cloudgpu.app/gpu-prices.json (
updated_at2026-11-14T07:15:49.882Z) and https://cloudgpu.app/v1/public/gpu-catalog (generated_at2026-11-14T07:15:50.490Z).GET /v1/public/compute-capabilitiesreturned feature flags and no GPU list. The catalog had no Haystack template
Cards you can start are on the Deploy page. Prices are on https://cloudgpu.app/pricing and are not repeated here.
Last updated: 2026-11-14 (Asia/Shanghai).