How to run DSPy with a self-hosted model on a cloud GPU
Rent a GPU on Deploy, then open a terminal in JupyterLab or over SSH. Install DSPy the way the official README prints, serve a model on that machine, and point dspy.LM at the Ollama model string or the OpenAI-compatible api_base the docs print. Copy result.haiku or the printed answer before you destroy the instance, because destroying the instance deletes its disk.
An agent can rent the GPU and run this library on it. The agent pages are GPU compute for AI agents and the agent integration guide. Those pages are how an agent discovers a GPU and runs work on it. This page is the DSPy install. A Python agent library whose CodeAgent can take TransformersModel or an OpenAI-compatible server is How to run smolagents with a self-hosted model on a cloud GPU. A crew whose LLM class takes base_url for Ollama or an OpenAI-compatible server is How to run CrewAI with a self-hosted model on a cloud GPU. A graph runtime whose overview example does not name a model is How to run LangGraph agents with a self-hosted model on a cloud GPU. An AgentChat library whose Ollama example sets model to llama3.2 is How to run AutoGen agents with a self-hosted model on a cloud GPU. A typed agent whose vLLM example sets base_url to http://localhost:8000/v1 is How to run Pydantic AI agents with a self-hosted model on a cloud GPU. A framework whose README sets Ollama and HuggingFaceEmbedding is How to build LlamaIndex agents with a self-hosted model on a cloud GPU. A browser agent whose models page sets base_url to http://localhost:11434/v1 is How to run browser-use with a self-hosted model on a cloud GPU. A pipeline whose OllamaChatGenerator sets url to http://localhost:11434 is How to run Haystack agents with a self-hosted model on a cloud GPU. An agent whose Ollama example sets id to llama3.1:8b is How to run Agno agents with a self-hosted model on a cloud GPU. A stateful CLI whose connect help sets --base-url to http://localhost:8000/v1 for openai-compatible is How to run Letta agents with a self-hosted model on a cloud GPU. A TypeScript agent whose local-models section sets url on the model object is How to run Mastra agents with a self-hosted model on a cloud GPU. A ChatAgent whose vLLM example sets ModelPlatformType.VLLM is How to run CAMEL-AI agents with a self-hosted model on a cloud GPU. A general vLLM install is How to serve an OpenAI-compatible API with vLLM on a cloud GPU. A general SGLang install is How to run SGLang on a cloud GPU. Ollama’s install and port 11434 are How to run Ollama on a cloud GPU. An embedding server is How to serve embeddings and rerankers with Text Embeddings Inference on a cloud GPU. CloudGPU’s hosted gateway is a different base URL: How do I use an OpenAI-compatible API for open models?. How published weight sizes are read: How much VRAM do I need to run an LLM?. The tunnel is How to use SSH port forwarding on a cloud GPU. If the process runs out of memory: How to fix CUDA out of memory on a cloud GPU. If the process raises no kernel image on an RTX 5090: How to fix CUDA no kernel image on an RTX 5090. A Docker Compose app whose self-host page opens http://localhost/install is How to run Dify with a self-hosted model on a cloud GPU. A canvas whose ChatOllama page names a base URL is How to run Flowise with a self-hosted model on a cloud GPU. A flow editor whose Ollama component defaults the base URL to http://127.0.0.1:11434 is How to run Langflow with a self-hosted model on a cloud GPU.
The LICENSE file begins with “MIT License”, its copyright line names Stanford Future Data Systems 2023, and the GitHub license field on 2026-11-14 was MIT.
The commands below are copied from the upstream files named in each step. They were not run on a CloudGPU instance for this page. Anything this page adds around them is marked. The installation page and the language-models page also name other providers and set some model strings to closed-model ids. This page does not list those names and does not copy those ids.
What the docs print about memory
The README, the installation page, and the language-models page print no GPU GB figure. On 2026-11-14, GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3-8B-Instruct reported gated manual, cardData.license llama3, pipeline_tag text-generation, lastModified 2025-06-18T23:49:51.000Z, and sha 8afb486c1db24fe5011ec46dfbe5b5dccdb575c2. The card README returned HTTP 401, so this page does not copy a GPU GB figure from that card. This page does not treat 8B in that repository id as vram_gb. This page does not mark a catalog row as enough.
GET /v1/public/compute-capabilities does not list GPUs. It returns feature flags. The memory column is vram_gb from gpu-prices.json, updated_at 2026-11-14T07:15:49.882Z. GET /v1/public/gpu-catalog the same day (generated_at 2026-11-14T07:15:50.490Z) listed the same names and the same vram_gb values, plus Ascend 910B at 64, which is not in gpu-prices.json and is not listed here. The catalog response also includes an availability field. This page does not copy it. Every prices row has gpu_count 1. Every catalog row has min_gpu_count 1. Which listing you can start is on Deploy.
The catalog template list had no DSPy template. It did have Ollama templates (jupyterlab-ollama, ollama-qwen3, ollama). Those templates are a different install. This page does not treat them as DSPy.
| Catalog name | vram_gb |
|---|---|
| RTX 2080 | 8 |
| RTX 3080 | 10 |
| RTX 2080 Ti | 11 |
| RTX 3080 Ti | 12 |
| V100 16G | 16 |
| A30 | 24 |
| RTX 3090 | 24 |
| RTX 4090 | 24 |
| RTX 4090D | 24 |
| RTX 4080 SUPER 32G | 32 |
| RTX 5090 | 32 |
| RTX 5090D | 32 |
| V100 32G | 32 |
| A100 40G | 40 |
| A800 40G | 40 |
| A40 | 48 |
| RTX 4090 48G | 48 |
| RTX 4090D 48G | 48 |
| RTX A6000 | 48 |
| NVIDIA RTX PRO 5000 72G | 72 |
| RTX PRO 5000 72G | 72 |
| A100 80G | 80 |
| A800 80G | 80 |
| H800 | 80 |
| RTX 6000D | 83 |
| H20 96G | 96 |
| RTX PRO 6000 96G | 96 |
| RTX PRO 6000 Max-Q | 96 |
| H20-3e 141G | 141 |
Install
The README installation section, copied:
pip install dspy
The installation page also prints uv add dspy and says DSPy works in Python 3.10 or newer environments. pyproject.toml on main says version="3.5.0" and requires-python = ">=3.10, <3.16". Tag 3.5.0 is commit 4e9db7a83e71e4c36315a2402ae7ffd0c6493404, committed 2026-11-10T16:18:23Z. That commit’s pyproject.toml says version="3.4.1". Main is commit 2c07ebbee3a66f6e04b71e141c6c198f1b31a129, committed 2026-11-12T14:09:40Z. Those version strings are not the same. The README, LICENSE, the installation page, and the language-models page are the same blob on both. dspy/_vendor/lm15/compat.py is also the same blob on both. PyPI info.version for dspy on 2026-11-14 was 3.5.0, requires_python >=3.10, <3.16, uploaded 2026-11-10T16:24:08.845107Z. The publish workflow comment in pyproject.toml says a marker can replace the version field. This page did not install the package.
A minimal program
The first-program page, copied:
haiku_signature = "subject -> haiku"
haiku_generator = dspy.Predict(haiku_signature)
result = haiku_generator(subject="computer science")
print(result.haiku)
That page says the call checks that an LM is configured. The installation page sets it with dspy.configure(lm=lm) before this program. This page did not run haiku_generator. The language-models page also prints dspy.ChainOfThought('question -> answer'). That is a different module. This page does not copy the closed-model ids on that later section.
Ollama and an OpenAI-compatible server
The installation page lists Local Ollama as the model string ollama_chat/llama3.2:1b and says to start Ollama and pull the model. It says the native route supplies its local address and a placeholder key. It does not print the URL in that table cell. dspy/_vendor/lm15/router.py maps the prefix ollama_chat to ollama. The same mapping names other prefixes. This page does not list them. OPENAI_CHAT_PRESET_BASE_URLS in dspy/_vendor/lm15/compat.py sets "ollama" to "http://localhost:11434/v1". AccessPolicy OLLAMA sets base_url to that value. The registry entry sets placeholder_key="ollama" and the note "local ollama server (keyless)".
The language-models page laptop section prints a > prompt before each shell line. The commands, copied:
curl -fsSL https://ollama.ai/install.sh | sh
ollama run llama3.2:1b
The Python that follows, copied:
import dspy
lm = dspy.LM('ollama_chat/llama3.2:1b') # native local-server address and placeholder key
dspy.configure(lm=lm)
llama3.2:1b is an Ollama tag. The files this page read do not print a Hugging Face repository id for it. This page does not add hf download for that tag. How to run Ollama on a cloud GPU is the Ollama install. The blocks above print port 11434 in the preset URL. The ollama run line does not print a port.
The same language-models page has a GPU-server section. It says to install SGLang and launch its server, then connect as an OpenAI-compatible endpoint. The page prints a > prompt before each shell line. The commands, copied:
pip install "sglang[all]"
pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/
CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server --port 7501 --model-path meta-llama/Meta-Llama-3-8B-Instruct
The client, copied:
lm = dspy.LM("openai/meta-llama/Meta-Llama-3-8B-Instruct",
api_base="http://localhost:7501/v1", # ensure this points to your port
api_key="EMPTY", model_type='chat')
dspy.configure(lm=lm)
That api_base is port 7501. The preset URL for the name sglang in OPENAI_CHAT_PRESET_BASE_URLS is http://localhost:30000/v1. Those ports are not the same. This page does not treat them as one server. The model string uses the openai/ prefix the page prints for an OpenAI-compatible endpoint. How to run SGLang on a cloud GPU is the general SGLang install. The language-models page does not print hf download. The block below is added on this page. This page did not download the repository. The model API reported gated manual. A gated repository can refuse the download.
# Added on this page. The GPU-server example passes this id to sglang.launch_server and to dspy.LM.
pip install -U "huggingface_hub"
hf download meta-llama/Meta-Llama-3-8B-Instruct
The installation page’s OpenAI-compatible Chat Completions example, copied:
lm = dspy.LM(
"openai/your-model-name",
api_base="https://your-provider.example/v1",
api_key="YOUR_PROVIDER_KEY",
)
your-model-name and https://your-provider.example/v1 are the placeholders that page prints. The placeholder does not print a numeric port. This page does not replace them with a port. A vLLM server on the instance is a different install: How to serve an OpenAI-compatible API with vLLM on a cloud GPU. OPENAI_CHAT_PRESET_BASE_URLS sets "vllm" to "http://localhost:8000/v1". The language-models GPU example does not use that URL.
For a server you start on the instance, the SSH tunnel is How to use SSH port forwarding on a cloud GPU. The Ollama preset prints port 11434. The GPU-server example prints port 7501.
When it fails
CUDA out of memory. The DSPy files this page read print no GPU GB figure. If the process raises CUDA out of memory, use how to fix CUDA out of memory on a cloud GPU. This page does not pick a smaller model id for you.
“CUDA error: no kernel image is available for execution on the device” on an RTX 5090. The installation page and pyproject.toml do not name compute capability 12.0. The flashinfer index in the GPU-server example names cu121/torch2.4. The RTX 5090 guide says that card is compute capability 12.0 (sm_120) and that PyTorch 2.7 and later, on a build for CUDA 12.8 or newer, is the line that announced support. This page did not install a torch wheel and did not re-test this library on an RTX 5090. Catalog RTX 5090 and RTX 5090D are 32. That is the catalog number only.
What to copy off, then destroy
print(result.haiku) prints the program output in the process that called it. Copy that text before you destroy the instance. Save your work before you destroy a cloud GPU is the checklist. The terms say data on local instance storage is permanently deleted when the instance is destroyed.
The pricing page FAQ states the billing rule in these words:
Per-minute options have a 1-minute minimum; unused prepaid minutes are refunded when you destroy the instance. Some configurations use whole-hour billing with a 1-hour minimum; unused time within a charged hour is not refunded on voluntary stop. The billing period, minimum charge and initial prepayment are shown before confirmation. No subscription.
Prices are not copied here.
Frequently asked questions
What memory figure do the DSPy docs print?
The README, the installation page, and the language-models page print no GPU GB figure. The model API for meta-llama/Meta-Llama-3-8B-Instruct reports gated manual. Its card README returned HTTP 401, so this page does not copy a GPU GB figure from that card. This page does not treat 8B in that repository id as vram_gb. This page does not mark a catalog row as enough. Which listing you can start is on https://cloudgpu.app/deploy.
Which install lines and the first program do the docs print?
The README prints pip install dspy. The installation page also prints uv add dspy and says Python 3.10 or newer. pyproject.toml on main prints version 3.5.0 and requires-python >=3.10, <3.16. The tag 3.5.0 commit prints version 3.4.1 in pyproject.toml. Those version strings are not the same. The installation page blob is the same on both. PyPI info.version on 2026-11-14 was 3.5.0, requires_python >=3.10, <3.16. The first program sets a signature to subject -> haiku and calls dspy.Predict. This page did not install the package.
How do the docs point dspy.LM at a local or OpenAI-compatible server?
The installation page lists Local Ollama with the model string ollama_chat/llama3.2:1b and says the native route supplies its local address and a placeholder key. router.py maps the prefix ollama_chat to ollama. OPENAI_CHAT_PRESET_BASE_URLS sets ollama to http://localhost:11434/v1. The language-models page runs ollama run llama3.2:1b and sets dspy.LM to ollama_chat/llama3.2:1b. The same page GPU-server example sets api_base to http://localhost:7501/v1 and the model string to openai/meta-llama/Meta-Llama-3-8B-Instruct. The sglang preset URL is http://localhost:30000/v1. Those ports are not the same. The OpenAI-compatible example on the installation page sets api_base to https://your-provider.example/v1. That value does not print a numeric port. The same pages also set model strings to closed-model ids. This page does not copy those ids.
What license does the file print?
The LICENSE file begins with MIT License. Its copyright line names Stanford Future Data Systems 2023. The GitHub license field on 2026-11-14 was MIT. PyPI info.license begins with MIT License. That license text is not a CloudGPU price.
Sources (read 2026-11-14, Asia/Shanghai)
- stanfordnlp/dspy README (
pip install dspy): commit2c07ebbee3a66f6e04b71e141c6c198f1b31a129, committed 2026-11-12T14:09:40Z. GitHub license fieldMIT.pushed_at2026-11-13T14:13:09Z.LICENSEbegins with MIT License. The copyright line names Stanford Future Data Systems 2023 - Tag
3.5.0, commit4e9db7a83e71e4c36315a2402ae7ffd0c6493404, committed 2026-11-10T16:18:23Z. That commit’spyproject.tomlprintsversion="3.4.1". Main printsversion="3.5.0". README,LICENSE,docs/docs/getting-started/installation.md,docs/docs/getting-started/first-program.md,docs/docs/learn/programming/language_models.md, anddspy/_vendor/lm15/compat.pymatch main. PyPIdspyinfo.version3.5.0, uploaded 2026-11-10T16:24:08.845107Z,requires_python>=3.10, <3.16 docs/docs/getting-started/installation.md(uv add dspy, Python 3.10+,ollama_chat/llama3.2:1b,dspy.LMwithapi_base="https://your-provider.example/v1"): same blob as the tag. Other provider rows and closed-model ids on that page are not copieddocs/docs/getting-started/first-program.md(dspy.Predict,"subject -> haiku",result.haiku): same blob as the tagdocs/docs/learn/programming/language_models.md(ollama run llama3.2:1b,dspy.LM('ollama_chat/llama3.2:1b'), SGLang--port 7501,meta-llama/Meta-Llama-3-8B-Instruct,http://localhost:7501/v1): same blob as the tag. The page prints a>prompt before the shell lines. Closed-model ids on that page are not copieddspy/_vendor/lm15/router.pymapsollama_chattoollama. Other prefixes in that mapping are not listed.dspy/_vendor/lm15/compat.pyOPENAI_CHAT_PRESET_BASE_URLSsetsollamatohttp://localhost:11434/v1,vllmtohttp://localhost:8000/v1, andsglangtohttp://localhost:30000/v1. Other entries in that dict are not listed.dspy/_vendor/lm15/access.pyOLLAMA.base_urluses the ollama preset.dspy/_vendor/lm15/registry.pysetsplaceholder_key="ollama"on that entry- Hugging Face model API
meta-llama/Meta-Llama-3-8B-Instruct:gatedmanual,cardData.licensellama3,pipeline_tagtext-generation,lastModified2025-06-18T23:49:51.000Z,sha8afb486c1db24fe5011ec46dfbe5b5dccdb575c2. The card README returned HTTP 401.hf downloadis added on this page and was not run - Public GPU memory: https://cloudgpu.app/gpu-prices.json (
updated_at2026-11-14T07:15:49.882Z) and https://cloudgpu.app/v1/public/gpu-catalog (generated_at2026-11-14T07:15:50.490Z).GET /v1/public/compute-capabilitiesreturned feature flags and no GPU list. The catalog had no DSPy template
Cards you can start are on the Deploy page. Prices are on https://cloudgpu.app/pricing and are not repeated here.
Last updated: 2026-11-14 (Asia/Shanghai).