How to run Dify with a self-hosted model on a cloud GPU
Rent a GPU on Deploy, then open a terminal in JupyterLab or over SSH. Clone a Dify release the way the self-host docs print, run the Docker Compose commands those docs print, and point the Ollama or OpenAI-API-compatible plugin at a model server on that machine. Copy the exported agent file and any files the agent wrote before you destroy the instance, because destroying the instance deletes its disk.
An agent can rent the GPU and run this application on it. The agent pages are GPU compute for AI agents and the agent integration guide. Those pages are how an agent discovers a GPU and runs work on it. This page is the Dify install. A canvas whose ChatOllama page names a base URL is How to run Flowise with a self-hosted model on a cloud GPU. A flow editor whose Ollama component defaults the base URL to http://127.0.0.1:11434 is How to run Langflow with a self-hosted model on a cloud GPU. A CLI whose connect help sets --base-url to http://localhost:8000/v1 for openai-compatible is How to run Letta agents with a self-hosted model on a cloud GPU. A browser chat whose serve command listens on port 8080 is How to run Open WebUI on a cloud GPU. A Python agent library whose CodeAgent can take TransformersModel or an OpenAI-compatible server is How to run smolagents with a self-hosted model on a cloud GPU. A crew whose LLM class takes base_url for Ollama or an OpenAI-compatible server is How to run CrewAI with a self-hosted model on a cloud GPU. A graph runtime whose overview example does not name a model is How to run LangGraph agents with a self-hosted model on a cloud GPU. An AgentChat library whose Ollama example sets model to llama3.2 is How to run AutoGen agents with a self-hosted model on a cloud GPU. A typed agent whose vLLM example sets base_url to http://localhost:8000/v1 is How to run Pydantic AI agents with a self-hosted model on a cloud GPU. A framework whose README sets Ollama and HuggingFaceEmbedding is How to build LlamaIndex agents with a self-hosted model on a cloud GPU. A browser agent whose models page sets base_url to http://localhost:11434/v1 is How to run browser-use with a self-hosted model on a cloud GPU. A pipeline whose OllamaChatGenerator sets url to http://localhost:11434 is How to run Haystack agents with a self-hosted model on a cloud GPU. A program whose language-models page sets dspy.LM to ollama_chat/llama3.2:1b is How to run DSPy with a self-hosted model on a cloud GPU. An agent whose Ollama example sets id to llama3.1:8b is How to run Agno agents with a self-hosted model on a cloud GPU. A TypeScript agent whose local-models section sets url on the model object is How to run Mastra agents with a self-hosted model on a cloud GPU. A ChatAgent whose vLLM example sets ModelPlatformType.VLLM is How to run CAMEL-AI agents with a self-hosted model on a cloud GPU. A general vLLM install is How to serve an OpenAI-compatible API with vLLM on a cloud GPU. A general SGLang install is How to run SGLang on a cloud GPU. Ollama’s install and port 11434 are How to run Ollama on a cloud GPU. An embedding server is How to serve embeddings and rerankers with Text Embeddings Inference on a cloud GPU. CloudGPU’s hosted gateway is a different base URL: How do I use an OpenAI-compatible API for open models?. How published weight sizes are read: How much VRAM do I need to run an LLM?. The tunnel is How to use SSH port forwarding on a cloud GPU. If the process runs out of memory: How to fix CUDA out of memory on a cloud GPU. If the process raises no kernel image on an RTX 5090: How to fix CUDA no kernel image on an RTX 5090.
The LICENSE file says Dify is licensed under a modified version of the Apache License 2.0, with additional conditions, and the copyright line is © 2025 LangGenius, Inc.
The commands below are copied from the upstream files and docs pages named in each step. They were not run on a CloudGPU instance for this page. Anything this page adds around them is marked. The README’s model-support sentence names model families this page does not copy. It also says “any OpenAI API-compatible models.”
What the docs print about memory
The README quick start and the Docker Compose doc both print CPU at least 2 cores and RAM at least 4 GiB. The Docker Compose doc’s macOS row also prints 8 GiB of memory for the Docker virtual machine. Those lines are system memory. They are not vram_gb. The Ollama plugin manifest prints resource.memory: 268435456. The OpenAI-API-compatible plugin manifest prints resource.memory: 1048576. Those integers are the plugin resource field. This page does not treat them as vram_gb. The model information table on unsloth/Llama-3.1-8B-Instruct prints parameter counts 8B, 70B, and 405B, and a context length of 128k. This page does not treat those parameter counts as vram_gb. A later section of that card names training hardware. This page does not copy that section and does not use it as an inference memory figure. This page does not mark a catalog row as enough.
GET /v1/public/compute-capabilities does not list GPUs. It returns feature flags. The memory column is vram_gb from GET /v1/public/gpu-catalog (generated_at 2026-11-14T09:10:32.863Z). The catalog response also includes an availability field. This page does not copy it. Every catalog row below has min_gpu_count 1. Which listing you can start is on Deploy.
The catalog template list had no Dify template. It did have Ollama templates (jupyterlab-ollama, ollama-qwen3, ollama). Those templates are a different install. This page does not treat them as Dify.
| Catalog name | vram_gb |
|---|---|
| RTX 2080 | 8 |
| RTX 3080 | 10 |
| RTX 2080 Ti | 11 |
| RTX 3080 Ti | 12 |
| V100 16G | 16 |
| A30 | 24 |
| RTX 3090 | 24 |
| RTX 4090 | 24 |
| RTX 4090D | 24 |
| RTX 4080 SUPER 32G | 32 |
| RTX 5090 | 32 |
| RTX 5090D | 32 |
| V100 32G | 32 |
| A100 40G | 40 |
| A800 40G | 40 |
| A40 | 48 |
| RTX 4090 48G | 48 |
| RTX 4090D 48G | 48 |
| RTX A6000 | 48 |
| Ascend 910B | 64 |
| NVIDIA RTX PRO 5000 72G | 72 |
| RTX PRO 5000 72G | 72 |
| A100 80G | 80 |
| A800 80G | 80 |
| H800 | 80 |
| RTX 6000D | 83 |
| H20 96G | 96 |
| RTX PRO 6000 96G | 96 |
| RTX PRO 6000 Max-Q | 96 |
| H20-3e 141G | 141 |
Install
Main of langgenius/dify on 2026-11-14 was commit 29f044b57602e251058925988cba693bbea56370, committed 2026-11-14T06:13:34Z. pushed_at the same day was 2026-11-14T09:01:59Z. The GitHub license field was NOASSERTION. Release tag 1.19.0 is commit 8e689e868b63b6dd2b449d0eb568ca5cc7127759, committed 2026-11-04T07:19:22Z, published 2026-11-04T07:36:16Z. Those commits are not the same. This page did not diff the tag 1.19.0 README against main. The README and docker/README.md quoted here are from that main commit. The docs pages were fetched the same day from the URLs in Sources.
The Docker Compose doc prints the clone before any command inside docker/. The branch is an angle-bracket token. The fence below is that command. Replace the token with a tag from the releases page the doc links. This page did not clone the repository.
git clone --branch <version> --depth 1 https://github.com/langgenius/dify.git
cd dify/docker
The README quick start does not print that clone. It starts at cd docker from the repository root. After the clone above, dify/docker is that directory, so do not run cd docker a second time. From that directory, the README prints:
cp .env.example .env
openssl rand -hex 32
docker compose up -d
docker compose ps --all
That README says to open .env and set DIFY_AGENT_LOCAL_SANDBOX_AUTH_TOKEN to the generated value, keep the default COMPOSE_PROFILES value, then open http://localhost/install. It also says Docker Compose v2.35.0 or later, and OpenSSL. .env.example at the same commit sets COMPOSE_PROFILES=weaviate,postgresql,collaboration and leaves DIFY_AGENT_LOCAL_SANDBOX_AUTH_TOKEN empty. The comment in that file says to generate the token with openssl rand -hex 32.
docker/docker-compose.yaml at the same commit publishes nginx on port 80 when EXPOSE_NGINX_PORT and NGINX_PORT are unset. The Docker Compose doc fetched the same day says the default HTTP port is 80, the setup URL is http://localhost/install, and sign-in is http://localhost. It also says that for a remote deployment you replace localhost with the server’s IP address or domain, and that if port 80 is in use you set EXPOSE_NGINX_PORT in docker/.env. The README’s docker guide prints the example EXPOSE_NGINX_PORT=8080 and then http://localhost:8080/install. Those two URLs are not the same. The default in the compose file is 80. The files this page read do not print an SSH command. For a server you start on the instance, the tunnel is How to use SSH port forwarding on a cloud GPU.
The same doc says long-running services should show Up or healthy, and the init_permissions task shows Exited (0) when it is done. The README says that one-time service is expected to exit with code 0.
Agent
The self-host page Build an Agent, fetched 2026-11-14, says: from Agents, click Create > Create from Blank, give the agent a name, then shape it in Configure. The Model section says to pick the chat model the agent runs on, and to favor models that natively support tool calling. It does not print a model id. Publish says edits autosave as a draft, and that Access Point can host a web app or call the service API. Export Agent exports the current draft.
That page’s run limits, copied as figures and not as a CloudGPU billing rule: a single agent run is stopped after 1 hour by default, and every call the agent makes to its model counts toward a cap of 500 per run. Each file the agent sends back in a reply can be up to 50 MB by default. The longer-run variables it names are APP_MAX_EXECUTION_TIME, WORKFLOW_MAX_EXECUTION_TIME, and DIFY_AGENT_RUN_TIMEOUT_SECONDS.
The Agent API page says Agent apps run in agent mode and that reply text streams as agent_message events. Send Chat Message is streaming mode only, and blocking mode returns a 400 bad_request error. The OpenAPI block on that page is post /chat-messages. Its server description says that for self-hosted deployments you replace the base URL with your own API base URL. The variable default printed there is api.dify.ai/v1. That default is not http://localhost. This page does not invent a joined URL the OpenAPI block does not print.
The same build page’s troubleshooting accordion says: “Models served through OpenAI-compatible endpoints (vLLM and similar) often lack full native tool-calling support.” It then says to pick a model that supports tool calling natively, and to check Token parameter name because some newer models require max_completion_tokens. The rest of that accordion names a vendor. This page does not copy that sentence.
Ollama and an OpenAI-compatible server
The workspace page Model Providers, fetched 2026-11-14, says to install a provider from Integrations > Model Provider, then Add API Key, including a custom endpoint URL when the provider asks for one. That page does not name Ollama, vLLM, or a base URL.
The provider plugins live in langgenius/dify-official-plugins. Main on 2026-11-14 was commit 60804e30f6fa41fbe890c02c9849b0bdbe28d13a, committed 2026-11-11T03:33:57Z. The GitHub license field was Apache-2.0. models/ollama and models/openai_api_compatible exist at that commit. models/vllm/README.md returned 404. This page does not invent a vLLM provider form.
The Ollama plugin README at that commit says to use the Ollama host root as the Base URL, http://localhost:11434 for a local server, and not to include /api because the plugin appends /api/chat, /api/generate, and /api/embed. Its run lines, copied. The same block prints other tags. This page does not copy those tags.
ollama pull qwen3
After startup, that README says the local API is available at http://localhost:11434. The add-model steps say Settings > Model Providers > Ollama. The Base URL line prints an angle-bracket host. This page writes that token as <your-ollama-host> so the renderer does not treat it as a tag: http://<your-ollama-host>:11434 for local Ollama. Model Name examples this page does copy are qwen3 and deepseek-r1. Other example tags on that line are not copied. For Docker deployments, the same README says to use a host address reachable from the Dify container, such as http://host.docker.internal:11434 or a LAN IP address.
The OpenAI-API-compatible plugin manifest label.en_US is OpenAI-API-compatible. Its description.en_US is “Model providers compatible with OpenAI’s API standard, such as LM Studio or llama.cpp Service.” The manifest version is 0.0.83. The README says to configure the model with Type, Name, API Key, and URL. The two examples under that README use remote base URLs and model names this page does not copy. That README does not print http://localhost:8000/v1. How to serve an OpenAI-compatible API with vLLM on a cloud GPU is the general server install. How to run Ollama on a cloud GPU is the Ollama install.
The Ollama plugin README does not print hf download. The block below is added on this page. How to download a Hugging Face model prints pip install -U "huggingface_hub" and hf download. This page did not download the repository. The Dify files this page read do not print this repository id.
pip install -U "huggingface_hub"
hf download unsloth/Llama-3.1-8B-Instruct
On 2026-11-14, GET https://huggingface.co/api/models/unsloth/Llama-3.1-8B-Instruct reported gated false, cardData.license llama3.1, pipeline_tag text-generation, lastModified 2025-05-09T19:32:57.000Z, and sha 4699cc75b550f9c6f3173fb80f4703b62d946aa5. The card YAML also includes extra_gated_prompt. Those two fields are not the same fact. This page did not run hf download.
When it fails
CUDA out of memory. The Dify files and docs pages this page read print no GPU GB figure. If the model server raises CUDA out of memory, use how to fix CUDA out of memory on a cloud GPU. This page does not pick a smaller model id for you.
“CUDA error: no kernel image is available for execution on the device” on an RTX 5090. The Dify README and the plugin READMEs this page read do not name compute capability 12.0. The RTX 5090 guide says that card is compute capability 12.0 (sm_120) and that PyTorch 2.7 and later, on a build for CUDA 12.8 or newer, is the line that announced support. This page did not install a torch wheel and did not re-test this program on an RTX 5090. Catalog RTX 5090 and RTX 5090D are 32. That is the catalog number only.
What to copy off, then destroy
Export Agent writes the current draft. The build page also says the agent handles files in its sandbox. Copy that export and those files before you destroy the instance. Save your work before you destroy a cloud GPU is the checklist. The terms say data on local instance storage is permanently deleted when the instance is destroyed.
The billing page states the billing rule in these words:
Per-minute options have a 1-minute minimum; unused prepaid minutes are refunded when you destroy the instance. Some configurations use whole-hour billing with a 1-hour minimum; unused time within a charged hour is not refunded on voluntary stop. The billing period, minimum charge and initial prepayment are shown before confirmation. No subscription.
Amounts are not copied here.
Frequently asked questions
What memory figure do the Dify docs print?
The README and the Docker Compose doc print CPU at least 2 cores and RAM at least 4 GiB. The macOS row prints 8 GiB for the Docker virtual machine. The Ollama plugin manifest prints resource.memory 268435456. The OpenAI-API-compatible plugin manifest prints resource.memory 1048576. The unsloth/Llama-3.1-8B-Instruct model information table prints parameter counts 8B, 70B, and 405B, and context length 128k. This page does not treat those figures as vram_gb. This page does not mark a catalog row as enough. Which listing you can start is on https://cloudgpu.app/deploy.
Which install lines do the docs print?
The Docker Compose doc prints git clone and cd dify/docker first. The README quick start does not print that clone. It starts at cd docker from the repository root, then cp .env.example .env, openssl rand -hex 32, docker compose up -d, and docker compose ps --all, then http://localhost/install. After the clone, dify/docker is that directory, so do not run cd docker a second time. It says Docker Compose v2.35.0 or later. Main is commit 29f044b57602e251058925988cba693bbea56370, committed 2026-11-14. Tag 1.19.0 is an earlier commit. The compose file publishes EXPOSE_NGINX_PORT with a default of 80. The Docker Compose doc says the default HTTP port is 80. This page did not clone the repository and did not run the commands.
How do the docs point Dify at a local or OpenAI-compatible server?
The Ollama plugin README says the local Base URL is http://localhost:11434, without /api, and for Docker http://host.docker.internal:11434 or a LAN IP. The copied pull line is ollama pull qwen3. The OpenAI-API-compatible manifest label is OpenAI-API-compatible. That plugin README does not print http://localhost:8000/v1. models/vllm/README.md at the plugins commit returned 404. The agent build page says OpenAI-compatible endpoints (vLLM and similar) often lack full native tool-calling support. The files do not print an SSH command.
What license does the file print?
The LICENSE file says Dify is licensed under a modified version of the Apache License 2.0, with additional conditions, and the copyright line is © 2025 LangGenius, Inc. The GitHub license field on 2026-11-14 was NOASSERTION. That license text is not a CloudGPU rate.
Sources (read 2026-11-14, Asia/Shanghai)
- langgenius/dify README and
docker/README.mdat commit29f044b57602e251058925988cba693bbea56370, committed 2026-11-14T06:13:34Z. GitHub license fieldNOASSERTION.pushed_at2026-11-14T09:01:59Z. Tag1.19.0is commit8e689e868b63b6dd2b449d0eb568ca5cc7127759, committed 2026-11-04T07:19:22Z. This page did not diff that tag against main.docker/.env.examplesetsCOMPOSE_PROFILES=weaviate,postgresql,collaboration.docker/docker-compose.yamlpublishes nginx on port 80 whenEXPOSE_NGINX_PORTandNGINX_PORTare unset - LICENSE at that commit: modified Apache License 2.0 with additional conditions. Copyright line © 2025 LangGenius, Inc.
- Docker Compose doc https://docs.dify.ai/en/self-host/deploy/quick-start/docker-compose, fetched 2026-11-14. CPU at least 2 cores. RAM at least 4 GiB. macOS Docker virtual machine 8 GiB. Default HTTP port 80.
http://localhost/install. Clone command with a release tag - Build an Agent https://docs.dify.ai/en/self-host/use-dify/build/new-agent/build, fetched 2026-11-14. Create from Blank. Run cap 1 hour. Model-request cap 500. Reply file cap 50 MB. OpenAI-compatible endpoints (vLLM and similar) and
max_completion_tokens. The vendor sentence in that accordion is not copied - Agent API https://docs.dify.ai/en/api-reference/guides/agent and Send Chat Message https://docs.dify.ai/en/api-reference/chat-messages/send-chat-message, fetched 2026-11-14.
post /chat-messages. Server defaultapi.dify.ai/v1. Self-hosted note says to replace the base URL. Streaming only. Blocking mode returns 400bad_request - Model Providers https://docs.dify.ai/en/self-host/use-dify/workspace/model-providers, fetched 2026-11-14. Integrations, Model Provider, Add API Key, custom endpoint URL. The page does not name Ollama or vLLM
- langgenius/dify-official-plugins commit
60804e30f6fa41fbe890c02c9849b0bdbe28d13a, committed 2026-11-11T03:33:57Z. GitHub license field Apache-2.0.models/ollama/README.mdandmodels/openai_api_compatible/README.md.models/vllm/README.mdreturned 404. Ollama manifest version 1.0.3,resource.memory268435456. OpenAI-API-compatible manifest version 0.0.83,resource.memory1048576, labelOpenAI-API-compatible - Hugging Face model API
unsloth/Llama-3.1-8B-Instruct:gatedfalse,cardData.licensellama3.1,pipeline_tagtext-generation,lastModified2025-05-09T19:32:57.000Z,sha4699cc75b550f9c6f3173fb80f4703b62d946aa5. The card YAML includesextra_gated_prompt. The model information table prints 8B, 70B, 405B, and 128k.hf downloadis added on this page and was not run. The training-hardware section is not copied - Public GPU memory: https://cloudgpu.app/v1/public/gpu-catalog (
generated_at2026-11-14T09:10:32.863Z).GET /v1/public/compute-capabilitiesreturned feature flags and no GPU list. The catalog had no Dify template
Cards you can start are on the Deploy page. Amounts are not copied here.
Last updated: 2026-11-14 (Asia/Shanghai).