How to run RAGFlow with a self-hosted model on a cloud GPU

Rent a GPU on Deploy, then open a terminal in JupyterLab or over SSH. Check out tag v1.0.0 and start RAGFlow with the Docker Compose line that README prints, serve the chat model that page names with vLLM on the GPU, and point RAGFlow’s VLLM provider at that server. Dataset indexing also needs an embedding model. The local-model page says the chat model is not an embedding model, and it prints a separate Ollama embedding tag. Copy the dataset export or the downloaded weights before you destroy the instance, because destroying the instance deletes its disk.

An agent can rent the GPU and run this application against a model served on it. The agent pages are GPU compute for AI agents and the agent integration guide. Those pages are how an agent discovers a GPU and runs work on it. This page is the RAGFlow install. CloudGPU does not provide an embedding model for this app. A general vLLM install is How to serve an OpenAI-compatible API with vLLM on a cloud GPU. A general SGLang install is How to run SGLang on a cloud GPU. Ollama’s install and port 11434 are How to run Ollama on a cloud GPU. A gateway whose README prints uv tool install 'litellm[proxy]' is How to run LiteLLM Proxy in front of a self-hosted model on a cloud GPU. A pipeline whose Ollama page sets url to http://localhost:11434 is How to run Haystack agents with a self-hosted model on a cloud GPU. A framework whose README sets Ollama and HuggingFaceEmbedding is How to build LlamaIndex agents with a self-hosted model on a cloud GPU. A workspace app whose Ollama page sets the URL to http://127.0.0.1:11434 is How to run AnythingLLM agents with a self-hosted model on a cloud GPU. An embedding server whose README publishes port 8080 is How to serve embeddings and rerankers with Text Embeddings Inference on a cloud GPU. CloudGPU’s hosted gateway is a different base URL: How do I use an OpenAI-compatible API for open models?. How published weight sizes are read: How much VRAM do I need to run an LLM?. The tunnel is How to use SSH port forwarding on a cloud GPU. If the process runs out of memory: How to fix CUDA out of memory on a cloud GPU. If the process raises no kernel image on an RTX 5090: How to fix CUDA no kernel image on an RTX 5090.

The LICENSE file is the Apache License 2.0, and the GitHub license field on 2026-11-14 was Apache-2.0.

The commands below are copied from the upstream files named in each step. They were not run on a CloudGPU instance for this page. Anything this page adds around them is marked. The model-provider page names other providers. This page does not list those names.

What the docs print about memory

The v1.0.0 README prints a recommended starting configuration of 4 CPU cores and 16 GB RAM. The same sentence also prints a disk-space figure. This page does not copy that figure. The same sentence says actual requirements depend on the document engine, data volume, parsing tasks, and concurrency, and that local models and other optional components may require additional resources. The 16 GB figure is system RAM. It is not vram_gb.

The local-model page says the vLLM example limits context length to 8192 tokens to reduce GPU memory consumption. It prints no GPU GB figure. The Docker README says the optional embedding service’s memory requirement depends on the model, runtime backend, precision, batch-token limit, and concurrency, and that the tei-gpu profile primarily uses GPU memory and also consumes system RAM. It prints no GPU GB figure. The openbmb/MiniCPM5-2B model card fetched on 2026-11-14 prints a comparison table. This page does not copy that table and does not treat it as a GPU GB figure. This page does not mark a catalog row as enough.

GET /v1/public/compute-capabilities does not list GPUs. It returns feature flags. The memory column is vram_gb from GET /v1/public/gpu-catalog (generated_at 2026-11-14T19:13:01.996Z). The catalog response also includes an availability field. This page does not copy it. Every catalog row below has min_gpu_count 1. Which listing you can start is on Deploy.

The catalog template list had no RAGFlow template. It did have Ollama templates (jupyterlab-ollama, ollama-qwen3, ollama). Those templates are a different install. This page does not treat them as RAGFlow.

Catalog name vram_gb
RTX 2080 8
RTX 3080 10
RTX 2080 Ti 11
RTX 3080 Ti 12
V100 16G 16
A30 24
RTX 3090 24
RTX 4090 24
RTX 4090D 24
RTX 4080 SUPER 32G 32
RTX 5090 32
RTX 5090D 32
V100 32G 32
A100 40G 40
A800 40G 40
A40 48
RTX 4090 48G 48
RTX 4090D 48G 48
RTX A6000 48
Ascend 910B 64
NVIDIA RTX PRO 5000 72G 72
RTX PRO 5000 72G 72
A100 80G 80
A800 80G 80
H800 80
RTX 6000D 83
H20 96G 96
RTX PRO 6000 96G 96
RTX PRO 6000 Max-Q 96
H20-3e 141G 141

Install

Tag v1.0.0 is commit db1cd5929a7885a3074547791f5ecc4d2f891739, committed 2026-11-06T12:08:30Z, published 2026-11-06T12:31:31Z. Main on 2026-11-14 was commit f9a53a3779a12b5abb8a254ec9384c2af7fae4b4, committed 2026-11-13T12:55:30Z. pushed_at was 2026-11-13T13:08:52Z. Those commits are not the same. The install lines in this section are on the tag v1.0.0 README. This page did not diff the rest of the two READMEs.

The README prerequisite line for Docker, copied in words: Docker >= 24.0.0 and Docker Compose >= v2.26.1. The Elasticsearch step, copied:

sysctl vm.max_map_count

sudo sysctl -w vm.max_map_count=262144

The README says that change resets after a reboot, and that a permanent change is the line vm.max_map_count=262144 in /etc/sysctl.conf. It says this step is usually unnecessary with Infinity. The clone and Compose lines, copied:

git clone https://github.com/infiniflow/ragflow.git

cd ragflow/docker

git checkout v1.0.0

docker compose -f docker-compose.yml up -d

The README says to use docker ps, then:

curl -f http://localhost/api/v1/system/healthz

It says an HTTP 200 means you can open the Web UI. The log command it prints is docker logs --tail 50 ragflow-cpu. The default web URL it prints is http://IP_OF_YOUR_MACHINE with no port, because SVR_WEB_HTTP_PORT defaults to 80. In docker/.env on that tag, the uncommented image line is RAGFLOW_IMAGE=infiniflow/ragflow:v1.0.0, and DEVICE=${DEVICE:-cpu}. The same file’s commented image lines name other registries. This page does not copy those lines. The Docker README says the RAGFlow Docker image does not include embedding models.

The files this page read do not print an SSH command. For the web port and for a model server you start on the instance, the tunnel is How to use SSH port forwarding on a cloud GPU.

Chat model on the GPU, embedding model beside it

The local-model page at tag v1.0.0 (docs/guides/models/deploy_local_llm.mdx) says RAGFlow supports Ollama and vLLM, and it uses openbmb/MiniCPM5-2B for both. It says that model is a chat model, not an embedding model, and that a Dataset also needs an embedding model. The Ollama embedding example it prints is qwen3-embedding:0.6b.

The vLLM serve command, copied. The page says this command listens on all interfaces and that port 8000 should not be open to the public internet:

vllm serve openbmb/MiniCPM5-2B \
  --served-model-name MiniCPM5-2B \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype auto \
  --max-model-len 8192

The page says --served-model-name is the name RAGFlow must use. The check it prints, copied:

curl http://localhost:8000/v1/models

It says the response should contain a model whose ID is MiniCPM5-2B. When RAGFlow runs in Docker and vLLM runs on the same host, the base URL it prints is http://host.docker.internal:8000/v1. When both run on the same host outside Docker, it prints http://localhost:8000/v1. The third row is a host-IP placeholder. This page does not repeat that token. The container check it prints uses a container-name placeholder. Find the name with the command it prints:

sudo docker ps \
  --filter "label=com.docker.compose.service=ragflow-cpu" \
  --format '{{.Names}}'

The VLLM provider steps, copied in order: open the avatar menu, select Model providers, find VLLM, add an instance, enter that base URL, leave API Key empty unless you set --api-key on vllm serve, add MiniCPM5-2B as a Chat model, verify it, and save. Then select MiniCPM5-2B in Set default models. The page says you can use that vLLM chat model with an embedding model from another provider, and its example is qwen3-embedding:0.6b from Ollama.

The Ollama pull it prints for that embedding tag, copied:

sudo docker exec ollama ollama pull qwen3-embedding:0.6b

The chat pull on the same page is sudo docker exec ollama ollama pull openbmb/minicpm5-2b. The Ollama base URL it prints for RAGFlow in Docker on the same host is http://host.docker.internal:11434. On the same host outside Docker it prints http://localhost:11434. Add qwen3-embedding:0.6b as an Embedding model, verify, and save. The page says the names must match the names Ollama returns. A general Ollama install is How to run Ollama on a cloud GPU. This page does not claim the rented GPU serves embeddings through CloudGPU.

On 2026-11-14, GET https://huggingface.co/api/models/openbmb/MiniCPM5-2B reported gated false, private false, cardData.license apache-2.0, pipeline_tag text-generation, lastModified 2026-11-09T08:14:20.000Z, and sha 42433e90dad95123ab87729fd7011e7110629556. The Ollama library page for qwen3-embedding:0.6b, fetched the same day, shows the labels 639MB, parameters 596M, and quantization Q8_0 next to that tag. This page did not find a license string on that page. Those labels are not vram_gb. The library page does not print hf download. This page does not add a Hugging Face id for that tag.

docker/.env on tag v1.0.0 sets TEI_MODEL default to Qwen/Qwen3-Embedding-0.6B, TEI_IMAGE_GPU to infiniflow/text-embeddings-inference:1.8, and TEI_PORT to 6380. docker/docker-compose-base.yml defines a service tei-gpu with profile tei-gpu, image ${TEI_IMAGE_GPU}, and command ["--model-id", "/data/${TEI_MODEL}", "--auto-truncate"]. The Docker README says that profile starts a local text-embeddings-inference service and that the RAGFlow image does not include embedding models. The default COMPOSE_PROFILES line is ${DOC_ENGINE},${DEVICE},metadata-${METADATA_DB_PROFILE}. It does not include tei-gpu. The README does not print a command that adds the profile. This page does not add one. On 2026-11-14, GET https://huggingface.co/api/models/Qwen/Qwen3-Embedding-0.6B reported gated false, private false, cardData.license apache-2.0, pipeline_tag feature-extraction, lastModified 2026-04-20T02:45:58.000Z, and sha 97b0c614be4d77ee51c0cef4e5f07c00f9eb65b3.

The model-provider page says to open User settings, then Model providers, create an instance, set Base URL to the service address for a local or compatible API, add a custom model whose name matches the model id the provider exposes, verify, and save. It says the minimum defaults are Default LLM and Default embedding model. The v1.0.0 README links a /docs/dev/llm_api_key_setup path. Fetched on 2026-11-14, that path returned HTTP 404. The page that returned this text is the URL above.

Dataset and chat

The quickstart tutorial section, and docs/quickstart.mdx on the tag, say to open User settings, then Model providers, create an instance, add the Chat and Embedding models, verify, save, and set both as defaults. If they use different providers, configure an instance for each. Then:

  1. Select Dataset, click Create dataset, enter a name, select the Embedding model, choose Parsing mode General, and click Save.
  2. On Files, click Add file, then Upload file, select a document, enable Parse on creation, and click Save.
  3. Select Chat, click Create chat, open Chat setting, select the Chat model and the Dataset, enable Show citations, and save.

The quickstart says to check that parsing finished and the document has chunks before asking a question. It says a model that is not in the list has not been added and verified. This page did not upload a document.

When it fails

CUDA out of memory. The RAGFlow files this page read print no GPU GB figure for the chat model. The vLLM example lowers --max-model-len to 8192 and says that reduces GPU memory consumption. If the model server raises CUDA out of memory, use how to fix CUDA out of memory on a cloud GPU. This page does not pick a smaller model id for you.

“CUDA error: no kernel image is available for execution on the device” on an RTX 5090. The RAGFlow README does not name compute capability 12.0. The RTX 5090 guide says that card is compute capability 12.0 (sm_120) and that PyTorch 2.7 and later, on a build for CUDA 12.8 or newer, is the line that announced support. This page did not install a torch wheel and did not re-test this program on an RTX 5090. Catalog RTX 5090 and RTX 5090D are 32. That is the catalog number only.

What to copy off, then destroy

The quickstart’s dataset and chat live in the Compose project on the instance. Copy any uploaded file and any weights vllm serve or ollama pull stored on the instance before you destroy it. Save your work before you destroy a cloud GPU is the checklist. The terms say data on local instance storage is permanently deleted when the instance is destroyed.

The pricing page FAQ states the billing rule in these words:

Per-minute options have a 1-minute minimum; unused prepaid minutes are refunded when you destroy the instance. Some configurations use whole-hour billing with a 1-hour minimum; unused time within a charged hour is not refunded on voluntary stop. The billing period, minimum charge and initial prepayment are shown before confirmation. No subscription.

Prices are not copied here.

Frequently asked questions

What memory figure do the RAGFlow docs print?

The v1.0.0 README prints 4 CPU cores and 16 GB RAM. The same sentence prints a disk-space figure. This page does not copy that figure. The 16 GB figure is system RAM, not vram_gb. The local-model page prints --max-model-len 8192 and no GPU GB figure. The Docker README says the tei-gpu profile uses GPU memory and prints no GPU GB figure. This page does not mark a catalog row as enough. Which listing you can start is on https://cloudgpu.app/deploy.

Which install lines does the v1.0.0 README print?

The README prints Docker >= 24.0.0 and Docker Compose >= v2.26.1, then git clone https://github.com/infiniflow/ragflow.git, cd ragflow/docker, git checkout v1.0.0, and docker compose -f docker-compose.yml up -d. Tag v1.0.0 is commit db1cd5929a7885a3074547791f5ecc4d2f891739, published 2026-11-06T12:31:31Z. docker/.env on that tag sets RAGFLOW_IMAGE to infiniflow/ragflow:v1.0.0 and DEVICE default to cpu. The health check is curl -f http://localhost/api/v1/system/healthz. The log command is docker logs --tail 50 ragflow-cpu. The Docker README says the RAGFlow image does not include embedding models. This page did not run the Compose command.

How do the docs point RAGFlow at a chat model and an embedding model?

The local-model page serves openbmb/MiniCPM5-2B with vllm serve, --served-model-name MiniCPM5-2B, and port 8000. When RAGFlow is in Docker, the vLLM base URL it prints is http://host.docker.internal:8000/v1. It says MiniCPM5-2B is a chat model, not an embedding model. The embedding example it prints is ollama pull qwen3-embedding:0.6b, added as an Embedding model. The model-provider page says to set Default LLM and Default embedding model. docker/.env defaults TEI_MODEL to Qwen/Qwen3-Embedding-0.6B under profile tei-gpu. The default COMPOSE_PROFILES line does not include tei-gpu, and the README does not print a command that adds it. This page does not add one. CloudGPU does not provide that embedding model. The model API on 2026-11-14 reported openbmb/MiniCPM5-2B and Qwen/Qwen3-Embedding-0.6B gated false.

What license does the file print?

The LICENSE file is the Apache License 2.0, and the GitHub license field on 2026-11-14 was Apache-2.0. The openbmb/MiniCPM5-2B card license was apache-2.0. The Qwen/Qwen3-Embedding-0.6B card license was apache-2.0.

Sources (read 2026-11-14, Asia/Shanghai)

  • infiniflow/ragflow tag v1.0.0, commit db1cd5929a7885a3074547791f5ecc4d2f891739, committed 2026-11-06T12:08:30Z, published 2026-11-06T12:31:31Z. README: 4 CPU cores, 16 GB RAM, disk figure not copied, Docker >= 24.0.0, Docker Compose >= v2.26.1, vm.max_map_count 262144, git checkout v1.0.0, docker compose -f docker-compose.yml up -d, curl -f http://localhost/api/v1/system/healthz, docker logs --tail 50 ragflow-cpu. Main commit f9a53a3779a12b5abb8a254ec9384c2af7fae4b4, committed 2026-11-13T12:55:30Z. GitHub license field Apache-2.0. LICENSE is the Apache License 2.0
  • docker/.env on that tag: RAGFLOW_IMAGE=infiniflow/ragflow:v1.0.0, DEVICE=${DEVICE:-cpu}, TEI_MODEL default Qwen/Qwen3-Embedding-0.6B, TEI_IMAGE_GPU=infiniflow/text-embeddings-inference:1.8, TEI_PORT=6380. Commented registry lines are not copied
  • docker/docker-compose-base.yml on that tag: service tei-gpu, profile tei-gpu, command ["--model-id", "/data/${TEI_MODEL}", "--auto-truncate"]. Default COMPOSE_PROFILES does not list tei-gpu. No enable command is printed in the Docker README
  • Docker README on that tag: the RAGFlow image does not include embedding models. tei-gpu uses GPU memory and system RAM. No GPU GB figure
  • Local models https://ragflow.io/docs/deploy_local_llm and docs/guides/models/deploy_local_llm.mdx on tag v1.0.0: vllm serve openbmb/MiniCPM5-2B, --served-model-name MiniCPM5-2B, --max-model-len 8192, base URL http://host.docker.internal:8000/v1, embedding qwen3-embedding:0.6b. The page says the chat model is not an embedding model
  • Model providers https://ragflow.io/docs/llm_api_key_setup, fetched 2026-11-14. The README path /docs/dev/llm_api_key_setup returned HTTP 404 the same day. Minimum defaults: Default LLM and Default embedding model. Other provider names are not copied
  • Quickstart https://ragflow.io/docs/ and docs/quickstart.mdx on the tag: Dataset, Embedding model, Parsing mode General, Chat model, Show citations
  • Hugging Face model API openbmb/MiniCPM5-2B: gated false, cardData.license apache-2.0, pipeline_tag text-generation, lastModified 2026-11-09T08:14:20.000Z, sha 42433e90dad95123ab87729fd7011e7110629556. The comparison table is not copied
  • Hugging Face model API Qwen/Qwen3-Embedding-0.6B: gated false, cardData.license apache-2.0, pipeline_tag feature-extraction, lastModified 2026-04-20T02:45:58.000Z, sha 97b0c614be4d77ee51c0cef4e5f07c00f9eb65b3
  • Ollama library page https://ollama.com/library/qwen3-embedding:0.6b, fetched 2026-11-14: labels 639MB, parameters 596M, quantization Q8_0. No license string found on that page. Not vram_gb
  • Public GPU memory: https://cloudgpu.app/v1/public/gpu-catalog (generated_at 2026-11-14T19:13:01.996Z). GET /v1/public/compute-capabilities returned feature flags and no GPU list. The catalog had no RAGFlow template

Cards you can start are on the Deploy page. Prices are on https://cloudgpu.app/pricing and are not repeated here.

Last updated: 2026-11-14 (Asia/Shanghai).