How to run AnythingLLM agents with a self-hosted model on a cloud GPU
Rent a GPU on Deploy, then open a terminal in JupyterLab or over SSH. Install AnythingLLM with the Docker command the official quickstart prints, serve a model on that machine, and point the Ollama provider or the Generic OpenAI provider at a base URL those docs print. Copy the STORAGE_LOCATION directory, including the .env file mounted into the container, before you destroy the instance, because destroying the instance deletes its disk.
An agent can rent the GPU and run this application on it. The agent pages are GPU compute for AI agents and the agent integration guide. Those pages are how an agent discovers a GPU and runs work on it. This page is the AnythingLLM install. A terminal agent whose install page runs curl -fsSL https://www.openinterpreter.com/install | sh, and whose models page sets the Ollama base URL to http://localhost:11434/v1, is How to run Open Interpreter with a self-hosted model on a cloud GPU. A repository agent whose local-models section sets api_base to http://localhost:11434 is How to run SWE-agent with a self-hosted model on a cloud GPU. A workflow editor whose Docker install publishes port 5678, and whose Ollama credentials default the Base URL to http://localhost:11434, is How to run n8n AI agents with a self-hosted model on a cloud GPU. A CLI whose providers page sets OLLAMA_HOST to http://localhost:11434 is How to run the goose agent with a self-hosted model on a cloud GPU. A coding agent whose providers page sets package to the OpenAI-compatible provider is How to use OpenCode with a self-hosted model on a cloud GPU. A canvas whose self-host page opens http://localhost/install is How to run Dify with a self-hosted model on a cloud GPU. A canvas whose ChatOllama page names a base URL is How to run Flowise with a self-hosted model on a cloud GPU. A flow editor whose Ollama component defaults the base URL to http://127.0.0.1:11434 is How to run Langflow with a self-hosted model on a cloud GPU. A CLI whose connect help sets --base-url to http://localhost:8000/v1 for openai-compatible is How to run Letta agents with a self-hosted model on a cloud GPU. A browser chat whose serve command listens on port 8080 is How to run Open WebUI on a cloud GPU. A Python agent library whose CodeAgent can take TransformersModel or an OpenAI-compatible server is How to run smolagents with a self-hosted model on a cloud GPU. A crew whose LLM class takes base_url for Ollama or an OpenAI-compatible server is How to run CrewAI with a self-hosted model on a cloud GPU. A graph runtime whose overview example does not name a model is How to run LangGraph agents with a self-hosted model on a cloud GPU. An AgentChat library whose Ollama example sets model to llama3.2 is How to run AutoGen agents with a self-hosted model on a cloud GPU. A typed agent whose vLLM example sets base_url to http://localhost:8000/v1 is How to run Pydantic AI agents with a self-hosted model on a cloud GPU. A framework whose README sets Ollama and HuggingFaceEmbedding is How to build LlamaIndex agents with a self-hosted model on a cloud GPU. A browser agent whose models page sets base_url to http://localhost:11434/v1 is How to run browser-use with a self-hosted model on a cloud GPU. A pipeline whose OllamaChatGenerator sets url to http://localhost:11434 is How to run Haystack agents with a self-hosted model on a cloud GPU. A program whose language-models page sets dspy.LM to ollama_chat/llama3.2:1b is How to run DSPy with a self-hosted model on a cloud GPU. An agent whose Ollama example sets id to llama3.1:8b is How to run Agno agents with a self-hosted model on a cloud GPU. A TypeScript agent whose local-models section sets url on the model object is How to run Mastra agents with a self-hosted model on a cloud GPU. A ChatAgent whose vLLM example sets ModelPlatformType.VLLM is How to run CAMEL-AI agents with a self-hosted model on a cloud GPU. A local agent UI whose Docker command publishes port 3000 is How to run OpenHands with a self-hosted model on a cloud GPU. A general vLLM install is How to serve an OpenAI-compatible API with vLLM on a cloud GPU. A general SGLang install is How to run SGLang on a cloud GPU. Ollama’s install and port 11434 are How to run Ollama on a cloud GPU. CloudGPU’s hosted gateway is a different base URL: How do I use an OpenAI-compatible API for open models?. How published weight sizes are read: How much VRAM do I need to run an LLM?. The tunnel is How to use SSH port forwarding on a cloud GPU. If the process runs out of memory: How to fix CUDA out of memory on a cloud GPU. If the process raises no kernel image on an RTX 5090: How to fix CUDA no kernel image on an RTX 5090.
The README says this project is MIT licensed.
The commands below are copied from the upstream files and docs pages named in each step. They were not run on a CloudGPU instance for this page. Anything this page adds around them is marked. The quickstart’s first sentence uses a timing comparison this page does not copy. Provider names on the docs navigation are not copied, except the Ollama provider and the Generic OpenAI provider.
What the docs print about memory
The system requirements page, from docs commit f931945ab77570b662c2bfb7db2c98ff96563a35 committed 2026-11-13T17:51:44Z, prints a recommended table: RAM 2GB, a 2-core CPU with AVX2, and a storage row. The storage row is not copied. The RAM cell is system memory. It is not vram_gb. The repository file docker/HOW_TO_USE_DOCKER.md at the commit in Sources says you should aim for at least 2GB of RAM. That line is also system memory. The same tip prints a disk sentence this page does not copy. The built-in engine section of that Docker file says an empty context window uses each model’s own context length, capped at 16,384 tokens. That cap is a token count. It is not vram_gb. The model information table on unsloth/Llama-3.1-8B-Instruct prints parameter counts 8B, 70B, and 405B, and a context length of 128k. This page does not treat those parameter counts as vram_gb. A later section of that card names training hardware. This page does not copy that section and does not use it as an inference memory figure. This page does not mark a catalog row as enough.
GET /v1/public/compute-capabilities does not list GPUs. It returns feature flags. The memory column is vram_gb from GET /v1/public/gpu-catalog (generated_at 2026-11-14T11:11:10.599Z). The catalog response also includes an availability field. This page does not copy it. Every catalog row below has min_gpu_count 1. Rows are sorted by vram_gb, then by name. Which listing you can start is on Deploy.
The catalog template list had no AnythingLLM template. It did have Ollama templates (jupyterlab-ollama, ollama-qwen3, ollama). Those templates are a different install. This page does not treat them as AnythingLLM.
| Catalog name | vram_gb |
|---|---|
| RTX 2080 | 8 |
| RTX 3080 | 10 |
| RTX 2080 Ti | 11 |
| RTX 3080 Ti | 12 |
| V100 16G | 16 |
| A30 | 24 |
| RTX 3090 | 24 |
| RTX 4090 | 24 |
| RTX 4090D | 24 |
| RTX 4080 SUPER 32G | 32 |
| RTX 5090 | 32 |
| RTX 5090D | 32 |
| V100 32G | 32 |
| A100 40G | 40 |
| A800 40G | 40 |
| A40 | 48 |
| RTX 4090 48G | 48 |
| RTX 4090D 48G | 48 |
| RTX A6000 | 48 |
| Ascend 910B | 64 |
| NVIDIA RTX PRO 5000 72G | 72 |
| RTX PRO 5000 72G | 72 |
| A100 80G | 80 |
| A800 80G | 80 |
| H800 | 80 |
| RTX 6000D | 83 |
| H20 96G | 96 |
| RTX PRO 6000 96G | 96 |
| RTX PRO 6000 Max-Q | 96 |
| H20-3e 141G | 141 |
Install
Main of Mintplex-Labs/anything-llm on 2026-11-14 was commit 4d09fa5f730698d39f1d783e47316744b260a700, committed 2026-11-13T22:29:19Z. pushed_at was 2026-11-13T22:55:56Z. The GitHub license field was MIT. The docs site banner the same day said v1.19.0. This page did not diff that version label against the commit.
docker/HOW_TO_USE_DOCKER.md at that commit prints docker pull mintplexlabs/anythingllm, then this Linux and macOS run. The image has no tag. The command includes --rm.
export STORAGE_LOCATION=$HOME/anythingllm && \
mkdir -p $STORAGE_LOCATION && \
touch "$STORAGE_LOCATION/.env" && \
docker run -d --rm -p 3001:3001 \
--cap-add SYS_ADMIN \
-v ${STORAGE_LOCATION}:/app/server/storage \
-v ${STORAGE_LOCATION}/.env:/app/server/.env \
-e STORAGE_DIR="/app/server/storage" \
mintplexlabs/anythingllm
The docs quickstart, pages/installation-docker/quickstart.mdx at docs commit f931945ab77570b662c2bfb7db2c98ff96563a35, prints a different command. The image is mintplexlabs/anythingllm:latest. The command does not include --rm.
export STORAGE_LOCATION=$HOME/anythingllm && \
mkdir -p $STORAGE_LOCATION && \
touch "$STORAGE_LOCATION/.env" && \
docker run -d -p 3001:3001 \
--cap-add SYS_ADMIN \
-v ${STORAGE_LOCATION}:/app/server/storage \
-v ${STORAGE_LOCATION}/.env:/app/server/.env \
-e STORAGE_DIR="/app/server/storage" \
mintplexlabs/anythingllm:latest
Both pages say the editor is at http://localhost:3001. The repository docker/docker-compose.yml at the same app commit is a third install. It builds from ./docker/Dockerfile, publishes 3001:3001, sets extra_hosts to host.docker.internal:host-gateway, and mounts ./.env, ../server/storage, and the collector hotdir and outputs directories. It does not name the Hub image. This page does not merge the three installs.
The Docker file’s NVIDIA built-in engine command is a different run. It adds --gpus all and ENABLE_BUILT_IN_LLM=true. It says Linux needs the NVIDIA Container Toolkit first, and that CUDA needs a Turing or RTX 20 series GPU or newer. This page does not mark a catalog row as that series.
docker run -d -p 3001:3001 \
--cap-add SYS_ADMIN \
--gpus all \
-v ${STORAGE_LOCATION}:/app/server/storage \
-v ${STORAGE_LOCATION}/.env:/app/server/.env \
-e STORAGE_DIR="/app/server/storage" \
-e ENABLE_BUILT_IN_LLM=true \
mintplexlabs/anythingllm
The health check that file prints, after a first chat, is:
docker exec <container> node -e 'fetch("http://127.0.0.1:13355/api/v1/health").then(r=>r.json()).then(h=>h.all_models_loaded.forEach(m=>console.log(m.model_name, m.recipe_options?.llamacpp_backend)))'
The angle-bracket token is the container name placeholder that file prints. NVIDIA GPUs should print cuda. The engine port inside the container is EMBEDDED_LEMONADE_PORT, default 13355. That port is not published in the run command.
The files this page read do not print an SSH command. For the editor on port 3001, the tunnel is How to use SSH port forwarding on a cloud GPU.
Ollama and the Generic OpenAI provider
The Ollama LLM page says Ollama is a separate application, and that when Ollama is running locally you should connect with http://127.0.0.1:11434 when using the default settings. The UI placeholder in OllamaLLMOptions at the app commit is the same URL.
The localhost page says localhost, 127.0.0.1, and 0.0.0.0 do not work from the AnythingLLM container. It says Linux must use 172.17.0.1 instead of host.docker.internal. The troubleshooting page lists three URLs it tries to detect: http://127.0.0.1:11434, http://host.docker.internal:11434, and http://172.17.0.1:11434. It rewrites the Ollama URL to http://host.docker.internal:11434 on Windows and macOS, and to http://172.17.0.1:11434 on Linux. docker/HOW_TO_USE_DOCKER.md says Linux should add --add-host=host.docker.internal:host-gateway. The embedded compose example in that file sets OLLAMA_BASE_PATH to http://127.0.0.1:11434 and OLLAMA_MODEL_PREF to llama2. docker/.env.example sets OLLAMA_BASE_PATH to http://host.docker.internal:11434, OLLAMA_MODEL_PREF to llama2, and OLLAMA_MODEL_TOKEN_LIMIT to 4096. Those strings are not the same URL. This page does not choose one.
The Generic OpenAI page says the provider is for an endpoint that is OpenAI-compatible in API functionality and inference response. It says to set Base URL to the provider’s API root, for example https://llm.example.com/v1, rather than the full chat-completions URL, and to use the exact model ID that endpoint serves. Max Tokens defaults to 1024. Setting it to 0 omits the output-token limit. The environment names it prints are GENERIC_OPEN_AI_MODEL_TOKEN_LIMIT and GENERIC_OPEN_AI_MAX_TOKENS. docker/.env.example sets LLM_PROVIDER to generic-openai and names GENERIC_OPEN_AI_BASE_PATH, GENERIC_OPEN_AI_MODEL_PREF, and GENERIC_OPEN_AI_API_KEY. The example values for the base path and the model preference name a host and a model id this page does not copy. The UI placeholder in GenericOpenAiOptions names a host this page does not copy.
How to serve an OpenAI-compatible API with vLLM on a cloud GPU is the general server install. How to run Ollama on a cloud GPU is the Ollama install. The AnythingLLM files this page read do not print a vLLM port.
The AnythingLLM files this page read do not print hf download. The block below is added on this page. How to download a Hugging Face model prints pip install -U "huggingface_hub" and hf download. This page did not download the repository. The AnythingLLM files this page read do not print this repository id. llama2 is the model preference the compose example and .env.example print. This page does not treat it as this repository id.
pip install -U "huggingface_hub"
hf download unsloth/Llama-3.1-8B-Instruct
On 2026-11-14, GET https://huggingface.co/api/models/unsloth/Llama-3.1-8B-Instruct reported gated false, cardData.license llama3.1, pipeline_tag text-generation, lastModified 2025-05-09T19:32:57.000Z, and sha 4699cc75b550f9c6f3173fb80f4703b62d946aa5. The card YAML also includes extra_gated_prompt. Those two fields are not the same fact. The card’s base_model field names a repository this page does not copy. This page did not run hf download.
Agent skills
The AI agents page says agents are an LLM with tools, invoked in a workspace with @agent, and that you exit by typing exit. The usage page says you must mention @agent when the @ symbol is present in the prompt, and that you end the session with /exit. Those two exit strings are not the same. It says a started session logs Agent @agent invoked and a finished session logs Agent session completed. The tool titles it lists include RAG Search, Web Browsing, Web Scraping, Save Files, List Documents, Summarize Documents, Chart Generation, SQL Agent, File System Agent, and Create Scheduled Jobs. Other titles on the docs navigation name products this page does not copy.
The setup page says the workspace uses the system LLM for agentic chat unless you change the model per workspace. It says Settings > Agent Skills is where you choose skills. Some skills are enabled by default. Some need more configuration. It says to click Save. It also says the prompt input has a Tools button. A sentence on that page uses a word this page does not copy. The settings icon on the same page enables Intelligent Tool Selection, which the page says adds only the tools that are useful for the chat.
The agent not using tools page says a real tool call shows a thought in the UI, and that a reply without that thought may be invented. It says smaller quantized models often fail to emit JSON that matches the tool schema. A comparison on that page names a closed model this page does not copy. The fixes it prints are to change the model, to run /reset, and to turn off tools you are not using.
When it fails
CUDA out of memory. The AnythingLLM files and docs pages this page read print no GPU GB figure. The 2GB lines above are system RAM. If the model server raises CUDA out of memory, use how to fix CUDA out of memory on a cloud GPU. This page does not pick a smaller model id for you.
“CUDA error: no kernel image is available for execution on the device” on an RTX 5090. The AnythingLLM README and the docs pages this page read do not name compute capability 12.0. The RTX 5090 guide says that card is compute capability 12.0 (sm_120) and that PyTorch 2.7 and later, on a build for CUDA 12.8 or newer, is the line that announced support. This page did not install a torch wheel and did not re-test this program on an RTX 5090. Catalog RTX 5090 and RTX 5090D are 32. That is the catalog number only.
localhost from Docker. The localhost page and the Ollama troubleshooting page say localhost and 127.0.0.1 refer to the container, not the host. The troubleshooting page’s Linux rewrite is http://172.17.0.1:11434. The Docker how-to instead tells Linux to add --add-host=host.docker.internal:host-gateway. This page does not choose between those two lines.
AVX2. The system requirements page says the CPU must support AVX2, and it prints this failure when LanceDB loads on a CPU without it: Illegal instruction (core dumped) node /app/server/index.js. The check it prints is lscpu | grep -o avx2.
Built-in engine not on the GPU. The Docker file says the health line should print cuda on NVIDIA, and that without --gpus all the model can run on the CPU. It also says Vulkan can run on the CPU when the container cannot open the GPU.
What to copy off, then destroy
The quickstart and the Docker how-to both set STORAGE_LOCATION to $HOME/anythingllm and mount it at /app/server/storage, and they mount $STORAGE_LOCATION/.env at /app/server/.env. The Docker how-to says that mount is so a later image update does not delete existing data. The repository compose file mounts different paths: ./.env, ../server/storage, ../collector/hotdir, and ../collector/outputs. Copy the directory you actually mounted, and the .env file, before you destroy the instance. This page does not print a database dump command. The quickstart’s pg image command is a different image, mintplexlabs/anythingllm:pg, and this page does not copy the database URL from that command.
Save your work before you destroy a cloud GPU is the checklist. The terms say data on local instance storage is permanently deleted when the instance is destroyed.
The billing page states the billing rule in these words:
Per-minute options have a 1-minute minimum; unused prepaid minutes are refunded when you destroy the instance. Some configurations use whole-hour billing with a 1-hour minimum; unused time within a charged hour is not refunded on voluntary stop. The billing period, minimum charge and initial prepayment are shown before confirmation. No subscription.
Amounts are not copied here.
Frequently asked questions
What memory figure do the AnythingLLM docs print?
The Docker system requirements table prints RAM 2GB. The Docker how-to tip prints at least 2GB of RAM. Those lines are system memory. They are not vram_gb. Both pages also print a storage line this page does not copy. The built-in engine section says an empty context window uses the model’s own context length, capped at 16,384 tokens. That cap is not vram_gb. The unsloth/Llama-3.1-8B-Instruct model information table prints parameter counts 8B, 70B, and 405B, and context length 128k. This page does not treat those figures as vram_gb. This page does not mark a catalog row as enough. Which listing you can start is on https://cloudgpu.app/deploy.
Which install lines do the docs print?
The repository Docker page prints docker pull mintplexlabs/anythingllm and a docker run that publishes port 3001 with image mintplexlabs/anythingllm and --rm. The docs quickstart prints docker run with image mintplexlabs/anythingllm:latest and does not print --rm. The repository docker-compose.yml builds from the Dockerfile and publishes port 3001. Main of Mintplex-Labs/anything-llm was commit 4d09fa5f730698d39f1d783e47316744b260a700, committed 2026-11-13. This page did not run the commands.
How do the docs point Ollama or the Generic OpenAI provider at a server on the same machine?
The Ollama LLM page says to connect with http://127.0.0.1:11434 when using the default settings. The Docker localhost page says localhost, 127.0.0.1, and 0.0.0.0 do not work from the container, and that Linux must use 172.17.0.1. The troubleshooting page rewrites the Ollama URL to http://host.docker.internal:11434 on Windows and macOS and to http://172.17.0.1:11434 on Linux. The .env.example line sets OLLAMA_BASE_PATH to http://host.docker.internal:11434 and OLLAMA_MODEL_PREF to llama2. The Generic OpenAI docs say to set Base URL to the API root, for example https://llm.example.com/v1, not the full chat-completions URL. The .env.example model preference for that provider is a model id this page does not copy. The features page starts a session with @agent and exits with exit. The usage page ends a session with /exit. Those two exit strings are not the same. The files do not print an SSH command.
What license sentence does the README print?
The README says this project is MIT licensed. The GitHub license field on 2026-11-14 was MIT. That license text is not a CloudGPU rate.
Sources (read 2026-11-14, Asia/Shanghai)
- Mintplex-Labs/anything-llm README at commit
4d09fa5f730698d39f1d783e47316744b260a700, committed 2026-11-13T22:29:19Z. GitHub license field MIT.pushed_at2026-11-13T22:55:56Z. README sentence: this project is MIT licensed. Docker pointer isdocker/HOW_TO_USE_DOCKER.md docker/HOW_TO_USE_DOCKER.mdat that commit. At least 2GB of RAM. A disk sentence is not copied.docker pull mintplexlabs/anythingllm. Linux run uses--rm, imagemintplexlabs/anythingllm, port 3001, volume at/app/server/storage. NVIDIA run adds--gpus allandENABLE_BUILT_IN_LLM=true. Context cap 16,384 tokens. Health URLhttp://127.0.0.1:13355/api/v1/health. Linux--add-host=host.docker.internal:host-gateway. Compose exampleOLLAMA_BASE_PATH=http://127.0.0.1:11434andOLLAMA_MODEL_PREF=llama2docker/docker-compose.ymlat that commit. Builds from./docker/Dockerfile. Port3001:3001.extra_hostshost.docker.internal:host-gateway. Mounts./.env,../server/storage,../collector/hotdir,../collector/outputsdocker/.env.exampleat that commit.LLM_PROVIDER=ollamaexample usesOLLAMA_BASE_PATHhttp://host.docker.internal:11434,OLLAMA_MODEL_PREFllama2,OLLAMA_MODEL_TOKEN_LIMIT4096.LLM_PROVIDER=generic-openainamesGENERIC_OPEN_AI_BASE_PATH,GENERIC_OPEN_AI_MODEL_PREF, andGENERIC_OPEN_AI_API_KEY. The example base path and model preference are not copiedfrontend/src/components/LLMSelection/OllamaLLMOptions/index.jsxplaceholderhttp://127.0.0.1:11434.GenericOpenAiOptions/index.jsxfield namesGenericOpenAiBasePath,GenericOpenAiKey,GenericOpenAiTokenLimit,GenericOpenAiMaxTokens. The Base URL placeholder host is not copied. Max tokens default in that file is 1024- Mintplex-Labs/anythingllm-docs commit
f931945ab77570b662c2bfb7db2c98ff96563a35, committed 2026-11-13T17:51:44Z. GitHub license field MIT. Quickstart imagemintplexlabs/anythingllm:latest, no--rm, editorhttp://localhost:3001. System requirements RAM 2GB. Storage row not copied. AVX2 checklscpu | grep -o avx2 - Ollama LLM https://docs.anythingllm.com/setup/llm-configuration/local/ollama. URL
http://127.0.0.1:11434 - Localhost https://docs.anythingllm.com/installation-docker/localhost. Linux address
172.17.0.1. Example rewritehttp://localhost:11434tohttp://host.docker.internal:11434 - Ollama troubleshooting https://docs.anythingllm.com/ollama-connection-troubleshooting. Detected URLs
http://127.0.0.1:11434,http://host.docker.internal:11434,http://172.17.0.1:11434 - Generic OpenAI https://docs.anythingllm.com/setup/llm-configuration/cloud/openai-generic. Example base URL
https://llm.example.com/v1. Max Tokens default1024.GENERIC_OPEN_AI_MODEL_TOKEN_LIMIT.GENERIC_OPEN_AI_MAX_TOKENS - AI agents https://docs.anythingllm.com/features/ai-agents.
@agent. Exit stringexit - Agent usage https://docs.anythingllm.com/agent/usage/overview. Exit string
/exit. LogsAgent @agent invokedandAgent session completed - Agent setup https://docs.anythingllm.com/agent/setup.
Settings > Agent Skills. Save. Tools button. A word on that page is not copied - Agent not using tools https://docs.anythingllm.com/agent-not-using-tools. Thought in the UI.
/reset. A closed-model comparison is not copied - Hugging Face model API
unsloth/Llama-3.1-8B-Instruct:gatedfalse,cardData.licensellama3.1,pipeline_tagtext-generation,lastModified2025-05-09T19:32:57.000Z,sha4699cc75b550f9c6f3173fb80f4703b62d946aa5. The card YAML includesextra_gated_prompt. The model information table prints 8B, 70B, 405B, and 128k.base_modelnames a repository this page does not copy.hf downloadis added on this page and was not run. The training-hardware section is not copied - Public GPU memory: https://cloudgpu.app/v1/public/gpu-catalog (
generated_at2026-11-14T11:11:10.599Z).GET /v1/public/compute-capabilitiesreturned feature flags and no GPU list. The catalog had no AnythingLLM template
Cards you can start are on the Deploy page. Amounts are not copied here.
Last updated: 2026-11-14 (Asia/Shanghai).