How to build LlamaIndex agents with a self-hosted model on a cloud GPU
Rent a GPU on Deploy, then open a terminal in JupyterLab or over SSH. Install the core package, the Ollama integration, and the local embedding integration the official README prints, serve a model on that machine, and build the index that page shows. Copy the query result and the storage directory before you destroy the instance, because destroying the instance deletes its disk.
An agent can rent the GPU and run this library on it. The agent pages are GPU compute for AI agents and the agent integration guide. Those pages are how an agent discovers a GPU and runs work on it. This page is the LlamaIndex install. A Python agent library whose CodeAgent can take TransformersModel or an OpenAI-compatible server is How to run smolagents with a self-hosted model on a cloud GPU. An AgentChat library whose Ollama example sets model to llama3.2 is How to run AutoGen agents with a self-hosted model on a cloud GPU. A typed agent whose vLLM example sets base_url to http://localhost:8000/v1 is How to run Pydantic AI agents with a self-hosted model on a cloud GPU. A local agent UI whose Docker command publishes port 3000 is How to run OpenHands with a self-hosted model on a cloud GPU. An embeddings server is How to serve embeddings and rerankers with Text Embeddings Inference on a cloud GPU. A general vLLM install is How to serve an OpenAI-compatible API with vLLM on a cloud GPU. A general SGLang install is How to run SGLang on a cloud GPU. Ollama’s install and port 11434 are How to run Ollama on a cloud GPU. A TurboMind server whose lmdeploy serve api_server listens on port 23333 is How to serve LLMs with LMDeploy on a cloud GPU. CloudGPU’s hosted gateway is a different base URL: How do I use an OpenAI-compatible API for open models?. How published weight sizes are read: How much VRAM do I need to run an LLM?. The tunnel is How to use SSH port forwarding on a cloud GPU. If the process runs out of memory: How to fix CUDA out of memory on a cloud GPU. If the process raises no kernel image on an RTX 5090: How to fix CUDA no kernel image on an RTX 5090. A crew whose LLM class takes base_url for Ollama or an OpenAI-compatible server is How to run CrewAI with a self-hosted model on a cloud GPU. A graph runtime whose overview example does not name a model is How to run LangGraph agents with a self-hosted model on a cloud GPU. A browser agent whose models page sets base_url to http://localhost:11434/v1 is How to run browser-use with a self-hosted model on a cloud GPU. A pipeline whose Ollama page sets url to http://localhost:11434, and whose vLLM page defaults api_base_url to http://localhost:8000/v1, is How to run Haystack agents with a self-hosted model on a cloud GPU. A program whose language-models page sets dspy.LM to ollama_chat/llama3.2:1b, and whose GPU-server example sets api_base to http://localhost:7501/v1, is How to run DSPy with a self-hosted model on a cloud GPU. An agent whose Ollama example sets id to llama3.1:8b, and whose VLLM class defaults base_url to http://localhost:8000/v1/, is How to run Agno agents with a self-hosted model on a cloud GPU.
The LICENSE file begins with “The MIT License”, its copyright notice names Jerry Liu, and the GitHub license field on 2026-11-14 was MIT.
The commands below are copied from the upstream files named in each step. They were not run on a CloudGPU instance for this page. Anything this page adds around them is marked. The README also names other products. This page does not copy that note.
What the docs print about memory
The README, the Ollama integration README, and the OpenAI-like integration README print no GPU GB figure. The BAAI/bge-small-en-v1.5 card prints no GPU GB figure. The Llama-3.1-8B-Instruct README fetch returned HTTP 401, so this page does not copy a memory line from that card. This page does not mark a catalog row as enough.
GET /v1/public/compute-capabilities does not list GPUs. It returns feature flags. The memory column is vram_gb from gpu-prices.json, updated_at 2026-11-14T06:12:25.272Z. GET /v1/public/gpu-catalog the same day (generated_at 2026-11-14T06:12:26.381Z) listed the same names and the same vram_gb values, plus Ascend 910B at 64, which is not in gpu-prices.json and is not listed here. The catalog response also includes an availability field. This page does not copy it. Every prices row has gpu_count 1. Every catalog row has min_gpu_count 1. Which listing you can start is on Deploy.
The catalog template list had no LlamaIndex template. It did have Ollama templates (jupyterlab-ollama, ollama-qwen3, ollama). Those templates are a different install. This page does not treat them as LlamaIndex.
| Catalog name | vram_gb |
|---|---|
| RTX 2080 | 8 |
| RTX 3080 | 10 |
| RTX 2080 Ti | 11 |
| RTX 3080 Ti | 12 |
| V100 16G | 16 |
| A30 | 24 |
| RTX 3090 | 24 |
| RTX 4090 | 24 |
| RTX 4090D | 24 |
| RTX 4080 SUPER 32G | 32 |
| RTX 5090 | 32 |
| RTX 5090D | 32 |
| V100 32G | 32 |
| A100 40G | 40 |
| A800 40G | 40 |
| A40 | 48 |
| RTX 4090 48G | 48 |
| RTX 4090D 48G | 48 |
| RTX A6000 | 48 |
| NVIDIA RTX PRO 5000 72G | 72 |
| RTX PRO 5000 72G | 72 |
| A100 80G | 80 |
| A800 80G | 80 |
| H800 | 80 |
| RTX 6000D | 83 |
| H20 96G | 96 |
| RTX PRO 6000 96G | 96 |
| RTX PRO 6000 Max-Q | 96 |
| H20-3e 141G | 141 |
Install
The README says a customized install is llama-index-core plus the integration packages the app needs. The lines this page keeps, copied. The README block also prints a fourth package line for a hosted client. This page does not copy that line:
pip install llama-index-core
pip install llama-index-llms-ollama
pip install llama-index-embeddings-huggingface
The OpenAI-like integration README prints a separate install:
pip install llama-index-llms-openai-like
llama-index-core/pyproject.toml at both commits below says version = "0.14.26" and requires-python = ">=3.11,<4.0". The root pyproject.toml says the same version and the same Python range for llama-index. PyPI info.version for llama-index on 2026-11-14 was 0.14.26, requires_python <4.0,>=3.11, uploaded 2026-11-13T22:38:11.236664Z. llama-index-core was 0.14.26, uploaded 2026-11-13T22:37:36.634460Z. Tag v0.14.26 is commit 3aaab93a2e823b9637ce3c6f0611e5e18a4a7a5b, committed 2026-11-13T22:36:32Z. Main is commit 99d7433f6f71888951379ccca9294e272442c832, committed 2026-11-14T02:10:23Z. Those are not the same commit. The README, LICENSE, and the three integration READMEs quoted here are the same blob on both. The integration pyproject.toml version lines are not. On the tag they say llama-index-llms-ollama 0.11.0, llama-index-llms-openai-like 0.8.1, and llama-index-embeddings-huggingface 0.8.0. On main they say 0.12.0, 0.9.0, and 0.9.0. PyPI on 2026-11-14 reported 0.12.0 (uploaded 2026-11-14T02:18:27.247256Z), 0.9.0 (uploaded 2026-11-14T02:18:34.674992Z), and 0.9.0 (uploaded 2026-11-14T02:07:07.520071Z). The unpinned pip install lines install the PyPI releases. This page did not install them.
Index, query, and a local embedding
The README example for a non-hosted model, copied. It reads a directory, builds VectorStoreIndex, and sets Settings.llm and Settings.embed_model:
from llama_index.core import Settings, VectorStoreIndex, SimpleDirectoryReader
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.llms.ollama import Ollama
from transformers import AutoTokenizer
Settings.llm = Ollama(
model="llama-3.1:latest",
request_timeout=360.0,
)
Settings.tokenizer = AutoTokenizer.from_pretrained(
"meta-llama/Llama-3.1-8B-Instruct"
)
Settings.embed_model = HuggingFaceEmbedding(
model_name="BAAI/bge-small-en-v1.5"
)
documents = SimpleDirectoryReader("YOUR_DATA_DIRECTORY").load_data()
index = VectorStoreIndex.from_documents(
documents,
)
The query and the disk lines, copied:
query_engine = index.as_query_engine()
query_engine.query("YOUR_QUESTION")
index.storage_context.persist()
The README says persist writes under ./storage. This page did not run the query.
The Ollama integration README says the app serves models on localhost:11434. Its constructor uses a different tag from the root README. Copied:
from llama_index.llms.ollama import Ollama
llm = Ollama(model="llama3.1:latest", request_timeout=120.0)
llama-3.1:latest and llama3.1:latest are not the same string. The setup list also prints model=":" and this line:
model="<model family>"
The setup step says a name without a version looks up the latest version, and that the default timeout is 30 seconds, with an example of request_timeout=300.0. The usage example uses 120.0. The root README uses 360.0. Those numbers are not the same. The integration README’s structured example defines Song with name and artist and calls llm.as_structured_llm(Song). This page did not run it.
The embedding integration README at this commit is only the title line. It does not print an install command or a model id. The model id is the one in the root README.
On 2026-11-14, GET https://huggingface.co/api/models/BAAI/bge-small-en-v1.5 reported gated false, cardData.license mit, pipeline_tag feature-extraction, lastModified 2024-02-22T03:36:23.000Z, and sha 5c38ec7c405ec4b44b94cc5a9bb96e735b38267a. The card prints no GPU GB figure. GET https://huggingface.co/api/models/meta-llama/Llama-3.1-8B-Instruct reported gated manual, cardData.license llama3.1, pipeline_tag text-generation, lastModified 2024-09-25T17:00:57.000Z, and sha 0e9e39f249a16976918f6564b8830bc894c89659. A fetch of that card README returned HTTP 401. The block below is added on this page. The README loads these ids itself and does not print hf download. This page did not download either repository.
# Added on this page. The README loads these ids itself.
pip install -U "huggingface_hub"
hf download BAAI/bge-small-en-v1.5
hf download meta-llama/Llama-3.1-8B-Instruct
The second id is gated. This page does not claim the download completes.
OpenAI-compatible base URL
The OpenAI-like README says the package can be used with any OpenAI-compatible API. The example, copied. model is the placeholder model-name. context_window=128000 is the value in that example. The comment says to set it to match the model. This page does not treat 128000 as vram_gb:
from llama_index.llms.openai_like import OpenAILike
llm = OpenAILike(
model="model-name",
api_base="http://localhost:1234/v1",
api_key="fake",
context_window=128000,
is_chat_model=True,
is_function_calling_model=False,
)
That README does not print a vLLM port. How to serve an OpenAI-compatible API with vLLM on a cloud GPU is the server install. How to run Ollama on a cloud GPU is the Ollama install. The Ollama integration README prints localhost:11434. The OpenAI-like example prints port 1234.
For a server you start on the instance, the SSH tunnel is How to use SSH port forwarding on a cloud GPU.
When it fails
CUDA out of memory. The LlamaIndex files this page read print no GPU GB figure. If the process raises CUDA out of memory, use how to fix CUDA out of memory on a cloud GPU. This page does not pick a smaller model id for you.
“CUDA error: no kernel image is available for execution on the device” on an RTX 5090. pyproject.toml does not name compute capability 12.0. The RTX 5090 guide says that card is compute capability 12.0 (sm_120) and that PyTorch 2.7 and later, on a build for CUDA 12.8 or newer, is the line that announced support. This page did not install a torch wheel and did not re-test this library on an RTX 5090. Catalog RTX 5090 and RTX 5090D are 32. That is the catalog number only.
What to copy off, then destroy
query_engine.query returns in the process that called it. index.storage_context.persist() writes ./storage. Copy the printed result and that directory before you destroy the instance. Save your work before you destroy a cloud GPU is the checklist. The terms say data on local instance storage is permanently deleted when the instance is destroyed.
The pricing page FAQ states the billing rule in these words:
Per-minute options have a 1-minute minimum; unused prepaid minutes are refunded when you destroy the instance. Some configurations use whole-hour billing with a 1-hour minimum; unused time within a charged hour is not refunded on voluntary stop. The billing period, minimum charge and initial prepayment are shown before confirmation. No subscription.
Prices are not copied here.
Frequently asked questions
What memory figure do the LlamaIndex docs print?
The README, the Ollama integration README, and the OpenAI-like integration README print no GPU GB figure. The BAAI/bge-small-en-v1.5 card prints no GPU GB figure. The Llama-3.1-8B-Instruct card README fetch returned HTTP 401, so this page does not copy a memory line from that card. This page does not mark a catalog row as enough. Which listing you can start is on https://cloudgpu.app/deploy.
Which install lines and classes do the docs print?
The README prints pip install llama-index-core, pip install llama-index-llms-ollama, and pip install llama-index-embeddings-huggingface. It also prints a fourth package line for a hosted client. This page does not copy that line. The OpenAI-like README prints pip install llama-index-llms-openai-like. llama-index-core pyproject.toml says version 0.14.26 and requires Python 3.11 or newer and below 4.0. PyPI llama-index on 2026-11-14 was 0.14.26. Tag v0.14.26 and main are not the same commit. The integration READMEs match. The integration pyproject versions do not. PyPI reported llama-index-llms-ollama 0.12.0, llama-index-llms-openai-like 0.9.0, and llama-index-embeddings-huggingface 0.9.0. The tag file says 0.11.0, 0.8.1, and 0.8.0. This page did not install them.
How do the docs point the index at a local or OpenAI-compatible server?
The root README sets Ollama model to llama-3.1:latest and request_timeout to 360.0, the tokenizer to meta-llama/Llama-3.1-8B-Instruct, and HuggingFaceEmbedding to BAAI/bge-small-en-v1.5. The Ollama integration README says the server is on localhost:11434 and sets model to llama3.1:latest with request_timeout 120.0. Those model strings are not the same. The OpenAI-like example sets api_base to http://localhost:1234/v1, model to the placeholder model-name, and context_window to 128000. That value is not treated as vram_gb. The files do not print a vLLM port. The model API reported the Llama-3.1-8B-Instruct repo as gated manual.
What license does the file print?
The LICENSE file begins with The MIT License. Its copyright notice names Jerry Liu. The GitHub license field on 2026-11-14 was MIT. That license text is not a CloudGPU price.
Sources (read 2026-11-14, Asia/Shanghai)
- run-llama/llama_index README (
pip install llama-index-core,llama-index-llms-ollama,llama-index-embeddings-huggingface,Ollamamodelllama-3.1:latest,request_timeout=360.0,meta-llama/Llama-3.1-8B-Instruct,BAAI/bge-small-en-v1.5,VectorStoreIndex,persist): commit99d7433f6f71888951379ccca9294e272442c832, committed 2026-11-14T02:10:23Z. GitHub license fieldMIT.pushed_at2026-11-14T02:26:05Z.LICENSEcopyright notice names Jerry Liu. A fourth install line and a product note on the README are not copied - Tag
v0.14.26, commit3aaab93a2e823b9637ce3c6f0611e5e18a4a7a5b, committed 2026-11-13T22:36:32Z. README,LICENSE, and the three integration READMEs match main. Integrationpyproject.tomlversions on the tag are 0.11.0, 0.8.1, and 0.8.0. Main says 0.12.0, 0.9.0, and 0.9.0. PyPIllama-index0.14.26, uploaded 2026-11-13T22:38:11.236664Z,requires_python<4.0,>=3.11. PyPIllama-index-core0.14.26, uploaded 2026-11-13T22:37:36.634460Z. PyPIllama-index-llms-ollama0.12.0, uploaded 2026-11-14T02:18:27.247256Z. PyPIllama-index-llms-openai-like0.9.0, uploaded 2026-11-14T02:18:34.674992Z. PyPIllama-index-embeddings-huggingface0.9.0, uploaded 2026-11-14T02:07:07.520071Z llama-index-integrations/llms/llama-index-llms-ollama/README.md(localhost:11434,llama3.1:latest,request_timeout=120.0,model=":",Song,as_structured_llm): same blob on both commitsllama-index-integrations/llms/llama-index-llms-openai-like/README.md(OpenAILike,api_base="http://localhost:1234/v1",model="model-name",context_window=128000): same blob on both commits. 128000 is not treated asvram_gbllama-index-integrations/embeddings/llama-index-embeddings-huggingface/README.md: title line only, same blob on both commits- Hugging Face model API
BAAI/bge-small-en-v1.5:gatedfalse,cardData.licensemit,pipeline_tagfeature-extraction,lastModified2024-02-22T03:36:23.000Z,sha5c38ec7c405ec4b44b94cc5a9bb96e735b38267a. The card prints no GPU GB figure - Hugging Face model API
meta-llama/Llama-3.1-8B-Instruct:gatedmanual,cardData.licensellama3.1,pipeline_tagtext-generation,lastModified2024-09-25T17:00:57.000Z,sha0e9e39f249a16976918f6564b8830bc894c89659. Card README fetch returned HTTP 401.hf downloadis added on this page and was not run - Public GPU memory: https://cloudgpu.app/gpu-prices.json (
updated_at2026-11-14T06:12:25.272Z) and https://cloudgpu.app/v1/public/gpu-catalog (generated_at2026-11-14T06:12:26.381Z).GET /v1/public/compute-capabilitiesreturned feature flags and no GPU list. The catalog had no LlamaIndex template
Cards you can start are on the Deploy page. Prices are on https://cloudgpu.app/pricing and are not repeated here.
Last updated: 2026-11-14 (Asia/Shanghai).