How to run Letta agents with a self-hosted model on a cloud GPU
Rent a GPU on Deploy, then open a terminal in JupyterLab or over SSH. Install Letta the way the official letta README prints, serve a model on that machine, and point the local backend at the Ollama or OpenAI-compatible URL the current docs print. Copy assistant text and the files under the local state directory before you destroy the instance, because destroying the instance deletes its disk.
An agent can rent the GPU and run this program on it. The agent pages are GPU compute for AI agents and the agent integration guide. Those pages are how an agent discovers a GPU and runs work on it. This page is the Letta install. A Python agent library whose CodeAgent can take TransformersModel or an OpenAI-compatible server is How to run smolagents with a self-hosted model on a cloud GPU. A crew whose LLM class takes base_url for Ollama or an OpenAI-compatible server is How to run CrewAI with a self-hosted model on a cloud GPU. A graph runtime whose overview example does not name a model is How to run LangGraph agents with a self-hosted model on a cloud GPU. An AgentChat library whose Ollama example sets model to llama3.2 is How to run AutoGen agents with a self-hosted model on a cloud GPU. A typed agent whose vLLM example sets base_url to http://localhost:8000/v1 is How to run Pydantic AI agents with a self-hosted model on a cloud GPU. A framework whose README sets Ollama and HuggingFaceEmbedding is How to build LlamaIndex agents with a self-hosted model on a cloud GPU. A browser agent whose models page sets base_url to http://localhost:11434/v1 is How to run browser-use with a self-hosted model on a cloud GPU. A pipeline whose OllamaChatGenerator sets url to http://localhost:11434 is How to run Haystack agents with a self-hosted model on a cloud GPU. A program whose language-models page sets dspy.LM to ollama_chat/llama3.2:1b is How to run DSPy with a self-hosted model on a cloud GPU. An agent whose Ollama example sets id to llama3.1:8b is How to run Agno agents with a self-hosted model on a cloud GPU. A TypeScript agent whose local-models section sets url on the model object is How to run Mastra agents with a self-hosted model on a cloud GPU. A ChatAgent whose vLLM example sets ModelPlatformType.VLLM is How to run CAMEL-AI agents with a self-hosted model on a cloud GPU. A general vLLM install is How to serve an OpenAI-compatible API with vLLM on a cloud GPU. A general SGLang install is How to run SGLang on a cloud GPU. Ollama’s install and port 11434 are How to run Ollama on a cloud GPU. An embedding server is How to serve embeddings and rerankers with Text Embeddings Inference on a cloud GPU. CloudGPU’s hosted gateway is a different base URL: How do I use an OpenAI-compatible API for open models?. How published weight sizes are read: How much VRAM do I need to run an LLM?. The tunnel is How to use SSH port forwarding on a cloud GPU. If the process runs out of memory: How to fix CUDA out of memory on a cloud GPU. If the process raises no kernel image on an RTX 5090: How to fix CUDA no kernel image on an RTX 5090. A Docker Compose app whose self-host page opens http://localhost/install is How to run Dify with a self-hosted model on a cloud GPU. A canvas whose ChatOllama page names a base URL is How to run Flowise with a self-hosted model on a cloud GPU. A flow editor whose Ollama component defaults the base URL to http://127.0.0.1:11434 is How to run Langflow with a self-hosted model on a cloud GPU.
The letta-code LICENSE file begins with “Apache License Version 2.0, January 2004”, its appendix copyright line is “Copyright 2025, Letta authors”, and the GitHub license field on 2026-11-14 was Apache-2.0.
The commands below are copied from the upstream files and docs pages named in each step. They were not run on a CloudGPU instance for this page. Anything this page adds around them is marked. The models page and the connect help name other providers. This page does not list those names.
What the docs print about memory
The letta README, the letta-code README, the models page, the self-hosting page, and src/providers/byok-providers.ts print no GPU GB figure. Those files also print no Hugging Face repository id, so this page does not add hf download. This page does not mark a catalog row as enough.
GET /v1/public/compute-capabilities does not list GPUs. It returns feature flags. The memory column is vram_gb from gpu-prices.json, updated_at 2026-11-14T08:11:19.550Z. GET /v1/public/gpu-catalog the same day (generated_at 2026-11-14T08:11:20.588Z) listed the same names and the same vram_gb values, plus Ascend 910B at 64, which is not in gpu-prices.json and is not listed here. The catalog response also includes an availability field. This page does not copy it. Every prices row has gpu_count 1. Every catalog row has min_gpu_count 1. Which listing you can start is on Deploy.
The catalog template list had no Letta template. It did have Ollama templates (jupyterlab-ollama, ollama-qwen3, ollama). Those templates are a different install. This page does not treat them as Letta.
| Catalog name | vram_gb |
|---|---|
| RTX 2080 | 8 |
| RTX 3080 | 10 |
| RTX 2080 Ti | 11 |
| RTX 3080 Ti | 12 |
| V100 16G | 16 |
| A30 | 24 |
| RTX 3090 | 24 |
| RTX 4090 | 24 |
| RTX 4090D | 24 |
| RTX 4080 SUPER 32G | 32 |
| RTX 5090 | 32 |
| RTX 5090D | 32 |
| V100 32G | 32 |
| A100 40G | 40 |
| A800 40G | 40 |
| A40 | 48 |
| RTX 4090 48G | 48 |
| RTX 4090D 48G | 48 |
| RTX A6000 | 48 |
| NVIDIA RTX PRO 5000 72G | 72 |
| RTX PRO 5000 72G | 72 |
| A100 80G | 80 |
| A800 80G | 80 |
| H800 | 80 |
| RTX 6000D | 83 |
| H20 96G | 96 |
| RTX PRO 6000 96G | 96 |
| RTX PRO 6000 Max-Q | 96 |
| H20-3e 141G | 141 |
Install and server
The letta repository README says the current source lives in letta-ai/letta-code, and that the archive branch is the retired V1 API server. The install and launch lines on that README, copied:
npm install -g @letta-ai/letta-code
letta
letta server
That README does not print a Node version. Main of letta-ai/letta is commit 8816d3a8516893bfdc0fa14bfb1235226012e295, committed 2026-10-18T22:28:01Z. pushed_at on 2026-11-14 was 2026-10-18T22:28:04Z. Tag 0.16.8 is commit 1131535716e8a31c9a437f8695e25ac98f203a24, committed 2026-05-14T16:58:54Z. Those are not the same commit. The GitHub license field for letta-ai/letta on 2026-11-14 was Apache-2.0. That repository LICENSE appendix copyright line is “Copyright 2023, Letta authors”. This page did not diff the tag 0.16.8 README against main.
The letta-code README repeats the npm line and also prints:
uv tool install letta
# or
pipx install letta
That README says the uv and pipx path requires Python 3.9+. npm @letta-ai/letta-code latest on 2026-11-14 was 0.34.61, license Apache-2.0, published 2026-11-14T02:49:41.466Z. Tag v0.34.61 is commit df7045d26f46e6ce49e7bc9bfd3d96d4f574d8dc, committed 2026-11-14T02:42:42Z, and that commit is main. package.json at that commit sets version to 0.34.61, license to Apache-2.0, and engines to node ^22.23.4 || ^24.22.1 || ^26.12.1 and bun >=1.3.2. PyPI letta info.version the same day was 0.34.12, requires_python >=3.9, info.license empty, uploaded 2026-10-12T20:47:35.858621Z. npm 0.34.61 and PyPI 0.34.12 are not the same version. The self-hosting page says the npm install requires Node.js version 22.19+. That sentence and the engines field are not the same string. This page did not install the package.
The self-hosting page server line is not the bare letta server from the README. Copied:
letta server --backend local --listen ws://127.0.0.1:4500
The quickstart says that process prints Listening on ws://127.0.0.1:4500 and WebSocket: ws://127.0.0.1:4500/ws. The same page says letta app-server remains a deprecated alias and new integrations should use letta server --listen. The files this page read do not print an SSH command. For a server you start on the instance, the tunnel is How to use SSH port forwarding on a cloud GPU.
The quickstart also prints letta server --backend local --listen ws://127.0.0.1:4500 --openai-api and says the client base URL is http://127.0.0.1:4500/v1. That flag makes App Server answer Chat Completions. It is not the model-provider URL in the next section.
Agent
The letta-code README says Letta Cloud is the default, and that the first interactive launch asks you to sign in or proceed locally. It also prints:
letta --new-agent --personality tutorial
The self-hosting page says a Tutor agent is auto-created the first time you run the CLI interactively. Its local-backend SDK example, copied:
import { LettaAgentClient } from "@letta-ai/letta-agent-sdk";
const client = new LettaAgentClient({ backend: "local" });
const agentId = await client.createAgent({
persona: "You are Patch, a resident engineering teammate for this repository.",
});
await using session = client.createSession(agentId, {
cwd: process.cwd(),
});
await session.send("Inspect this repository and write an onboarding memo.");
for await (const message of session.stream()) {
if (message.type === "assistant") {
console.log(message.content);
}
}
That page says the local backend starts App Server as a subprocess, and that agent state stays on the machine running the code. The snippet imports @letta-ai/letta-agent-sdk. The self-hosting page does not print an install line for that package next to the snippet. npm @letta-ai/letta-agent-sdk latest on 2026-11-14 was 0.8.90, license Apache-2.0, published 2026-11-13T18:32:23.487Z. This page did not install it and did not read its exports. This page did not run createAgent.
The same page says local state defaults to ~/.letta/lc-local-backend, and that each agent MemFS repository is under ~/.letta/lc-local-backend/memfs/<agent-id>/memory. The angle-bracket token is written that way here so the page renderer does not treat it as a tag. It also prints export LETTA_LOCAL_BACKEND_DIR="$PWD/.letta-local" before letta --backend local --new-agent.
Ollama and an OpenAI-compatible server
The self-hosting page connect lines, copied:
letta --backend local connect ollama
letta --backend local connect lmstudio --base-url http://127.0.0.1:1234/v1
The second of those lines is the one that passes --base-url. The ollama line on that page does not. The models page says local setup can use the built-in Ollama provider, and that an OpenAI-compatible API custom provider takes a base URL and an API key. It says the OpenAI-compatible endpoint must support Chat Completions and tool calling, then /model selects the model. The models page table this page read does not name vLLM. src/providers/byok-providers.ts at tag v0.34.61 does not name vLLM either. Its Ollama entry description is “Connect Ollama at http://localhost:11434/v1 or a remote URL”, and the base URL placeholder is http://localhost:11434/v1. Its openai-compatible entry description is “Connect an OpenAI-compatible Chat Completions endpoint”, with a baseUrl field and no placeholder.
src/cli/subcommands/connect.ts at the same tag prints these examples. The source line for openai-compatible also has an optional --api-key flag whose placeholder is an angle-bracket token. This page omits that token. Copied without it:
letta connect openai-compatible --base-url http://localhost:8000/v1
letta connect ollama --base-url http://192.168.1.50:11434/v1
letta connect lmstudio --base-url http://127.0.0.1:1234/v1 --timeout 600s
letta connect llama-cpp --base-url http://localhost:8080/v1
--timeout 600s is the flag on the lmstudio help line. This page does not treat it as a startup time. Those four lines do not include --backend local. The self-hosting ollama line, the help ollama line, and the placeholder http://localhost:11434/v1 are not the same string. The help openai-compatible URL is http://localhost:8000/v1. That line does not name vLLM. How to serve an OpenAI-compatible API with vLLM on a cloud GPU is the general server install. How to run Ollama on a cloud GPU is the Ollama install. The help function also prints other provider examples. This page does not copy those lines.
When it fails
CUDA out of memory. The Letta files and docs pages this page read print no GPU GB figure. If the process raises CUDA out of memory, use how to fix CUDA out of memory on a cloud GPU. This page does not pick a smaller model id for you.
“CUDA error: no kernel image is available for execution on the device” on an RTX 5090. The letta README, the letta-code README, and package.json do not name compute capability 12.0. The RTX 5090 guide says that card is compute capability 12.0 (sm_120) and that PyTorch 2.7 and later, on a build for CUDA 12.8 or newer, is the line that announced support. This page did not install a torch wheel and did not re-test this program on an RTX 5090. Catalog RTX 5090 and RTX 5090D are 32. That is the catalog number only.
What to copy off, then destroy
The SDK example prints message.content when message.type is assistant. The self-hosting page says self-hosted agents are not automatically backed up. Copy that text and the directory ~/.letta/lc-local-backend before you destroy the instance. Save your work before you destroy a cloud GPU is the checklist. The terms say data on local instance storage is permanently deleted when the instance is destroyed.
The pricing page FAQ states the billing rule in these words:
Per-minute options have a 1-minute minimum; unused prepaid minutes are refunded when you destroy the instance. Some configurations use whole-hour billing with a 1-hour minimum; unused time within a charged hour is not refunded on voluntary stop. The billing period, minimum charge and initial prepayment are shown before confirmation. No subscription.
Prices are not copied here.
Frequently asked questions
What memory figure do the Letta docs print?
The letta README, the letta-code README, the models page, the self-hosting page, and byok-providers.ts print no GPU GB figure. Those files print no Hugging Face repository id. This page does not add hf download. This page does not mark a catalog row as enough. Which listing you can start is on https://cloudgpu.app/deploy.
Which install lines and server command do the docs print?
The letta README prints npm install -g @letta-ai/letta-code, then letta, then letta server. It says current source lives in letta-ai/letta-code. Main of that README is commit 8816d3a8516893bfdc0fa14bfb1235226012e295, committed 2026-10-18. Tag 0.16.8 is an earlier commit. The letta-code README also prints uv tool install letta and pipx install letta, and says that path requires Python 3.9 or newer. npm @letta-ai/letta-code on 2026-11-14 was 0.34.61. PyPI letta info.version the same day was 0.34.12. Those versions are not the same. package.json engines and the self-hosting Node.js 22.19 sentence are not the same string. The self-hosting server line is letta server --backend local --listen ws://127.0.0.1:4500. This page did not install the package.
How do the docs point Letta at a local or OpenAI-compatible server?
The self-hosting page prints letta --backend local connect ollama, and a lmstudio line with --base-url http://127.0.0.1:1234/v1. The connect help prints letta connect openai-compatible --base-url http://localhost:8000/v1, and letta connect ollama --base-url http://192.168.1.50:11434/v1. The provider file placeholder is http://localhost:11434/v1. Those ollama strings are not the same. The models page and byok-providers.ts at tag v0.34.61 do not name vLLM. The help line that uses port 8000 does not name vLLM. The files do not print an SSH command.
What license does the file print?
The letta-code LICENSE file begins with Apache License Version 2.0, January 2004. Its appendix copyright line is Copyright 2025, Letta authors. The GitHub license field on 2026-11-14 was Apache-2.0. npm license for @letta-ai/letta-code 0.34.61 was Apache-2.0. The letta repository LICENSE appendix copyright line is Copyright 2023, Letta authors. That license text is not a CloudGPU price.
Sources (read 2026-11-14, Asia/Shanghai)
- letta-ai/letta README (
npm install -g @letta-ai/letta-code,letta,letta server, pointer to letta-code, archive branch note): commit8816d3a8516893bfdc0fa14bfb1235226012e295, committed 2026-10-18T22:28:01Z. GitHub license fieldApache-2.0.pushed_at2026-10-18T22:28:04Z.LICENSEbegins with Apache License Version 2.0, January 2004. The appendix copyright line is Copyright 2023, Letta authors. Tag0.16.8is commit1131535716e8a31c9a437f8695e25ac98f203a24, committed 2026-05-14T16:58:54Z. This page did not diff that tag README against main - letta-ai/letta-code README and tag
v0.34.61, commitdf7045d26f46e6ce49e7bc9bfd3d96d4f574d8dc, committed 2026-11-14T02:42:42Z. That commit is main.package.jsonversion0.34.61, licenseApache-2.0, engines node^22.23.4 || ^24.22.1 || ^26.12.1and bun>=1.3.2. The README also printsuv tool install lettaandpipx install letta, Python 3.9 or newer, andletta --new-agent --personality tutorial. npm@letta-ai/letta-code0.34.61, published 2026-11-14T02:49:41.466Z, license Apache-2.0. PyPIletta0.34.12, uploaded 2026-10-12T20:47:35.858621Z,requires_python>=3.9,info.licenseempty. Those versions are not the same.LICENSEbegins with Apache License Version 2.0, January 2004. The appendix copyright line is Copyright 2025, Letta authors. The file also has a Brand Assets Exclusion paragraph. This page does not quote it - Self-hosting page https://docs.letta.com/self-hosting and App Server quickstart https://docs.letta.com/self-hosting/app-server/quickstart, fetched 2026-11-14. Node.js 22.19 or newer on the install step.
letta server --backend local --listen ws://127.0.0.1:4500.letta --backend local connect ollama. lmstudio--base-url http://127.0.0.1:1234/v1. Local SDKLettaAgentClientwithbackend: "local"andcreateAgent. State directory~/.letta/lc-local-backend. The deployment section names other hosts and a closed-vendor key. This page does not copy that section.--openai-apiis the App Server HTTP API on port 4500, not the model-provider URL - Models page https://docs.letta.com/configuration/models, fetched 2026-11-14. Ollama is a local endpoint. OpenAI-compatible API is an API key plus a base URL, and the endpoint must support Chat Completions and tool calling. The table this page read does not name vLLM. The page names other providers. This page does not list them
src/providers/byok-providers.tsandsrc/cli/subcommands/connect.tsatdf7045d26f46e6ce49e7bc9bfd3d96d4f574d8dc. Ollama placeholderhttp://localhost:11434/v1. Help examples foropenai-compatible,ollama,lmstudio, andllama-cppas copied above.--timeout 600sis the flag on the lmstudio help line, not a startup time. The optional api-key placeholder is omitted here. Neither file names vLLM- npm
@letta-ai/letta-agent-sdk0.8.90, published 2026-11-13T18:32:23.487Z, license Apache-2.0. The self-hosting snippet imports it. This page did not install it - Public GPU memory: https://cloudgpu.app/gpu-prices.json (
updated_at2026-11-14T08:11:19.550Z) and https://cloudgpu.app/v1/public/gpu-catalog (generated_at2026-11-14T08:11:20.588Z).GET /v1/public/compute-capabilitiesreturned feature flags and no GPU list. The catalog had no Letta template
Cards you can start are on the Deploy page. Prices are on https://cloudgpu.app/pricing and are not repeated here.
Last updated: 2026-11-14 (Asia/Shanghai).