How to serve LLMs with LMDeploy on a cloud GPU
Rent a GPU on Deploy, then open a terminal in JupyterLab or over SSH. Install LMDeploy from the official InternLM/lmdeploy repository the way the README and the installation page print, and start lmdeploy serve api_server on the port those pages print. Copy the model directory and anything you saved before you destroy the instance, because destroying the instance deletes its disk and stops the server.
A general vLLM install is How to serve an OpenAI-compatible API with vLLM on a cloud GPU. A general SGLang install is How to run SGLang on a cloud GPU. An ExLlama server whose sample config listens on 127.0.0.1 port 5000 is How to serve EXL2/EXL3 models with TabbyAPI on a cloud GPU. A llama.cpp CUDA build is How to build llama.cpp with CUDA on a cloud GPU. CloudGPU’s hosted gateway is a different base URL: How do I use an OpenAI-compatible API for open models?. How published weight sizes are read: How much VRAM do I need to run an LLM?. If the process runs out of memory: How to fix CUDA out of memory on a cloud GPU. If the server raises no kernel image on an RTX 5090: How to fix CUDA no kernel image on an RTX 5090. The tunnel is How to use SSH port forwarding on a cloud GPU. The download helper this site prints is How to download a Hugging Face model. An editor whose OpenAI Compatible page sets a Base URL for a server on the instance is How to use Roo Code with a self-hosted model on a cloud GPU. A Python agent library whose OpenAIModel takes api_base for an OpenAI-compatible server is How to run smolagents with a self-hosted model on a cloud GPU.
The LMDeploy LICENSE file begins with “Apache License Version 2.0, January 2004”. The appendix copyright line is “Copyright 2023-2024 Shanghai AI Laboratory. All rights reserved.” The GitHub license field on 2026-11-14 was Apache-2.0. That license text is not a CloudGPU price. The copyright line names the lab. This page does not treat it as a machine location.
The commands below are copied from the upstream files named in each step. They were not run on a CloudGPU instance for this page. Anything this page adds around them is marked.
What the docs list
The README at commit cf24f197cdb80afd9900bc6977c28a4b1458f472 (committed 2026-11-13T10:46:14Z) says LMDeploy has two engines, TurboMind and PyTorch. TurboMind is C++ and CUDA. PyTorch is Python. The installation page says the same, and says TurboMind takes precedence when pipeline is not given an engine.
The supported-models page’s TurboMind CUDA table columns are Model, Size, Type, FP16/BF16, KV INT8, KV INT4, and W4A16. The quick start’s example id internlm/internlm2_5-7b-chat matches the InternLM2.5 row: Size 7B, Type LLM, and Yes in the four precision columns. The VLM example OpenGVLab/InternVL2-8B matches the InternVL2 row, whose Size cell is “1 - 2B, 8B - 76B”. The same table includes MiniCPM-Llama3-V-2_5 and MiniCPM-V-2_6, both with Size “-” and Type MLLM. It does not include a MiniCPM-V 4.7 row. This page does not copy every row. “-” on that page means not verified yet.
The PyTorch CUDA table adds a W8A8 column. Its MiniCPM-V-2_6 row has Size 8B and Type LLM. Its MiniCPM3 row has Size 4B. This page does not copy every PyTorch row, and it does not treat the 2.6 row as MiniCPM-V 4.7. A current MiniCPM-V install is How to run MiniCPM-V on a cloud GPU.
Note [2] on the TurboMind table says that when head_dim is not 128, TurboMind does not support that model’s KV cache 4/8 bit quantization. Note [3] says TurboMind does not currently support the vision encoder for the Qwen3.5 series.
The AWQ page says TurboMind can run 4-bit models quantized by AWQ or GPTQ, and that LMDeploy’s own quantization module only implements AWQ. The command it prints is lmdeploy lite auto_awq. The serve line for a local AWQ folder is:
lmdeploy serve api_server ./internlm2_5-7b-chat-4bit --backend turbomind --model-format awq
The KV cache page says that since v0.4.0, online KV quantization is int4 or int8, asymmetric, per head and per token. It says quant_policy=4 is 4-bit KV, quant_policy=8 is 8-bit KV, and quant_policy=42 is TurboQuant. The serve example is:
lmdeploy serve api_server internlm/internlm2_5-7b-chat --quant-policy 8
The same page says TurboQuant (quant_policy=42) is PyTorchEngine only. This page does not copy that page’s throughput table.
Choosing a GPU
The installation page, the README install section, and the quick start print no GPU GB figure. The quick start says cache_max_entry_count is the proportion of free GPU memory occupied by the K/V cache after the weights load, and that the default is 0.8. That 0.8 is a ratio. This page does not turn it into a vram_gb value and does not mark a catalog row as enough.
The installation page’s NVIDIA list is Volta (sm70) V100, Turing (sm75) 20 series and T4, Ampere (sm80, sm86) 30 series, A10, A16, A30, and A100, and Ada Lovelace (sm89) 40 series. The AWQ page prints the same four lines. The KV cache page prints those four and also names a Hopper (sm90) row. This page does not print the card names on that Hopper row. The KV page’s summary sentence says it supports NVIDIA GPUs with Volta (sm70) and above. The installation list does not name the 50 series or compute capability 12.0.
The README install paragraph says that starting from v0.13.0, the default prebuilt wheels on PyPI are built against CUDA 12.8, and that pip install lmdeploy covers typical setups including GeForce RTX 50 series. A 2026/04 news line on the same README says v0.12.3 is available on PyPI. PyPI’s info.version for lmdeploy on 2026-11-14 was 0.18.0. This page did not install the wheel and did not read which CUDA tag 0.18.0 was built with. It does not treat those three version sentences as the same release.
GET /v1/public/compute-capabilities does not list GPUs. It returns feature flags. The memory column is vram_gb from gpu-prices.json, updated_at 2026-11-14T02:04:03.491Z. GET /v1/public/gpu-catalog the same day (generated_at 2026-11-14T02:04:03.382Z) listed the same names and the same vram_gb values, plus Ascend 910B at 64, which is not in gpu-prices.json and is not listed here. The catalog response also includes an availability field. This page does not copy it. Every prices row has gpu_count 1. Every catalog row has min_gpu_count 1. Which listing you can start is on Deploy.
The catalog template list had no LMDeploy template.
| Catalog name | vram_gb |
|---|---|
| RTX 2080 | 8 |
| RTX 3080 | 10 |
| RTX 2080 Ti | 11 |
| RTX 3080 Ti | 12 |
| V100 16G | 16 |
| A30 | 24 |
| RTX 3090 | 24 |
| RTX 4090 | 24 |
| RTX 4090D | 24 |
| RTX 4080 SUPER 32G | 32 |
| RTX 5090 | 32 |
| RTX 5090D | 32 |
| V100 32G | 32 |
| A100 40G | 40 |
| A800 40G | 40 |
| A40 | 48 |
| RTX 4090 48G | 48 |
| RTX 4090D 48G | 48 |
| RTX A6000 | 48 |
| NVIDIA RTX PRO 5000 72G | 72 |
| RTX PRO 5000 72G | 72 |
| A100 80G | 80 |
| A800 80G | 80 |
| H800 | 80 |
| RTX 6000D | 83 |
| H20 96G | 96 |
| RTX PRO 6000 96G | 96 |
| RTX PRO 6000 Max-Q | 96 |
| H20-3e 141G | 141 |
Install
On Deploy, choose Raw GPU. The README and docs/en/get_started/installation.md recommend a conda environment and Python 3.10 through 3.13. The example they print uses 3.12:
conda create -n lmdeploy python=3.12 -y
conda activate lmdeploy
pip install lmdeploy
The source install on the same page requires C++20, CMake 3.25.2 or newer, and CUDA Toolkit 12.0 or newer, then:
pip install git+https://github.com/InternLM/lmdeploy.git
DISABLE_TURBOMIND=1 in front of that command skips the TurboMind build. A pinned source URL on that page is https://github.com/InternLM/lmdeploy/archive/refs/tags/v0.11.0.zip. This page does not treat v0.11.0 as the PyPI version above. A conda environment, if you want one around those lines, is How to set up a conda environment on a cloud GPU. This page did not run pip install lmdeploy.
The installation page also prints a ROCm docker run line with --shm-size 32G. That 32G figure is the shared-memory flag in that command, not a vram_gb value and not a disk size. This page does not start that container.
Download a model
The quick start’s pipeline and serve examples pass internlm/internlm2_5-7b-chat. The README says LMDeploy downloads from Hugging Face by default and does not print hf download. The block below is added on this page. How to download a Hugging Face model prints pip install -U "huggingface_hub" and hf download. This page did not download the repository.
# Added on this page. The quick start passes the repo id to pipeline and api_server.
pip install -U "huggingface_hub"
hf download internlm/internlm2_5-7b-chat
The VLM example id is OpenGVLab/InternVL2-8B. The same added command with that id is not a different install. This page did not download it either.
Serve the OpenAI-compatible API
The quick start’s LLM serve command, copied:
lmdeploy serve api_server internlm/internlm2_5-7b-chat
It says this launches an OpenAI-compatible server on localhost at port 23333, and that --server-port changes the port. The client snippet sets base_url to http://0.0.0.0:23333/v1 and calls v1/chat/completions. The VLM command is the same binary with OpenGVLab/InternVL2-8B. The VLM note says an ImportError means the upstream VLM extra dependencies are not in LMDeploy’s own list. This page does not add those packages.
docs/en/llm/api_server.md lists three compatible routes:
/v1/chat/completions/v1/models/v1/completions
It says the site http://0.0.0.0:23333 is the Swagger UI after the server starts. It says that if neither the request nor the server sets a temperature, LMDeploy uses temperature=1.0. The CLI example on that page passes --server-port 23333 and does not pass --backend. A later Python helper defaults backend to turbomind and builds a command that also passes --server-name. This page does not run that helper.
The docker example publishes 23333:23333 and runs the same lmdeploy serve api_server internlm/internlm2_5-7b-chat inside openmmlab/lmdeploy:latest. This page does not start that container. How to use SSH port forwarding on a cloud GPU is how a program on your computer reaches port 23333. This page does not add an ssh command.
When it fails
CUDA out of memory. The quick start says an OOM can be addressed by lowering cache_max_entry_count below the default 0.8. The AWQ page says a quantization OOM can be addressed by reducing --calib-seqlen, increasing --calib-samples, and setting --batch-size to 1. If the process raises CUDA out of memory, use how to fix CUDA out of memory on a cloud GPU.
“CUDA error: no kernel image is available for execution on the device” on an RTX 5090. The installation page’s GPU list stops at Ada Lovelace (sm89) and does not name compute capability 12.0. The README says v0.13.0 wheels are CUDA 12.8 and include GeForce RTX 50 series. The RTX 5090 guide says that card is compute capability 12.0 (sm_120) and that PyTorch 2.7 and later, on a build for CUDA 12.8 or newer, is the line that announced support. This page did not install a torch wheel and did not re-test this server on an RTX 5090. Catalog RTX 5090 and RTX 5090D are 32. That is the catalog number only.
What to copy off, then destroy
The quick start does not name an output directory. lmdeploy lite auto_awq writes the directory you pass as --work-dir. The example work dir is internlm2_5-7b-chat-4bit. hf download without --local-dir writes the Hugging Face cache. Copy that directory and any replies you saved before you destroy the instance. Save your work before you destroy a cloud GPU is the checklist. The terms say data on local instance storage is permanently deleted when the instance is destroyed. The server stops with the instance.
The pricing page FAQ states the billing rule in these words:
Per-minute options have a 1-minute minimum; unused prepaid minutes are refunded when you destroy the instance. Some configurations use whole-hour billing with a 1-hour minimum; unused time within a charged hour is not refunded on voluntary stop. The billing period, minimum charge and initial prepayment are shown before confirmation. No subscription.
Prices are not copied here.
Frequently asked questions
Which engines and quant formats do the docs list?
The README and the installation page name TurboMind and PyTorch. The TurboMind CUDA table columns are FP16/BF16, KV INT8, KV INT4, and W4A16. The AWQ page says TurboMind runs 4-bit AWQ and GPTQ weights, and that the LMDeploy quantizer implements AWQ only. The KV page says quant_policy 4 is 4-bit KV, quant_policy 8 is 8-bit KV, and quant_policy 42 is TurboQuant on PyTorchEngine only. The TurboMind table lists MiniCPM-V-2_6 and does not list MiniCPM-V 4.7. This page does not copy every supported-model row.
What GPU memory figure do those files print?
The installation page, the README install section, and the quick start print no GPU GB figure. The quick start says cache_max_entry_count defaults to 0.8, which is the proportion of free GPU memory used for the K/V cache after the weights load. The installation GPU list is Volta, Turing, Ampere, and Ada Lovelace. The README says v0.13.0 PyPI wheels are CUDA 12.8 and include GeForce RTX 50 series. PyPI info.version on 2026-11-14 was 0.18.0. This page does not mark a catalog row as enough. Which listing you can start is on https://cloudgpu.app/deploy.
Which host, port, and routes does api_server document?
The quick start says the server listens on localhost port 23333. The client snippet uses http://0.0.0.0:23333/v1. api_server.md lists /v1/chat/completions, /v1/models, and /v1/completions, and says http://0.0.0.0:23333 is the Swagger UI. It says the temperature fallback is 1.0 when neither the request nor the server sets one. The SSH port-forwarding guide is how a program on your computer reaches port 23333. This page did not open a tunnel.
What license does the file print?
The LMDeploy LICENSE file begins with Apache License Version 2.0, January 2004. The appendix copyright line is Copyright 2023-2024 Shanghai AI Laboratory. The GitHub license field on 2026-11-14 was Apache-2.0. That license text is not a CloudGPU price. This page does not treat the copyright line as a machine location.
Sources (read 2026-11-14, Asia/Shanghai)
- InternLM/lmdeploy README (
pip install lmdeploy, Python 3.10–3.13, v0.12.3 news line, v0.13.0 CUDA 12.8 and GeForce RTX 50 series, TurboMind and PyTorch): commitcf24f197cdb80afd9900bc6977c28a4b1458f472, committed 2026-11-13T10:46:14Z. GitHub license fieldApache-2.0.LICENSEappendix: Copyright 2023-2024 Shanghai AI Laboratory docs/en/get_started/installation.md(conda example, GPU list through Ada Lovelace sm89, C++20, CMake 3.25.2, CUDA Toolkit 12.0, source URL tag v0.11.0, ROCm--shm-size 32G): same commitdocs/en/get_started/get_started.md(pipeline,cache_max_entry_countdefault 0.8,lmdeploy serve api_server, port 23333,http://0.0.0.0:23333/v1, examplesinternlm/internlm2_5-7b-chatandOpenGVLab/InternVL2-8B): same commitdocs/en/llm/api_server.md(/v1/chat/completions,/v1/models,/v1/completions, Swaggerhttp://0.0.0.0:23333,temperature=1.0, docker-p 23333:23333): same commitdocs/en/supported_models/supported_models.md(TurboMind and PyTorch tables, MiniCPM-V-2_6, no MiniCPM-V 4.7 row): same commitdocs/en/quantization/w4a16.md(AWQ and GPTQ inference, AWQ-only quantizer,lmdeploy lite auto_awq,--model-format awq): same commitdocs/en/quantization/kv_quant.md(quant_policy4, 8, and 42): same commit. The Hopper row’s card names are not copied on this page- PyPI
lmdeployinfo.version0.18.0, read 2026-11-14 from https://pypi.org/pypi/lmdeploy/json. This page did not install that wheel - Public GPU memory: https://cloudgpu.app/gpu-prices.json (
updated_at2026-11-14T02:04:03.491Z) and https://cloudgpu.app/v1/public/gpu-catalog (generated_at2026-11-14T02:04:03.382Z).GET /v1/public/compute-capabilitiesreturned feature flags and no GPU list. The catalog had no LMDeploy template
Cards you can start are on the Deploy page. Prices are on https://cloudgpu.app/pricing and are not repeated here.
Last updated: 2026-11-14 (Asia/Shanghai).