How to serve embeddings and rerankers with Text Embeddings Inference on a cloud GPU

Rent a GPU on Deploy, then open a terminal in JupyterLab or over SSH. Install Text Embeddings Inference with the Docker command in the official README, download the open embedding model that command names, and call the /embed, /rerank, and OpenAI-compatible /v1/embeddings routes that the README and its OpenAPI file document. Copy the mounted volume off the instance before you destroy it, because destroying the instance deletes its disk and stops the server.

An agent can rent the GPU and serve embeddings and rerankers for retrieval or memory. The agent pages are GPU compute for AI agents and the agent integration guide. Those pages are how an agent discovers a GPU and runs work on it. This page is the Text Embeddings Inference command. A general vLLM install is How to serve an OpenAI-compatible API with vLLM on a cloud GPU. CloudGPU’s hosted gateway is a different base URL: How do I use an OpenAI-compatible API for open models?. The tunnel is How to use SSH port forwarding on a cloud GPU. Document parsers that feed a retrieval pipeline are How to convert PDFs and documents with Docling on a cloud GPU, How to convert PDFs to Markdown with MinerU on a cloud GPU, and How to convert PDFs to text with olmOCR on a cloud GPU. If the process runs out of memory: How to fix CUDA out of memory on a cloud GPU. If the process raises no kernel image on an RTX 5090: How to fix CUDA no kernel image on an RTX 5090. An editor whose config.yaml sets an apiBase for chat, autocomplete, and embeddings is How to use Continue with a self-hosted model on a cloud GPU.

The LICENSE file at commit da9a9ce190444ef4326e1fbab15d298f0a1016c7 begins with “Apache License Version 2.0, January 2004”, and the appendix copyright line is “Copyright 2022 Hugging Face”. The GitHub license field on 2026-11-14 was Apache-2.0. That license text is not a CloudGPU price.

The commands below are copied from the README at that commit, blob c13dc4475b7fff27b32212513bd5496c785d265b. They were not run on a CloudGPU instance for this page. Anything this page adds around them is marked. The README includes benchmark images for BAAI/bge-base-en-v1.5 on an NVIDIA A10 with a sequence length of 512 tokens. This page does not read numbers off those images. The catalog this page read does not list A10.

Which models the README lists

The README says Text Embeddings Inference currently supports Nomic, BERT, CamemBERT, and XLM-RoBERTa models with absolute positions, JinaBERT with Alibi positions, and Mistral, Alibaba GTE, and Qwen2 models with Rope positions, plus MPNet, ModernBERT, Qwen3, and Gemma3.

Its Docker example sets model=Qwen/Qwen3-Embedding-0.6B. The supported-models table row for that id prints MTEB rank 4, model size 509M, and model type Qwen3. Other rows print a parenthetical in the size cell. This page does not copy those parentheticals. The re-ranker example sets model=BAAI/bge-reranker-large. The re-ranking table lists that id as task Re-Ranking and model type XLM-RoBERTa.

The sequence-classification section says Text Embeddings Inference currently supports CamemBERT and XLM-RoBERTa sequence classification models with absolute positions. The re-ranker section says v0.4.0 added support for CamemBERT, RoBERTa, XLM-RoBERTa, and GTE sequence classification models, and that re-rankers are sequence classification cross-encoders with a single class that scores the similarity between a query and a text.

The README prints parameter sizes in that table. It prints no GPU GB figure. This page does not turn 509M into a GPU GB figure and does not mark a catalog row as enough.

GET /v1/public/compute-capabilities does not list GPUs. It returns feature flags. The memory column is vram_gb from gpu-prices.json, updated_at 2026-11-14T00:03:37.866Z. GET /v1/public/gpu-catalog the same day (generated_at 2026-11-14T00:03:37.294Z) listed the same names and the same vram_gb values, plus Ascend 910B at 64, which is not in gpu-prices.json and is not listed here. The catalog response also includes an availability field. This page does not copy it. Every prices row has gpu_count 1. Every catalog row has min_gpu_count 1. Which listing you can start is on Deploy.

The catalog template list had no Text Embeddings Inference template.

Catalog name vram_gb
RTX 2080 8
RTX 3080 10
RTX 2080 Ti 11
RTX 3080 Ti 12
V100 16G 16
A30 24
RTX 3090 24
RTX 4090 24
RTX 4090D 24
RTX 4080 SUPER 32G 32
RTX 5090 32
RTX 5090D 32
V100 32G 32
A100 40G 40
A800 40G 40
A40 48
RTX 4090 48G 48
RTX 4090D 48G 48
RTX A6000 48
NVIDIA RTX PRO 5000 72G 72
RTX PRO 5000 72G 72
A100 80G 80
A800 80G 80
H800 80
RTX 6000D 83
H20 96G 96
RTX PRO 6000 96G 96
RTX PRO 6000 Max-Q 96
H20-3e 141G 141

Which image tag the README prints

The getting-started Docker command uses ghcr.io/huggingface/text-embeddings-inference:cuda-1.9. The architecture table does not contain the string cuda-1.9. This page does not decide which table row that tag matches. The table, copied without the Hopper row’s parenthetical, which names a card this page does not copy:

Architecture Platform Image
CPU x86_64 ghcr.io/huggingface/text-embeddings-inference:cpu-1.9
CPU aarch64 ghcr.io/huggingface/text-embeddings-inference:cpu-arm64-1.9
Volta x86_64 NOT SUPPORTED
Turing (T4, RTX 2000 series, …) x86_64 ghcr.io/huggingface/text-embeddings-inference:turing-1.9 (experimental)
Ampere 8.0 (A100, A30) x86_64 ghcr.io/huggingface/text-embeddings-inference:1.9
Ampere 8.6 (A10, A40, …) x86_64 ghcr.io/huggingface/text-embeddings-inference:86-1.9
Ada Lovelace (RTX 4000 series, …) x86_64 ghcr.io/huggingface/text-embeddings-inference:89-1.9
Hopper x86_64 ghcr.io/huggingface/text-embeddings-inference:hopper-1.9
Blackwell 10.0 (B200, GB200, …) x86_64 ghcr.io/huggingface/text-embeddings-inference:100-1.9 (experimental)
Blackwell 12.0 (GeForce RTX 50X0, …) x86_64 ghcr.io/huggingface/text-embeddings-inference:120-1.9 (experimental)
Blackwell 12.1 (DGX Spark GB10, …) multi ghcr.io/huggingface/text-embeddings-inference:121-1.9 (experimental)

The Turing row says Flash Attention is turned off by default because it suffers from precision issues, and that you can turn Flash Attention v1 on with USE_FLASH_ATTENTION=True.

The CUDA install section says GPUs with CUDA compute capabilities below 7.5 are not supported, and it names V100. The Volta row says NOT SUPPORTED. Catalog V100 16G is 16 and V100 32G is 32. This page does not mark those rows as supported.

The Ada Lovelace row says “RTX 4000 series”. The catalog lists RTX 4090, RTX 4090D, RTX 4080 SUPER 32G, RTX 4090 48G, and RTX 4090D 48G. The README does not print those catalog names. The tag that row prints is 89-1.9.

The Blackwell 12.0 row says “GeForce RTX 50X0” and marks the tag 120-1.9 experimental. The catalog lists RTX 5090 and RTX 5090D, both 32. The README does not print those catalog names. The Docker build comment for compute cap 120 says “GeForce RTX 50X0, RTX PRO 6000, …”. The catalog lists RTX PRO 6000 96G and RTX PRO 6000 Max-Q. The comment does not print “Max-Q” and does not print “96G”. This page does not assign the Max-Q row to that comment.

The Ampere 8.0 row names A100 and A30 and prints the tag 1.9. The catalog lists A100 40G, A100 80G, and A30. The README does not print “40G” or “80G”. The Ampere 8.6 row names A10 and A40 and an ellipsis, and prints the tag 86-1.9. The catalog lists A40. It does not list A10. This page does not read the ellipsis as a catalog name.

The Turing row says “RTX 2000 series”. The catalog lists RTX 2080 at 8 and RTX 2080 Ti at 11. The README does not print those two names. The tag is turing-1.9 and the row says experimental.

The Blackwell 10.0 row names B200 and GB200. The Blackwell 12.1 row names DGX Spark GB10. The catalog this page read does not list those names. The Hopper row’s parenthetical names a card this page does not copy. Catalog rows named H800, H20 96G, and H20-3e 141G are not that parenthetical. This page does not assign them to hopper-1.9.

The Docker note says NVIDIA drivers need to be compatible with CUDA version 12.2 or higher, and that GPUs need the NVIDIA Container Toolkit. The local CUDA section repeats the 12.2 line and says compute capability below 7.5 is not supported.

Install and serve

On Deploy, choose Raw GPU. The README’s getting-started command, copied. It says the volume avoids downloading weights every run.

model=Qwen/Qwen3-Embedding-0.6B
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run

docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id $model

The architecture tag for the card you started is the table above. This page does not replace cuda-1.9 in that command, because the README does not say which table row that tag is. If you use a row’s tag instead, that replacement is not what the getting-started block prints.

The help text, printed by text-embeddings-router --help in the README, says --hostname defaults to 0.0.0.0 and --port defaults to 3000. The Docker command does not pass those flags. It publishes host port 8080 to container port 80. The curls below use 127.0.0.1:8080. This page does not treat the help text’s default 3000 as the port in those curls.

The same help text says --model-id can be a path to a local directory. The air-gapped section downloads with git, then passes that path. Copied:

# (Optional) create a `models` directory
mkdir models
cd models

# Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install
git clone https://huggingface.co/Qwen/Qwen3-Embedding-0.6B

# Set the models directory as the volume path
volume=$PWD

# Mount the models directory inside the container with a volume and set the model ID
docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id /data/Qwen3-Embedding-0.6B

The block below is added on this page. It keeps the repository id. It replaces git clone with hf. The Hub v1.0 migration doc says the deprecated huggingface-cli command has been removed and that hf replaces it. The air-gapped section does not print huggingface-cli. How to download a Hugging Face model prints pip install -U "huggingface_hub" and hf download. This page did not download the repository. The Docker --model-id /data/Qwen3-Embedding-0.6B path matches the git clone’s directory name, not this --local-dir name. This page does not rewrite the Docker command to the new directory.

# Added on this page. The air-gapped section uses git clone.
pip install -U "huggingface_hub"
hf download Qwen/Qwen3-Embedding-0.6B --local-dir Qwen3-Embedding-0.6B

The re-ranker command, copied:

model=BAAI/bge-reranker-large
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run

docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id $model

A local CUDA build is cargo install --path router -F candle-cuda-turing for Turing, and cargo install --path router -F candle-cuda for Ampere, Ada Lovelace, Hopper, and Blackwell. The launch after that build is text-embeddings-router --model-id $model --port 8080 with model=Qwen/Qwen3-Embedding-0.6B. This page did not compile the router.

Routes

The README’s embed curl, copied. It follows the getting-started Docker command.

curl 127.0.0.1:8080/embed \
    -X POST \
    -d '{"inputs":"What is Deep Learning?"}' \
    -H 'Content-Type: application/json'

The README’s rerank curl, copied:

curl 127.0.0.1:8080/rerank \
    -X POST \
    -d '{"query": "What is Deep Learning?", "texts": ["Deep Learning is not...", "Deep learning is..."]}' \
    -H 'Content-Type: application/json'

The OpenAPI file at https://huggingface.github.io/text-embeddings-inference/openapi.json (fetched 2026-11-14, info.version 1.9.4) lists POST /embed, POST /rerank, and POST /v1/embeddings. The /v1/embeddings summary is “OpenAI compatible route. Returns a 424 status code if the model is not an embedding model.” Its request schema requires input. The /embed summary says it returns 424 if the model is not an embedding model, and its schema requires inputs. The /rerank summary says it returns 424 if the model is not a sequence classification model with a single class, and its schema requires query and texts.

The help text’s --root-path example says --root-path jinaai/jina-embeddings-v2-base-en serves the OpenAI-compatible embeddings route at /jinaai/jina-embeddings-v2-base-en/v1/embeddings. The getting-started command does not pass --root-path. The README does not print a curl for /v1/embeddings. The block below is added on this page. The string What is Deep Learning? is the /embed example. The path and the field name input are the OpenAPI path and the required field.

# Added on this page. The README does not print this curl.
curl 127.0.0.1:8080/v1/embeddings \
    -X POST \
    -d '{"input":"What is Deep Learning?"}' \
    -H 'Content-Type: application/json'

The README says you can consult the OpenAPI documentation using the /docs route, and that a Swagger UI is also at https://huggingface.github.io/text-embeddings-inference. The Docker command publishes host port 8080. How to use SSH port forwarding on a cloud GPU is how a browser on your computer reaches that port. This page does not add an ssh command. The hosted gateway at https://cloudgpu.app/v1 is the other guide, not this server.

The help text says --api-key makes requests send the key as a Bearer token. The getting-started command does not pass --api-key. The help text says the payload size limit defaults to 2MB, and the default value it prints is 2000000. This page does not change that default.

When it fails

CUDA out of memory. The README prints no GPU GB figure. The help text says --max-batch-tokens defaults to 16384 and that text-embeddings-inference cannot infer that number automatically, because the memory overhead depends on the model implementation. It also says --max-concurrent-requests defaults to 512 and --max-client-batch-size defaults to 32. If the process raises CUDA out of memory, use how to fix CUDA out of memory on a cloud GPU. This page did not change those defaults.

“CUDA error: no kernel image is available for execution on the device” on an RTX 5090. The Blackwell 12.0 row prints the tag 120-1.9 and says experimental. The getting-started command prints cuda-1.9, which is not a row in that table. The README does not pin a PyTorch build and does not name compute capability 12.0. The RTX 5090 guide says that card is compute capability 12.0 (sm_120). This page did not run either tag on an RTX 5090. Catalog RTX 5090 and RTX 5090D are 32. That is the catalog number only.

424 from a route. The OpenAPI summaries say /embed and /v1/embeddings return 424 if the model is not an embedding model, and /rerank returns 424 if the model is not a sequence classification model with a single class. The embed example’s model id is Qwen/Qwen3-Embedding-0.6B. The rerank example’s model id is BAAI/bge-reranker-large. This page does not swap those ids.

Compute capability below 7.5. The CUDA section says that range is not supported and names V100. The Volta image row says NOT SUPPORTED. This page does not claim the catalog V100 rows run this server.

What to copy off, then destroy

The Docker command mounts $volume at /data. The getting-started comment says that volume is where the weights go so they are not downloaded every run. The air-gapped section clones into models and mounts that directory. Copy that directory, and any file you wrote on the instance, before you destroy it. Save your work before you destroy a cloud GPU is the checklist. The terms say data on local instance storage is permanently deleted when the instance is destroyed. The server stops with the instance.

The pricing page FAQ states the billing rule in these words:

Per-minute options have a 1-minute minimum; unused prepaid minutes are refunded when you destroy the instance. Some configurations use whole-hour billing with a 1-hour minimum; unused time within a charged hour is not refunded on voluntary stop. The billing period, minimum charge and initial prepayment are shown before confirmation. No subscription.

Prices are not copied here.

Frequently asked questions

How much VRAM does the README print?

The README prints no GPU GB figure. Its table row for Qwen/Qwen3-Embedding-0.6B prints model size 509M and MTEB rank 4. This page does not turn 509M into a GPU GB figure and does not mark a catalog row as enough. The CUDA section says compute capability below 7.5 is not supported and names V100. The Volta image row says NOT SUPPORTED. Which listing you can start is on https://cloudgpu.app/deploy.

Which image tag and launch command does the README print?

The getting-started command is docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id Qwen/Qwen3-Embedding-0.6B. The architecture table does not contain cuda-1.9. Ada Lovelace (RTX 4000 series, …) is tag 89-1.9. Blackwell 12.0 (GeForce RTX 50X0, …) is tag 120-1.9 and the row says experimental. Ampere 8.0 (A100, A30) is tag 1.9. Ampere 8.6 (A10, A40, …) is tag 86-1.9. Turing (T4, RTX 2000 series, …) is tag turing-1.9 and the row says experimental. The re-ranker command uses the same cuda-1.9 tag with --model-id BAAI/bge-reranker-large. This page did not run them.

Which routes does the README document?

The README curls POST 127.0.0.1:8080/embed with inputs, and POST 127.0.0.1:8080/rerank with query and texts. The OpenAPI file version 1.9.4 also lists POST /v1/embeddings, summarized as an OpenAI compatible route, with required field input. It says /embed and /v1/embeddings return 424 if the model is not an embedding model, and /rerank returns 424 if the model is not a sequence classification model with a single class. The README says the /docs route serves the OpenAPI documentation. The Docker command publishes host port 8080. The SSH port-forwarding guide is how a browser on your computer reaches that port. This page did not open a tunnel.

What license does the file print?

The LICENSE file at commit da9a9ce190444ef4326e1fbab15d298f0a1016c7 begins with Apache License Version 2.0, January 2004. The appendix copyright line is Copyright 2022 Hugging Face. The GitHub license field on 2026-11-14 was Apache-2.0. The README blob at that commit is c13dc4475b7fff27b32212513bd5496c785d265b.

Sources (read 2026-11-14, Asia/Shanghai)

Cards you can start are on the Deploy page. Prices are on https://cloudgpu.app/pricing and are not repeated here.

Last updated: 2026-11-14 (Asia/Shanghai).