How to serve EXL2/EXL3 models with TabbyAPI on a cloud GPU

Rent a GPU on Deploy, then open a terminal in JupyterLab or over SSH. Install TabbyAPI from the official theroyallab/tabbyAPI repository the way the getting-started page prints, download an Exl3 repository with the downloader that page prints, and start the server on the host and port config_sample.yml sets. Copy the model directory and anything you saved before you destroy the instance, because destroying the instance deletes its disk and stops the server.

A general vLLM install is How to serve an OpenAI-compatible API with vLLM on a cloud GPU. A general SGLang install is How to run SGLang on a cloud GPU. A GGUF web UI is How to run text-generation-webui (TextGen) on a cloud GPU. A llama.cpp CUDA build is How to build llama.cpp with CUDA on a cloud GPU. CloudGPU’s hosted gateway is a different base URL: How do I use an OpenAI-compatible API for open models?. How published weight sizes are read: How much VRAM do I need to run an LLM?. If the process runs out of memory: How to fix CUDA out of memory on a cloud GPU. If the server raises no kernel image on an RTX 5090: How to fix CUDA no kernel image on an RTX 5090. The tunnel is How to use SSH port forwarding on a cloud GPU. The download helper this site prints is How to download a Hugging Face model. An OpenAI-compatible server whose quick start listens on port 23333 is How to serve LLMs with LMDeploy on a cloud GPU.

The TabbyAPI LICENSE file begins with “GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007”. The GitHub license field on 2026-11-14 was AGPL-3.0. The ExLlamaV2 LICENSE file begins with “MIT License”, and that repository’s GitHub license field was MIT. The ExLlamaV3 LICENSE file begins with “MIT License”, and its copyright line is “Copyright © 2025 Turboderp”. That license text is not a CloudGPU price.

The commands below are copied from the upstream files named in each step. They were not run on a CloudGPU instance for this page. Anything this page adds around them is marked.

What the READMEs list

The TabbyAPI README at commit a32a82e255d9ffe1b6e805cb37527a1f53d9a62f (committed 2026-11-10T23:16:09Z) says TabbyAPI uses the ExLlamaV3 backend and is the official API backend server for ExLlamaV3. Its supported-model list is:

  • Exl3, which the README calls highly recommended
  • FP16/BF16

The same list says TabbyAPI supports parallel batching using paged attention for Nvidia Ampere GPUs and higher. It does not list EXL2. This page does not add EXL2 to that list.

The ExLlamaV2 README at commit 7dc12af3a81f34ac3f27cd7602ed539b638933ca (committed 2026-03-04T13:12:19Z) opens with “This project is archived for now” and says development continues on ExLlamaV3. It says the official and recommended backend server for ExLlamaV2 is TabbyAPI. Its EXL2 section says EXL2 supports 2, 3, 4, 5, 6 and 8-bit quantization, and that the format allows mixing quantization levels to achieve any average bitrate between 2 and 8 bits per weight. The same section says this scheme allows Llama2 70B to run on a single 24 GB GPU with a 2048-token context at 2.55 bits per weight, and that 13B models run at 2.65 bits within 8 GB of VRAM. This page does not copy that file’s speed table.

The ExLlamaV3 README at commit 8c593e4e5c88da76a4aaf8f1b1838bb7c1fce4e0 (committed 2026-11-13T22:34:13Z) says EXL3 is based on QTIP, plus 2–8 bit cache quantization. It says exllamav3 declares a minimum torch version of 2.6.0 and a minimum CUDA version of 12.4. It prints no GPU GB figure. Its install section also prints an example wheel URL whose filename contains cu128 and torch2.8.0. This page does not install that wheel. The EXL3 section links a writeup on another site. This page does not copy that link.

Choosing a GPU

The current TabbyAPI README prints no GPU GB figure. The ExLlamaV3 README prints no GPU GB figure. The only GPU GB sentences this page found are the two in the archived ExLlamaV2 README: 24 GB for Llama2 70B at 2.55 bits per weight with a 2048-token context, and 8 GB for 13B models at 2.65 bits. Those sentences name those models. This page does not mark a catalog row as enough for a different file.

GET /v1/public/compute-capabilities does not list GPUs. It returns feature flags. The memory column is vram_gb from gpu-prices.json, updated_at 2026-11-14T01:00:35.518Z. GET /v1/public/gpu-catalog the same day (generated_at 2026-11-14T01:00:36.104Z) listed the same names and the same vram_gb values, plus Ascend 910B at 64, which is not in gpu-prices.json and is not listed here. The catalog response also includes an availability field. This page does not copy it. Every prices row has gpu_count 1. Every catalog row has min_gpu_count 1. Catalog rows whose vram_gb is 24 or greater are the values that match the number in the 24 GB sentence. Catalog rows whose vram_gb is 8 or greater are the values that match the number in the 8 GB sentence. Every prices row is 8 or greater. Matching the number is not a statement that the row runs the model the sentence names. Which listing you can start is on Deploy.

The catalog template list had no TabbyAPI template.

Catalog name vram_gb
RTX 2080 8
RTX 3080 10
RTX 2080 Ti 11
RTX 3080 Ti 12
V100 16G 16
A30 24
RTX 3090 24
RTX 4090 24
RTX 4090D 24
RTX 4080 SUPER 32G 32
RTX 5090 32
RTX 5090D 32
V100 32G 32
A100 40G 40
A800 40G 40
A40 48
RTX 4090 48G 48
RTX 4090D 48G 48
RTX A6000 48
NVIDIA RTX PRO 5000 72G 72
RTX PRO 5000 72G 72
A100 80G 80
A800 80G 80
H800 80
RTX 6000D 83
H20 96G 96
RTX PRO 6000 96G 96
RTX PRO 6000 Max-Q 96
H20-3e 141G 141

config_sample.yml comments say cache_size defaults to 4096 tokens and must be a multiple of 256, and that autosplit_reserve defaults to 96 MB on GPU 0. Those are comments in the sample, not a measured load. The Docker note says Docker’s default /dev/shm is 64 MiB and that the docker run command passes --shm-size=8g because ExLlamaV3 uses that shared memory for tensor parallelism and CPU MoE offload. The same note says the setting reserves nothing up front. That 8g figure is shared memory, not a vram_gb value and not a disk size.

Install TabbyAPI

On Deploy, choose Raw GPU. The getting-started page’s beginner steps, copied. It says Python 3.12 is preferred, and the README badge says Python 3.10 through 3.14.

git clone https://github.com/theroyallab/tabbyAPI
cd tabbyAPI

Then run start.sh on Linux or start.bat on Windows. The page says the script creates a virtual environment and installs dependencies, and that the API should start with no model loaded. start.sh at this commit ends with python3 start.py "$@". This page did not run it.

The advanced path on the same page, for CUDA 12, is pip install -U .[cu12] inside the activated environment. For CUDA 13 it prints pip install -U .[cu13] and says Python 3.12 or newer. It then says python main.py starts the API and does not upgrade dependencies. This page does not merge those lines into start.sh.

The README’s Docker command, copied:

docker pull ghcr.io/theroyallab/tabbyapi:latest
docker run --gpus all --shm-size=8g --name tabbyapi -p 5000:5000 -v /path/to/models:/app/models ghcr.io/theroyallab/tabbyapi:latest

The README says the API is exposed on http://localhost:5000. It lists tags latest (CUDA 12.8), cu13 (CUDA 13), latest-rocm, and latest-extras. This page did not start the container.

Download a model

The getting-started page’s downloader example, copied. It says the built-in downloader works from the terminal, and the example revision is 4.0bpw:

.\Start.bat download turboderp/Qwen2.5-VL-7B-Instruct-exl3 --revision 4.0bpw

common/args.py at this commit adds a download subcommand with repo_id, --folder-name, --revision, and --token. start.bat passes its arguments to python start.py. The block below is added on this page. It keeps the repo id and the revision, and calls the Linux script, which forwards arguments the same way.

# Added on this page. The getting-started page prints Start.bat.
./start.sh download turboderp/Qwen2.5-VL-7B-Instruct-exl3 --revision 4.0bpw

The block below is also added on this page. It keeps the same repo id and revision, and uses hf download. How to download a Hugging Face model prints pip install -U "huggingface_hub" and hf download. This page did not download the repository.

# Added on this page. The getting-started page uses TabbyAPI's downloader.
pip install -U "huggingface_hub"
hf download turboderp/Qwen2.5-VL-7B-Instruct-exl3 --revision 4.0bpw

The same page’s gated example names meta-llama/Llama-3.1-8B and passes --token. This page does not print a token.

config.yml and the API

config_sample.yml says a config file is optional if you keep the defaults. If you want one, copy config_sample.yml to config.yml. Keys this page quotes from that file:

  • network.host defaults to 127.0.0.1. The comment says 0.0.0.0 exposes all adapters.
  • network.port defaults to 5000. The comment says TabbyAPI exits if the port is taken.
  • network.disable_auth defaults to false.
  • network.web_ui defaults to true. The comment says the chat client is at /ui and the status page is at /ui/status.
  • model.model_dir defaults to models.
  • model.model_name is the initial model. The comment says it must be filled out to load a model when the process starts.
  • model.backend comment says the option is exllamav3.
  • model.max_seq_len comment says the default is min(max_position_embeddings, cache_size), and -1 reads config.json.
  • model.cache_mode sample value is FP16. The comment says the pair form is k_bits,v_bits with integers from 2 to 8.

The sample also sets dummy_model_names. The default list in that comment names a closed model id. This page does not print that id.

The getting-started page says to copy api_keys_sample.yml to api_keys.yml if you set keys by hand, and that autogenerated keys should be used instead. The repository root at this commit has api_tokens_sample.yml, not api_keys_sample.yml. common/auth.py sets AUTH_FILE to api_tokens.yml. This page does not treat those two filenames as the same file.

The usage page’s completions example, copied:

curl http://localhost:5000/v1/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "Meta-Llama-3-8B-exl3",
  "prompt": "Once upon a time,",
  "max_tokens": 400,
  "stream": false,
  "min_p": 0.05,
  "repetition_penalty": 1.05
}'

That example does not send a key. common/auth.py check_api_key accepts an x-api-key header, or an Authorization header whose first word is Bearer, and returns 401 with “Please provide an API key” when neither is present, unless disable_auth is on. Admin checks use x-admin-key or a Bearer admin key. The usage page says API keys can call generation and model-list routes, and admin keys can load and unload. It says deleting api_tokens.yml and restarting generates new keys.

The usage page also names POST /v1/chat/completions and GET /v1/models. It says the chat client path is /ui. With the sample host and port, that path is http://127.0.0.1:5000/ui. How to use SSH port forwarding on a cloud GPU is how a program on your computer reaches port 5000. This page does not add an ssh command.

When it fails

CUDA out of memory. The current README prints no GPU GB figure. The ExLlamaV2 sentences above name 24 GB and 8 GB for the models they name. If the process raises CUDA out of memory, use how to fix CUDA out of memory on a cloud GPU.

“CUDA error: no kernel image is available for execution on the device” on an RTX 5090. The TabbyAPI README’s CUDA 12.8 tag and the ExLlamaV3 README’s torch>=2.6.0 and CUDA >=12.4 line do not name compute capability 12.0. The RTX 5090 guide says that card is compute capability 12.0 (sm_120) and that PyTorch 2.7 and later, on a build for CUDA 12.8 or newer, is the line that announced support. This page did not install a torch wheel and did not re-test this server on an RTX 5090. Catalog RTX 5090 and RTX 5090D are 32. That is the catalog number only.

Shared memory. The Docker note says loading fails with an error naming the 64 MiB /dev/shm limit when that default is left in place. The documented change is --shm-size=8g or shm_size: "8g" in the bundled compose file. This page did not reproduce the error.

What to copy off, then destroy

The downloader writes the repository you named. model.model_dir defaults to models. Copy that directory, config.yml if you wrote one, and api_tokens.yml if the server generated it, before you destroy the instance. Save your work before you destroy a cloud GPU is the checklist. The terms say data on local instance storage is permanently deleted when the instance is destroyed. The server stops with the instance.

The pricing page FAQ states the billing rule in these words:

Per-minute options have a 1-minute minimum; unused prepaid minutes are refunded when you destroy the instance. Some configurations use whole-hour billing with a 1-hour minimum; unused time within a charged hour is not refunded on voluntary stop. The billing period, minimum charge and initial prepayment are shown before confirmation. No subscription.

Prices are not copied here.

Frequently asked questions

Which quant formats do the current READMEs list?

The TabbyAPI README lists Exl3, which it calls highly recommended, and FP16/BF16. It also says parallel batching using paged attention is for Nvidia Ampere GPUs and higher. It does not list EXL2. The ExLlamaV2 README says the project is archived and that development continues on ExLlamaV3. That README defines EXL2 as 2, 3, 4, 5, 6 and 8-bit quantization with an average bitrate between 2 and 8 bits per weight, and it names TabbyAPI as the recommended server. The ExLlamaV3 README says EXL3 is based on QTIP, plus 2-8 bit cache quantization. This page does not claim the current TabbyAPI README loads an EXL2 file.

What GPU memory figures do those files print?

The TabbyAPI README prints no GPU GB figure. The ExLlamaV2 README says Llama2 70B can run on a single 24 GB GPU with a 2048-token context at 2.55 bits per weight, and that 13B models run at 2.65 bits within 8 GB of VRAM. The ExLlamaV3 README prints no GPU GB figure. It says torch >= 2.6.0 and CUDA >= 12.4. config_sample.yml comments say cache_size defaults to 4096 tokens and autosplit_reserve defaults to 96 MB on GPU 0. The Docker note says the Docker default shared memory is 64 MiB and the run command passes --shm-size=8g. That 8g figure is shared memory, not a vram_gb value. This page does not mark a catalog row as enough. Which listing you can start is on https://cloudgpu.app/deploy.

Which host, port, and API routes does TabbyAPI document?

config_sample.yml sets host to 127.0.0.1 and port to 5000. The docker run command in the README publishes 5000:5000 and says the API is exposed on http://localhost:5000. The curl on the usage page posts to http://localhost:5000/v1/completions. It also names /v1/chat/completions and GET /v1/models. The usage page says the chat client path is /ui and the status page path is /ui/status. common/auth.py accepts an x-api-key header or an Authorization header that starts with Bearer, unless disable_auth is on. The SSH port-forwarding guide is how a program on your computer reaches port 5000. This page did not open a tunnel.

What license do the files print?

The TabbyAPI LICENSE file begins with GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007. The GitHub license field on 2026-11-14 was AGPL-3.0. The ExLlamaV2 LICENSE file begins with MIT License, and the GitHub license field was MIT. The ExLlamaV3 LICENSE file begins with MIT License. The copyright line names Turboderp and the year 2025. That license text is not a CloudGPU price.

Sources (read 2026-11-14, Asia/Shanghai)

  • TabbyAPI README (ExLlamaV3 backend, Exl3 and FP16/BF16, Ampere and higher, Docker command, port 5000, --shm-size=8g): commit a32a82e255d9ffe1b6e805cb37527a1f53d9a62f, committed 2026-11-10T23:16:09Z. GitHub license field AGPL-3.0
  • docs/01.-Getting-Started.md (start.sh / start.bat, pip install -U .[cu12] and .[cu13], python main.py, Start.bat download example turboderp/Qwen2.5-VL-7B-Instruct-exl3 revision 4.0bpw, api_keys_sample.yml): same commit
  • config_sample.yml (host 127.0.0.1, port 5000, disable_auth false, web_ui true, model_dir, model_name, backend comment exllamav3, max_seq_len, cache_size 4096, cache_mode FP16, autosplit_reserve 96 MB): same commit
  • common/args.py (download subcommand: repo_id, --revision, --folder-name, --token): same commit
  • common/auth.py (AUTH_FILE api_tokens.yml, x-api-key, Authorization Bearer, x-admin-key): same commit
  • docs/03.-Usage.md (/v1/completions, /v1/chat/completions, /v1/models, /ui, /ui/status, API and admin keys): same commit
  • start.sh (forwards arguments to python3 start.py): same commit
  • ExLlamaV2 README (archived, TabbyAPI named as the server, EXL2 bit widths, 24 GB and 8 GB sentences): commit 7dc12af3a81f34ac3f27cd7602ed539b638933ca, committed 2026-03-04T13:12:19Z. GitHub license field MIT
  • ExLlamaV3 README (torch>=2.6.0, CUDA >=12.4, EXL3 and 2–8 bit cache, example wheel filename): commit 8c593e4e5c88da76a4aaf8f1b1838bb7c1fce4e0, committed 2026-11-13T22:34:13Z. LICENSE begins with MIT License, copyright line Copyright (c) 2025 Turboderp. GitHub license field MIT
  • Public GPU memory: https://cloudgpu.app/gpu-prices.json (updated_at 2026-11-14T01:00:35.518Z) and https://cloudgpu.app/v1/public/gpu-catalog (generated_at 2026-11-14T01:00:36.104Z). GET /v1/public/compute-capabilities returned feature flags and no GPU list. The catalog had no TabbyAPI template

Cards you can start are on the Deploy page. Prices are on https://cloudgpu.app/pricing and are not repeated here.

Last updated: 2026-11-14 (Asia/Shanghai).