How do I run text-generation-webui (TextGen) on a cloud GPU?

Cover: Run text-generation-webui (TextGen) on a cloud GPU, web UI and API over SSH

Clone https://github.com/oobabooga/textgen on a GPU instance, install requirements/portable/requirements.txt in a Python venv, put a GGUF file in user_data/models, start python server.py --portable --api --api-key <key>, and open the web UI (port 7860) and API (port 5000) on your own computer through an SSH tunnel.

Updated 2026-10-28

This guide follows the official TextGen README, the project’s OpenAI-compatible API doc and its download-model.py script, all read on 2026-10-28. The project is now called TextGen at https://github.com/oobabooga/textgen, and the old repository URL github.com/oobabooga/text-generation-webui redirects there. License: AGPL-3.0. The guide is for people who use TextGen on their own machine and want the same setup on a rented GPU such as the RTX 5090 32G or the other cards listed on /pricing.

Step 1: Deploy a Raw GPU and connect

On /deploy, pick a GPU and the Raw GPU option (JupyterLab + CUDA pre-installed). Connect over SSH, or open a terminal in JupyterLab. VRAM need depends on model, precision and workflow; see How much VRAM do I need to run an LLM? before you choose a card and a quantization.

Step 2: Check the environment first

nvidia-smi          # GPU name, driver, and the "CUDA Version" the driver supports
nvcc --version      # CUDA toolkit compiler (the portable requirements install prebuilt wheels; note the version anyway)
python3 --version   # the portable install needs Python 3.9 or newer
python3 -m venv --help >/dev/null && echo "venv OK"
git --version
tmux -V

Write down the CUDA Version shown at the top right of nvidia-smi; Step 4 uses it. If python3 -m venv fails, install the venv package for your distribution (on Debian/Ubuntu: sudo apt-get install -y python3-venv; drop sudo if you are root). Install tmux the same way if it is missing.

Step 3: Start a tmux session

Run TextGen inside tmux so it keeps running if your SSH connection drops:

tmux new -s textgen

Detach with Ctrl-b then d; reattach later with tmux attach -t textgen. For more on this, see How do I keep a job running after SSH disconnects?

Step 4: Clone TextGen and install the portable requirements

According to the official TextGen README, the “Manual portable install with venv” works on any Python 3.9+:

git clone https://github.com/oobabooga/textgen
cd textgen
python3 -m venv venv
source venv/bin/activate
pip install -r requirements/portable/requirements.txt --upgrade

The README says to choose the file under requirements/portable that matches your hardware. On 2026-10-28 the default requirements.txt pulled llama.cpp wheels tagged cu124, and requirements_cuda131.txt pulled wheels tagged cu131. Use the cuda131 file only if the CUDA Version in nvidia-smi from Step 2 is 13.1 or higher.

The portable install loads GGUF models through llama.cpp. The README notes that Transformers and EXL3 models, training and image generation need the full installation instead (one-click ./start_linux.sh, which asks for your GPU vendor, or the manual Conda steps in the README).

Step 5: Download a GGUF model

The repository ships download-model.py. Its built-in help lists --specific-file to pick one file and --output to choose the folder. Save a single GGUF file straight into user_data/models, where the README says the UI detects it:

python download-model.py <user>/<model>-GGUF \
  --specific-file <model>-Q4_K_M.gguf \
  --output user_data/models

Replace the placeholders with a GGUF repository and file name from Hugging Face. For other ways to fetch weights, see How do I download a Hugging Face model on a cloud GPU?

Step 6: Start the server with the API enabled

Create a key for the API, then start TextGen:

export TEXTGEN_KEY=$(python3 -c "import secrets; print(secrets.token_hex(16))")
echo "$TEXTGEN_KEY"   # keep this for your API client

python server.py --portable --api --api-key "$TEXTGEN_KEY" \
  --model <model>-Q4_K_M.gguf

According to the official TextGen docs:

  • The web UI is served on http://127.0.0.1:7860. --listen makes it reachable from the network; this guide leaves it off and uses an SSH tunnel instead.
  • --api starts the API on port 5000 by default (--api-port changes it), and --api-key makes clients send Authorization: Bearer <key>.
  • --model loads a file from user_data/models at launch, so you can skip the Model tab.
  • --gpu-layers sets how many layers go to the GPU (-1 = auto), and --ctx-size sets the context length.
  • --gradio-auth username:password adds a login to the web UI if you want one.

Leave --auto-launch out: it tries to open a browser on the server.

Step 7: Open the UI and the API from your computer

From your own computer, forward both ports through one SSH connection (use the SSH host, port and user from your instance page). For more on tunnels, see How to use SSH port forwarding on a cloud GPU:

ssh -N -L 7860:127.0.0.1:7860 -L 5000:127.0.0.1:5000 <user>@<instance-host> -p <ssh-port>

Then open http://127.0.0.1:7860 in your browser. Test the API from a second local terminal:

curl http://127.0.0.1:5000/v1/models \
  -H "Authorization: Bearer <your TEXTGEN_KEY>"

curl http://127.0.0.1:5000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <your TEXTGEN_KEY>" \
  -d '{"messages":[{"role":"user","content":"Say hello in one sentence."}]}'

The project’s OpenAI-compatible API doc says /v1/models lists models with the loaded one first, and the full endpoint reference is at http://127.0.0.1:5000/docs while the server runs. Any OpenAI SDK client works the same way with base_url="http://127.0.0.1:5000/v1" and your key.

Step 8: Confirm the model is on the GPU

While a reply is generating, run nvidia-smi in another terminal on the instance (or a new tmux window: Ctrl-b then c). TextGen’s --help describes its llama.cpp loader as passing flags to llama-server, so a llama-server or python process should be listed with GPU memory in use. If no process uses GPU memory, check the server log for how many layers were offloaded and try --gpu-layers -1.

Step 9: Save what you need, then destroy the instance

Data on the instance is deleted when you destroy it. Copy anything you want to keep (chat logs and settings under user_data, your own model files) to your computer first. CloudGPU bills per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance; see /terms.

Troubleshooting

  • python3 -m venv venv fails, or venv/bin/activate does not exist: the venv package is missing; see Step 2.
  • The model does not show up in the Model tab: a single GGUF file must sit directly in user_data/models; multi-file Transformers or EXL3 models go in a subfolder and need the full installation.
  • no kernel image is available for execution on the device: the llama.cpp wheel does not match your GPU or driver. Compare the CUDA Version in nvidia-smi with the cu124 / cu131 file choice in Step 4 and reinstall with the other file if it applies.
  • CUDA out of memory when loading: lower --ctx-size, choose a smaller quantization, or leave --gpu-layers on -1 (auto).
  • Browser cannot reach 7860, or curl cannot reach 5000: check that the SSH tunnel is still open and uses the same ports you passed to the server.
  • API returns 401: send the header Authorization: Bearer <key> with the exact value you passed to --api-key.

FAQ

Is text-generation-webui the same project as TextGen?

Yes. On 2026-10-28 the old GitHub address github.com/oobabooga/text-generation-webui redirected to github.com/oobabooga/textgen, and the install commands in this guide come from that repository’s README.

Which ports does TextGen use?

According to the official TextGen docs, the web UI is served on 127.0.0.1:7860 and the API started with --api listens on port 5000 by default. Use --listen-port and --api-port to change them.

Does TextGen have an OpenAI-compatible API?

Yes. Start the server with --api, and the project’s OpenAI-compatible API doc describes endpoints such as POST /v1/chat/completions and GET /v1/models. Add --api-key so clients must send Authorization: Bearer .

Which install should I use on a cloud GPU?

For GGUF models, the README’s manual portable install with venv is enough. Transformers and EXL3 models, training and image generation need the full installation, either the one-click start_linux.sh script or the manual Conda steps.

How much VRAM do I need for TextGen?

VRAM need depends on model, precision and workflow. See the CloudGPU guide How much VRAM do I need to run an LLM? at https://cloudgpu.app/guides/how-much-vram-do-i-need-to-run-an-llm before you pick a card and a quantization.

How is a TextGen session billed on CloudGPU?

CloudGPU bills GPU instances per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance. See https://cloudgpu.app/terms for the full terms.