How do I run SGLang on a cloud GPU?

Cover: Run SGLang on a cloud GPU, sglang serve step by step

Deploy a Raw GPU, confirm that nvidia-smi reports CUDA 13 or newer, install SGLang with uv pip install --prerelease=allow sglang inside a Python 3.12 virtual environment, then start sglang serve MODEL_PATH --host 127.0.0.1 --port 30000 and reach it from your computer through an SSH tunnel; according to the official SGLang installation docs, the pip and uv install needs Python 3.11 or higher and a CUDA 13 environment.

Updated 2026-10-28

This guide follows the official SGLang installation, quickstart, server arguments and troubleshooting pages, plus NVIDIA’s CUDA release notes and CUDA GPUs page, all read on 2026-10-28. SGLang is a serving engine for large language models, released under the Apache-2.0 license. This guide is for people who want an SGLang server with an OpenAI-compatible API on a rented NVIDIA GPU such as the RTX 5090 or the other NVIDIA cards listed on /pricing.

Step 1: Deploy a Raw GPU and connect

On /deploy, pick an NVIDIA GPU and the Raw GPU option (JupyterLab + CUDA pre-installed, root SSH). Connect over SSH or open a terminal in JupyterLab.

VRAM need depends on model, precision and workflow; see How much VRAM do I need to run an LLM? before you choose a card and a model.

Step 2: Check the environment first

nvidia-smi          # GPU name, driver version and the "CUDA Version" the driver supports
nvcc --version      # CUDA toolkit compiler, if installed
python3 --version   # the SGLang docs ask for Python 3.11 or higher (uv can install 3.12 for you)

What to look for:

  • Driver version and CUDA Version in the nvidia-smi header. According to the official SGLang installation docs, “SGLang requires CUDA 13”, and NVIDIA’s CUDA release notes list driver 580 or newer for CUDA 13.x. If the driver is 580 or newer, continue with Step 3. If it is older than 580, the current SGLang release does not match that driver. The docs say “SGLang 0.5.19 is the last release with a CUDA 12 lane”, so 0.5.19 is the version for that driver, but they give no pip command for a CUDA 12 install of it; the CUDA 12 build they name is the Docker image lmsysorg/sglang:v0.5.19-cu129, which needs a Docker host. Read the v0.5.19 release notes before you install it, or pick another instance.
  • Compute capability. The SGLang troubleshooting section says its default attention backend, FlashInfer, “only supports sm75 and above”. NVIDIA’s CUDA GPUs page lists compute capability 12.0 for the GeForce RTX 5090, 8.9 for the GeForce RTX 4090 and 8.0 for the NVIDIA A100. For any other card, look it up on that page.
  • nvcc. If an install step later fails with CUDA_HOME environment variable is not set, you will need the toolkit path (Step 7).

Step 3: Create a Python environment with uv

According to the official SGLang installation docs, install uv with pip and create a virtual environment:

pip install --upgrade pip
pip install uv
uv venv --python 3.12
source .venv/bin/activate

The docs also describe a Docker install (lmsysorg/sglang images). That path needs Docker Engine and the NVIDIA Container Toolkit on the host; a Raw GPU is a container, so this guide uses the uv path.

Step 4: Install SGLang

uv pip install --prerelease=allow sglang

The installation docs explain the flag: some SGLang dependencies only publish pre-releases on PyPI, and without --prerelease=allow “uv older than 0.12.0 silently installs SGLang 0.5.9”. On uv 0.12.0 and newer the flag does nothing, so keep it.

Step 5: Start the server on 127.0.0.1

Replace MODEL_PATH with a Hugging Face model ID or a local model folder:

export SGLANG_KEY="$(openssl rand -hex 16)"   # your own key for this server
sglang serve MODEL_PATH \
  --host 127.0.0.1 --port 30000 \
  --api-key "$SGLANG_KEY"

According to the official SGLang server arguments page:

  • --host defaults to 127.0.0.1 and --port to 30000. The installation page’s own example binds 0.0.0.0; this guide keeps 127.0.0.1 so the server is only reachable through your tunnel.
  • --api-key sets the server’s API key, which “is also used in the OpenAI API compatible server”.
  • --context-length caps the context; by default it comes from the model’s config.json.
  • --mem-fraction-static is the share of GPU memory for model weights and the KV cache pool; the page says to use a smaller value if you see out-of-memory errors.

For a gated or private model, request access on the model’s Hugging Face page first, then export your Hugging Face token as HF_TOKEN. The SkyPilot example in the SGLang installation docs uses meta-llama/Llama-3.1-8B-Instruct, which is a gated repo, so request access on Hugging Face before you try it. To download weights before you serve them, see How to download a Hugging Face model on a cloud GPU.

One model ships its own SGLang build: for that case, follow How do I run Unlimited-OCR-2 on a cloud GPU? instead of this step.

Keep this terminal open. The quickstart says to wait for The server is fired up and ready to roll! before you send a request. To keep the server running after you close SSH, start it inside tmux or screen.

Step 6: Reach it from your computer and send a request

On your own computer, open an SSH tunnel with the user, host and port from your instance’s SSH command:

ssh -N -L 30000:127.0.0.1:30000 <user>@<instance-host> -p <ssh-port>

See How to use SSH port forwarding on a cloud GPU for details. Then send the quickstart request, with your key added:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $SGLANG_KEY" \
  -d '{
    "model": "MODEL_PATH",
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ]
  }'

The quickstart says the answer is in choices[0].message.content. Any OpenAI SDK client works the same way; the SGLang examples page installs the client with uv pip install openai and sets the base URL:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="<your SGLANG_KEY>")
reply = client.chat.completions.create(
    model="MODEL_PATH",
    messages=[{"role": "user", "content": "Say hello in one sentence."}],
)
print(reply.choices[0].message.content)

While a request runs, run nvidia-smi in a second terminal on the instance: the SGLang process should be listed with GPU memory in use.

Step 7: Troubleshooting

From the official SGLang installation and troubleshooting pages:

  • OSError: CUDA_HOME environment variable is not set: set it to your CUDA toolkit folder, for example export CUDA_HOME=/usr/local/cuda-<version> (find it with ls -d /usr/local/cuda*), then install again.
  • FlashInfer errors on an sm75+ card: start the server with --attention-backend triton --sampling-backend pytorch.
  • CUDA out of memory: lower --mem-fraction-static (the page gives 0.8 or 0.7 as examples). If it happens during prefill, reduce --chunked-prefill-size to 4096 or 2048; during decoding, lower --max-running-requests. A smaller --context-length also reduces what the server has to hold.
  • The server hangs while loading: the troubleshooting page says a very low avail mem in the log points to memory; decrease --mem-fraction-static or --chunked-prefill-size.
  • curl cannot connect: check that the server printed the ready line and that the tunnel and server use the same port.

Step 8: Save what you need, then destroy the instance

Data on the instance is deleted when you destroy it, so copy anything you want to keep (configs, logs, your own fine-tuned weights) to your computer first. GPU time is billed per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance; see /terms.

FAQ

What does SGLang need on a cloud GPU?

According to the official SGLang installation docs, the pip and uv install needs Python 3.11 or higher and a CUDA 13 environment, and the docs target Linux with NVIDIA GPUs. NVIDIA’s CUDA release notes list driver 580 or newer for CUDA 13.x, so check the driver version that nvidia-smi reports before you install. If it is older than 580, the docs name SGLang 0.5.19 as the last release with a CUDA 12 lane.

Which port does sglang serve use?

The SGLang server arguments page lists 30000 as the default port and 127.0.0.1 as the default host. Pass --host 127.0.0.1 --port 30000 explicitly so your SSH tunnel and client settings match.

Is SGLang OpenAI-compatible?

Yes. The SGLang quickstart sends requests to /v1/chat/completions, and its examples point an OpenAI SDK client at http://localhost:30000/v1. If you start the server with --api-key, send the same key as a Bearer token.

How do I fix CUDA out of memory errors in SGLang?

The SGLang troubleshooting page suggests a smaller --mem-fraction-static such as 0.8 or 0.7, a smaller --chunked-prefill-size such as 4096 or 2048 if the error happens during prefill, and a lower --max-running-requests if it happens during decoding. VRAM need depends on model, precision and workflow.

How is a Raw GPU billed while SGLang runs?

GPU time is billed per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance. See https://cloudgpu.app/terms for the full terms.