How do I install DeepSpeed on a cloud GPU?

Install a CUDA build of PyTorch first, then run pip install deepspeed in the same environment and check it with ds_report; DeepSpeed compiles its CUDA ops just-in-time with nvcc and ninja, so most install errors come from a missing CUDA toolkit (CUDA_HOME does not exist), a toolkit whose version does not match torch.version.cuda, or ops built for a different GPU.

Updated 2026-11-02

This guide follows the official DeepSpeed installation docs, the DeepSpeed 0.19.9 source and release files, the PyTorch install page and source, the NVIDIA CUDA Installation Guide for Linux and the NVIDIA CUDA GPUs list, all read on 2026-11-02. It is for people whose training script (Hugging Face Trainer, Accelerate, Axolotl or plain DeepSpeed) stops with a DeepSpeed build error on a rented GPU.

What DeepSpeed needs

According to the official DeepSpeed docs and its PyPI page (read 2026-11-02):

  • The latest release is 0.19.9, published on GitHub on 2026-10-30 (UTC+8). PyPI has it as a source package only (deepspeed-0.19.9.tar.gz, no prebuilt wheel); the docs list “Pre-compiled DeepSpeed builds from PyPI” as “Coming soon”.
  • pip install deepspeed installs DeepSpeed “not tied to specific PyTorch or CUDA versions”. Its C++/CUDA extensions (“ops”) are, by default, “built just-in-time (JIT) using torch’s JIT C++ extension loader that relies on ninja”. In this default mode nothing is compiled during pip install; the CUDA compiler is needed the first time your code uses an op such as FusedAdam.
  • Its requirements file (0.19.9) lists torch>=2.0.0 and ninja among the dependencies.
  • To compile a CUDA op, DeepSpeed needs the CUDA toolkit compiler nvcc. A working nvidia-smi and a CUDA build of PyTorch are not enough on their own: the driver and the PyTorch wheel do not include nvcc.

Step 1: Deploy a Raw GPU and connect

  1. Open /deploy and choose Raw GPU. Our deploy page describes it as “a raw GPU with JupyterLab + CUDA pre-installed. SSH access, root on a dedicated container, install any Python/CUDA libs you want.”
  2. Pick a card. Cards listed on /pricing on 2026-11-02 include the RTX 5090 32G, RTX 4090 48G, RTX 4090 24G, RTX 3090 24G, A100 80G, A800 80G, H20 96G, V100 32G and RTX PRO 6000 96G. See current rates on /pricing. VRAM need depends on model, precision and workflow; see How much VRAM do I need to run an LLM?.
  3. Open a terminal in JupyterLab, or connect over SSH with your instance’s connection details:
ssh -p <port> <user>@<host>

Step 2: Check the environment first

nvidia-smi                                              # driver and "CUDA Version" in the header
nvidia-smi --query-gpu=name,compute_cap --format=csv    # card name and compute capability
nvcc --version                                          # CUDA toolkit compiler, needed for DeepSpeed ops
which nvcc; echo "CUDA_HOME=${CUDA_HOME}"; ls -d /usr/local/cuda* 2>/dev/null
cat /etc/os-release | head -3                           # distro, needed if you install the toolkit
python3 -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"

Write down four values: compute_cap, the nvcc release (or “not found”), torch.version.cuda, and whether torch.cuda.is_available() printed True.

  • If torch.version.cuda prints None, or is_available() prints False, you have a CPU-only PyTorch (or no GPU visible). Fix PyTorch first, in Step 3.
  • If import torch fails, PyTorch is not installed in this environment yet.
  • If nvcc is not found, Step 4 shows how to add it.

Our deploy page lists “JupyterLab + CUDA pre-installed” for Raw GPU but does not list the driver version, whether nvcc is on PATH, or whether PyTorch is pre-installed, so read them from the commands above on your own instance.

Step 3: Get a CUDA build of PyTorch

Use a fresh virtual environment so you don’t break an existing one:

python3 -m venv ~/venvs/ds && source ~/venvs/ds/bin/activate
python -m pip install --upgrade pip

Then install PyTorch with the command from the PyTorch install page. On 2026-11-02 the current stable release is 2.15.0, offered for CUDA 13.2 (cu132) and CUDA 13.4 (cu134). The “CUDA Version” in the nvidia-smi header is the newest CUDA version your driver supports, so compare it with the build you pick. For example:

pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu132
python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"

If your project pins a different PyTorch, see How to install a specific PyTorch CUDA version on a cloud GPU. Note the value torch.version.cuda prints (for example 13.2): the CUDA toolkit in Step 4 has to match it.

Step 4: If nvcc is missing, install the CUDA toolkit that matches torch.version.cuda

Run nvcc --version first. If it prints the same release as torch.version.cuda (for example release 13.2 and 13.2), skip to Step 5. If nvcc --version fails, install the CUDA toolkit that matches torch.version.cuda as shown below. If it works but prints a different release, install the matching toolkit the same way, or install a PyTorch build for the toolkit you have.

Reader check: our deploy page says Raw GPU comes with CUDA pre-installed but does not say whether that includes nvcc, so run nvcc --version on your own instance before you install anything.

DeepSpeed finds the CUDA toolkit through PyTorch. In the PyTorch 2.15.0 source, the lookup order is: the CUDA_HOME or CUDA_PATH variable, then the folder above nvcc on your PATH, then /usr/local/cuda. DeepSpeed then runs ${CUDA_HOME}/bin/nvcc -V and compares its release with torch.version.cuda.

If a toolkit is installed but PyTorch does not find it, point at it (replace 13.2 with the version from ls -d /usr/local/cuda* that matches torch.version.cuda):

export CUDA_HOME=/usr/local/cuda-13.2
export PATH="${CUDA_HOME}/bin:${PATH}"
nvcc --version

If no toolkit is installed, the NVIDIA CUDA Installation Guide for Linux uses NVIDIA’s network repository. On Ubuntu 24.04 (distro string ubuntu2404; the guide’s table also lists ubuntu2204 and ubuntu2604), install the toolkit package whose version matches torch.version.cuda. NVIDIA’s Ubuntu 24.04 repository listed cuda-toolkit-12-8, cuda-toolkit-12-9 and cuda-toolkit-13-0 to cuda-toolkit-13-4 on 2026-11-02:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb
apt update
apt install -y cuda-toolkit-13-2          # match torch.version.cuda, here 13.2
export CUDA_HOME=/usr/local/cuda-13.2
export PATH="${CUDA_HOME}/bin:${PATH}"
nvcc --version

You are root on a Raw GPU, so sudo is not needed. According to the NVIDIA guide, starting with CUDA 13.4 the toolkit and the driver are versioned and installed independently, and the toolkit meta packages do not install the driver. Do not try to install a driver inside the instance; nvidia-smi already shows the one you have. Add the two export lines to ~/.bashrc so new terminals keep them.

Why the exact version matters: in the DeepSpeed 0.19.9 source, a different minor version is accepted only within CUDA 10.x, 11.x and 12.x (for example a toolkit 12.8 with a PyTorch built for 12.6). For CUDA 13.x there is no such list, so the toolkit’s major and minor version must equal torch.version.cuda, or the op build stops with the mismatch error below.

Step 5: Install DeepSpeed and run ds_report

In the same environment:

pip install deepspeed
ds_report

ds_report (also python -m deepspeed.env_report) is DeepSpeed’s own environment report. According to the 0.19.9 source it prints:

  • ninja with [OKAY] or [FAIL]: JIT-compiled ops require ninja.
  • A table of ops with “installed” and “compatible” columns. The docs say the report shows “which ops your machine is compatible with”.
  • General info, including torch version, torch cuda version and nvcc version. If this line shows [FAIL] cannot find CUDA_HOME or [FAIL] nvcc missing, go back to Step 4.

The report includes install paths; remove anything sensitive before you post it in an issue.

Reader check: before you start training, confirm that ninja shows [OKAY] and that the nvcc version line shows the same release as torch cuda version.

Step 6: Test one CUDA op on the GPU

The import alone does not compile anything. Creating a FusedAdam optimizer does: in the 0.19.9 source its constructor calls FusedAdamBuilder().load(), which JIT-builds the fused_adam op the first time.

python - <<'PY'
import torch
from deepspeed.ops.adam import FusedAdam
p = torch.nn.Parameter(torch.zeros(1024, device="cuda"))
opt = FusedAdam([p], lr=1e-3)
p.grad = torch.ones_like(p)
opt.step()
print("fused_adam OK", p[:3])
PY

If this prints fused_adam OK, DeepSpeed can compile and run CUDA ops on this card. The built op is stored under ~/.cache/torch_extensions/ by default and reused next time. The DeepSpeed docs suggest setting TORCH_EXTENSIONS_DIR to a different folder per virtual environment, because ops built in one environment can fail to load in another:

export TORCH_EXTENSIONS_DIR=~/torch-extensions-ds

Reader check: run this test on every new instance before a long training job, and again if you switch to a different card, because an op built for one card may not load on another (see the “no kernel image” fix below).

Fix: “CUDA_HOME does not exist, unable to compile CUDA op(s)”

This is DeepSpeed’s MissingCUDAException (0.19.9 op_builder/builder.py). PyTorch found no CUDA toolkit: no CUDA_HOME/CUDA_PATH, no nvcc on PATH and no /usr/local/cuda. DeepSpeed GitHub issue #2772 gives the same cause: a working PyTorch has the CUDA runtime, but the CUDA compiler is a separate component. Fix it with Step 4, then run ds_report and check the nvcc version line. A related message, Unable to run .../bin/nvcc, means CUDA_HOME points at a folder without nvcc (for example a runtime-only install).

Fix: “Installed CUDA version X does not match the version torch was compiled with Y”

The full message in the docs is Exception: >- DeepSpeed Op Builder: Installed CUDA version {VERSION} does not match the version torch was compiled with {VERSION}, unable to compile cuda/cpp extensions without a matching cuda version. The docs’ fix is to change the installed CUDA toolkit (check with nvcc --version) or the PyTorch build so they match. They “only require that the major version matches”, but warn that a minor mismatch “may still” cause errors, and recommend matching both. As noted in Step 4, the 0.19.9 source only accepts a minor mismatch for CUDA 10.x to 12.x.

The docs also describe an override that skips the check, “but beware of unexpected behavior”:

DS_SKIP_CUDA_CHECK=1 python your_script.py

Use it only for a minor-version difference you have decided to accept; installing the matching cuda-toolkit-X-Y package is the cleaner fix.

Fix: “Unable to JIT load the … op due to ninja not being installed”

JIT builds need ninja. It is in DeepSpeed’s requirements, so this usually means a different environment is active. Check with which python ninja, then:

pip install ninja
ds_report

Fix: “CUDA error: no kernel image is available for execution on the device”

The DeepSpeed docs say this means “the CUDA extensions weren’t built for the card you’re trying to use it for”. In JIT mode DeepSpeed builds for the GPUs visible at build time, so an op built on one card and reused on another can fail. Set the architecture explicitly with TORCH_CUDA_ARCH_LIST, clear the old build and run again:

nvidia-smi --query-gpu=compute_cap --format=csv,noheader   # e.g. 12.0
export TORCH_CUDA_ARCH_LIST="12.0"
rm -rf ~/.cache/torch_extensions "${TORCH_EXTENSIONS_DIR}"

NVIDIA’s CUDA GPUs list gives the GeForce RTX 5090 and the RTX PRO 6000 Blackwell compute capability 12.0. In the 0.19.9 source, DeepSpeed’s own default architecture list adds 12.0 only when the toolkit is CUDA 12.8 or newer, so on these cards use a toolkit and a PyTorch build on CUDA 12.8 or newer. For PyTorch errors with the same text, see How to fix “no kernel image is available” on an RTX 5090.

Optional: pre-build ops at install time

The docs describe building ops during pip install instead of at first use. PyTorch must be installed first, and nvcc must match (Steps 3 and 4):

DS_BUILD_FUSED_ADAM=1 pip install deepspeed        # only the FusedAdam op
DS_BUILD_OPS=1 pip install deepspeed               # every op compatible with this machine

According to the docs, DeepSpeed “will only install any ops that are compatible with your machine”. Run long builds inside tmux or screen so they keep going if your SSH session drops (see How to keep a job running after SSH disconnects), and set TORCH_CUDA_ARCH_LIST to your card’s compute capability so the build targets it.

Data on local instance storage is permanently deleted when you destroy the instance (see /terms), so keep your install commands (Steps 3 to 5) in a script you can re-run on the next instance. Billed per minute, rounded up, with a 1-minute minimum; billing stops when you destroy the instance (see /terms).

Troubleshooting

Symptom What to check Source
CUDA_HOME does not exist, unable to compile CUDA op(s) No CUDA toolkit found; install cuda-toolkit-X-Y matching torch.version.cuda and set CUDA_HOME (Step 4) DeepSpeed 0.19.9 op_builder/builder.py, issue #2772
ds_report shows [FAIL] nvcc missing CUDA_HOME points at a folder without bin/nvcc DeepSpeed 0.19.9 env_report.py
Installed CUDA version X does not match the version torch was compiled with Y Toolkit and PyTorch CUDA differ; match both major and minor DeepSpeed install docs
Unable to JIT load the ... op due to ninja not being installed pip install ninja in the active environment DeepSpeed 0.19.9 op_builder/builder.py
CUDA error: no kernel image is available for execution on the device Ops built for another architecture; set TORCH_CUDA_ARCH_LIST, clear ~/.cache/torch_extensions DeepSpeed install docs
Op fails to load after switching virtual environments Shared torch_extensions folder; set TORCH_EXTENSIONS_DIR per environment DeepSpeed install docs

FAQ

Do I need the CUDA toolkit to install DeepSpeed?

Not for pip install deepspeed itself: by default DeepSpeed builds its CUDA ops just-in-time when your code first uses them. That build needs nvcc from a CUDA toolkit whose version matches torch.version.cuda, so install the toolkit before you train with ops such as FusedAdam.

How do I fix “CUDA_HOME does not exist, unable to compile CUDA op(s)”?

PyTorch found no CUDA toolkit. Install the cuda-toolkit-X-Y package that matches torch.version.cuda (for example cuda-toolkit-13-2 for 13.2), set CUDA_HOME to /usr/local/cuda-X.Y, add its bin folder to PATH, and check the nvcc version line in ds_report.

How do I check that DeepSpeed is installed correctly?

Run ds_report (or python -m deepspeed.env_report). It shows whether ninja is found, which ops are installed and compatible, and the torch, torch CUDA and nvcc versions. Then create a FusedAdam optimizer on a CUDA tensor to make DeepSpeed build and run one op.

What does “Installed CUDA version does not match the version torch was compiled with” mean?

The nvcc toolkit and your PyTorch build use different CUDA versions. Install the toolkit that matches torch.version.cuda, or install a PyTorch build for your toolkit. DS_SKIP_CUDA_CHECK=1 skips the check, but the DeepSpeed docs warn of unexpected behavior.

Which TORCH_CUDA_ARCH_LIST should I use on an RTX 5090?

NVIDIA lists the GeForce RTX 5090 at compute capability 12.0, so set TORCH_CUDA_ARCH_LIST=“12.0” and use a CUDA 12.8 or newer toolkit and PyTorch build. Confirm on your instance with nvidia-smi --query-gpu=compute_cap --format=csv.

Is data kept after I destroy the instance?

No. Data on local instance storage is permanently deleted when you destroy the instance, so keep your install commands in a script. Billed per minute, rounded up, with a 1-minute minimum; billing stops when you destroy the instance.

Sources

All read on 2026-11-02.