How do I install onnxruntime-gpu on a cloud GPU?

Uninstall both onnxruntime and onnxruntime-gpu, then install only the onnxruntime-gpu build that matches the CUDA major version of your PyTorch (the current release for CUDA 13, onnxruntime-gpu==1.26.0 for CUDA 12), and confirm that a real InferenceSession lists CUDAExecutionProvider first; most “CUDAExecutionProvider is not in available provider names” errors come from the CPU package sitting next to the GPU one or from a CUDA/cuDNN mismatch.

Updated 2026-11-02

This guide follows the official ONNX Runtime CUDA Execution Provider and install docs, the onnxruntime-gpu and torch package metadata on PyPI, ONNX Runtime GitHub issues #26996 and #27177, and the PyTorch install page, all read on 2026-11-02. It is for people whose ComfyUI face nodes (insightface, ReActor, IPAdapter FaceID), background removal (rembg), taggers, TTS or their own ONNX models fall back to the CPU on a rented GPU.

What onnxruntime-gpu needs

According to the official ONNX Runtime CUDA Execution Provider docs (read 2026-11-02):

  • “Starting with version 1.27, GPU packages published to PyPI (onnxruntime-gpu) … are built with CUDA 13.0 by default”, and “ONNX Runtime built with CUDA 13.0 require CUDA 13.0 or newer”.
  • Versions 1.21 to 1.26 were built with CUDA 12.8 (“Default GPU package build before 1.27”) and require CUDA 12.8 or newer. The docs mention separate CUDA 12.x packages for 1.27 and later, but the install page read the same day gives no pip command for them.
  • Every current build in the tables uses cuDNN 9.x, and “ONNX Runtime built with cuDNN 8.x is not compatible with cuDNN 9.x, and vice versa.”
  • With PyTorch, “the necessary CUDA and cuDNN DLLs are included, eliminating the need for separate installations of the CUDA toolkit or cuDNN”, provided both are built against the same CUDA and cuDNN major version. Run import torch before you create the session, or call onnxruntime.preload_dlls() (available since 1.21.0).

Package metadata on PyPI (read 2026-11-02):

  • The latest onnxruntime-gpu is 1.31.1 (uploaded 2026-10-27 UTC+8). Its cuda and cudnn extras pull CUDA 13 packages (nvidia-cuda-runtime~=13.0, nvidia-cudnn-cu13~=9.0). Version 1.26.0 pulls the -cu12 packages instead.
  • Recent releases require Python 3.11 or newer (requires_python >=3.11; Linux x86_64 wheels for cp311 to cp314).
  • torch 2.15.0 on PyPI depends on CUDA 13 packages on Linux. The PyTorch install page lists 2.15.0 as the stable release, offered for CUDA 13.2 (cu132) and CUDA 13.4 (cu134).

On the default CUDA version, the install page and the CUDA EP page currently disagree; we follow PyPI.

You do not need nvcc for onnxruntime-gpu: it ships prebuilt wheels and loads the CUDA runtime libraries at run time.

Step 1: Deploy a Raw GPU and connect

  1. Open /deploy and choose Raw GPU. Our deploy page describes it as “a raw GPU with JupyterLab + CUDA pre-installed. SSH access, root on a dedicated container, install any Python/CUDA libs you want.”
  2. Pick a card. Cards listed on /pricing on 2026-11-02 include the RTX 5090 32G, RTX 4090 48G, RTX 4090 24G, RTX 3090 24G, A100 80G, A800 80G, H20 96G and RTX PRO 6000 96G. See current rates on /pricing. VRAM need depends on model, precision and workflow; for image workflows see How much VRAM do I need for FLUX and SDXL?.
  3. Open a terminal in JupyterLab, or connect over SSH with your instance’s connection details:
ssh -p <port> <user>@<host>

Step 2: Check the environment first

Run these in the Python environment that will import ONNX Runtime (for ComfyUI, the environment that runs ComfyUI):

nvidia-smi                                   # driver, card, and "CUDA Version" in the header
nvcc --version                               # CUDA toolkit compiler; not required by onnxruntime-gpu, just record it
python3 --version                            # needs 3.11 or newer for current onnxruntime-gpu wheels
python3 -m pip list 2>/dev/null | grep -i -E "onnxruntime|^torch "
python3 -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"

Write down: the “CUDA Version” in the nvidia-smi header, the Python version, every onnxruntime* package that pip list shows, and torch.version.cuda.

  • Two lines such as onnxruntime and onnxruntime-gpu mean you have hit the most common cause of the error. Step 4 fixes it.
  • If torch.version.cuda prints None or import torch fails, you have no CUDA build of PyTorch. Either install one first (see How to install a specific PyTorch CUDA version) or use Path C in Step 3.

Our deploy page lists “JupyterLab + CUDA pre-installed” for Raw GPU but does not list the Python version, the driver version or whether PyTorch is pre-installed, so read them from the commands above on your own instance.

Step 3: Pick the build that matches your CUDA

Your environment Install Why
Path A: torch.version.cuda is 13.x, nvidia-smi header shows 13.0 or newer onnxruntime-gpu (current release) 1.27 and later are built with CUDA 13.0 and cuDNN 9
Path B: torch.version.cuda is 12.8 or newer 12.x onnxruntime-gpu==1.26.0 The last release whose default PyPI build targets CUDA 12, per PyPI/docs read 2026-11-02
Path C: no PyTorch in this environment onnxruntime-gpu[cuda,cudnn] (or onnxruntime-gpu[cuda,cudnn]==1.26.0 if the nvidia-smi header shows 12.x) The extras install the CUDA and cuDNN runtime libraries from pip

onnxruntime-gpu==1.26.0 is the last release whose default PyPI build targets CUDA 12, per PyPI/docs read 2026-11-02. Re-check the CUDA Execution Provider docs and PyPI before you pin it.

If torch.version.cuda is older than 12.8, upgrade PyTorch first; the CUDA 12 builds in the docs table require 12.8 or newer.

Step 4: Remove every ONNX Runtime package, then install one

ONNX Runtime assumes one package per environment. In issue #26996, an ONNX Runtime maintainer explains that the error comes from a “violation of an assumption that at most 1 onnxruntime packages should be installed”, and that the workaround is “to uninstall both package and re-install the onnxruntime-gpu package.”

python3 -m pip uninstall -y onnxruntime onnxruntime-gpu
python3 -m pip list 2>/dev/null | grep -i onnxruntime     # should print nothing now

# Path A (CUDA 13 PyTorch)
python3 -m pip install onnxruntime-gpu

# Path B (CUDA 12.8+ PyTorch)
# python3 -m pip install "onnxruntime-gpu==1.26.0"

# Path C (no PyTorch)
# python3 -m pip install "onnxruntime-gpu[cuda,cudnn]"

If pip list still shows another onnxruntime-* package after the uninstall (for example a DirectML or OpenVINO build pulled in by a dependency), uninstall it too.

Step 5: Confirm the GPU in a real session

ort.get_available_providers() is a first check, but issue #27177 notes that it reports providers the package was built with, not providers that can actually load. Creating a session is the real test. This script builds a one-node ONNX model, so you don’t need to download anything (pip install onnx first):

import torch  # Path A/B: loads the CUDA and cuDNN libraries from PyTorch; on Path C delete this line
import onnxruntime as ort
import numpy as np
import onnx
from onnx import helper, TensorProto

# ort.preload_dlls()  # Path C: uncomment to load CUDA/cuDNN from the pip packages

print("onnxruntime", ort.__version__, "device:", ort.get_device())
print("available:", ort.get_available_providers())

x = helper.make_tensor_value_info("X", TensorProto.FLOAT, [1, 4])
y = helper.make_tensor_value_info("Y", TensorProto.FLOAT, [1, 4])
graph = helper.make_graph([helper.make_node("Relu", ["X"], ["Y"])], "relu", [x], [y])
model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 17)])
model.ir_version = 9
onnx.save(model, "relu.onnx")

sess = ort.InferenceSession("relu.onnx", providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
print("session providers:", sess.get_providers())
print(sess.run(None, {"X": np.array([[-1, 0, 1, 2]], dtype=np.float32)})[0])

What to look for:

  • session providers: starts with CUDAExecutionProvider: the session runs on the GPU.
  • session providers: shows only CPUExecutionProvider, with a warning or a Failed to load library libonnxruntime_providers_cuda.so line above it: the GPU provider could not load. Use the troubleshooting table below.
  • On Path C (no PyTorch), check that ort.preload_dlls() runs before the session and that session providers: starts with CUDAExecutionProvider without changing LD_LIBRARY_PATH. If it only shows the CPU, read the library name in the warning and check that the [cuda,cudnn] extras installed.
  • While a longer job runs, nvidia-smi in a second terminal should list your Python process.

ComfyUI: keep the GPU package after installing nodes

Many ComfyUI custom nodes list onnxruntime in their requirements. Installing such a node can put the CPU package back next to onnxruntime-gpu, and the face or background nodes quietly run on the CPU again.

  1. Find the Python that runs ComfyUI (for a virtual environment, activate it and run which python), and use that Python for every command in Steps 2 to 5.
  2. After you install or update custom nodes, run python -m pip list | grep -i onnxruntime again. If the CPU package is back, repeat Step 4.
  3. Restart ComfyUI so it loads the reinstalled package.

If you choose a card for ComfyUI first, see How to choose a rental GPU for ComfyUI.

Keep the commands from Steps 3 to 5 in a script you can re-run on your next instance. Billed per minute, rounded up, with a 1-minute minimum; billing stops when you destroy the instance (see /terms).

Troubleshooting

Symptom What to check Source
Specified provider 'CUDAExecutionProvider' is not in available provider names. Available providers: 'AzureExecutionProvider, CPUExecutionProvider' Both onnxruntime and onnxruntime-gpu installed, or only the CPU package. Repeat Step 4 Issue #26996
Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.12: cannot open shared object file (or .so.13) CUDA libraries for that major version are not loaded. import torch before the session, or install the [cuda,cudnn] extras and call ort.preload_dlls(); check the build matches your CUDA major (Step 3) CUDA EP docs
libcudnn.so.9: cannot open shared object file cuDNN 9 is missing. PyTorch with CUDA brings it; without PyTorch install the cudnn extra CUDA EP docs
Worked on CUDA 12, fails after pip install -U onnxruntime-gpu 1.27 and later are CUDA 13 builds; pin onnxruntime-gpu==1.26.0 or move PyTorch to CUDA 13 CUDA EP docs
No matching distribution found for onnxruntime-gpu Python older than 3.11 for current releases; check python3 --version PyPI
GPU provider worked, CPU again after installing a ComfyUI node The node reinstalled the CPU package; see the ComfyUI section Issue #26996

FAQ

Why does onnxruntime say CUDAExecutionProvider is not in available provider names?

Usually the CPU package onnxruntime is installed next to onnxruntime-gpu, or only the CPU package is installed. ONNX Runtime assumes one package per environment. Uninstall both, reinstall only onnxruntime-gpu, and check ort.get_available_providers() again.

Which onnxruntime-gpu version matches my CUDA version?

According to the ONNX Runtime CUDA docs, onnxruntime-gpu 1.27 and later on PyPI is built with CUDA 13.0 and needs CUDA 13.0 or newer; 1.21 to 1.26 were built with CUDA 12.8. Match it to torch.version.cuda: 13.x takes the current release, 12.x takes onnxruntime-gpu==1.26.0.

Do I need to install the CUDA toolkit or cuDNN for onnxruntime-gpu?

Not with pip. A CUDA build of PyTorch already brings the CUDA and cuDNN libraries, and import torch before creating the session loads them. Without PyTorch, install onnxruntime-gpu[cuda,cudnn] and call onnxruntime.preload_dlls() before you create the session.

How do I check that ONNX Runtime really runs on the GPU?

Create an InferenceSession with providers=[‘CUDAExecutionProvider’, ‘CPUExecutionProvider’] and print session.get_providers(). If CUDAExecutionProvider is listed first, the session runs on the GPU. get_available_providers() alone only shows what the package was built with.

Why did onnxruntime-gpu stop working after I installed a ComfyUI custom node?

Some node requirements install the CPU package onnxruntime, which then sits next to onnxruntime-gpu in ComfyUI’s Python environment. Run pip list in that environment, uninstall both packages, reinstall onnxruntime-gpu and restart ComfyUI.

Sources

All read on 2026-11-02.