How do I install bitsandbytes on a cloud GPU?

Cover: Install bitsandbytes on a cloud GPU

Install a CUDA build of PyTorch first, then run pip install bitsandbytes in the same environment and check it with python -m bitsandbytes, which should end with SUCCESS!; bitsandbytes picks its CUDA library from torch.version.cuda, so almost every install error comes from a CPU-only PyTorch, a CUDA version it has no library for, or a GPU its library was not built for.

Updated 2026-11-01

This guide follows the official bitsandbytes installation and troubleshooting docs, the bitsandbytes 0.50.2 source and release files, the PyTorch install page, the NVIDIA CUDA GPUs list and the Hugging Face Transformers bitsandbytes docs, all read on 2026-11-01. It is for people whose QLoRA or 4-bit loading script (Transformers, PEFT, Unsloth, Axolotl) stops with a bitsandbytes CUDA error on a rented GPU.

What bitsandbytes needs

According to the bitsandbytes docs and its PyPI page (read 2026-11-01):

  • The latest release is 0.50.2, published on 2026-08-27. It needs Python 3.10 or newer and PyTorch 2.4 or newer (below 3).
  • NVIDIA GPUs with compute capability 6.0 or newer are supported. LLM.int8() needs 7.5 or newer; 8-bit optimizers and NF4/FP4 4-bit quantization need 6.0 or newer.
  • The Linux x86-64 wheel is a single file with one native library per CUDA version. The docs list which GPU architectures each one is built for:
bitsandbytes CUDA build GPU architectures (compute capability)
CUDA 11.8 to 12.6 sm60 to sm90 (6.0 to 9.0)
CUDA 12.8 sm70, sm75, sm80, sm86, sm89, sm90, sm100, sm120
CUDA 13.0 to 13.2 sm75 to sm120

Step 1: Deploy a Raw GPU and connect

  1. Open /deploy and choose Raw GPU. Our deploy page describes it as “a raw GPU with JupyterLab + CUDA pre-installed. SSH access, root on a dedicated container, install any Python/CUDA libs you want.”
  2. Pick a card. Cards listed on /pricing on 2026-11-01 include the RTX 5090 32G, RTX 4090 48G, RTX 4090 24G, RTX 3090 24G, A100 80G, A800 80G, H20 96G, V100 32G and RTX PRO 6000 96G. See current rates on /pricing. VRAM need depends on model, precision and workflow; see How much VRAM do I need to run an LLM?.
  3. Open a terminal in JupyterLab, or connect over SSH with your instance’s connection details:
ssh -p <port> <user>@<host>

Step 2: Check the environment first

nvidia-smi                                              # driver and "CUDA Version" in the header
nvidia-smi --query-gpu=name,compute_cap --format=csv    # card name and compute capability
nvcc --version                                          # CUDA toolkit, only needed for a source build
python3 --version                                       # bitsandbytes 0.50.2 needs 3.10 or newer
python3 -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"

Write down three values: compute_cap, torch.version.cuda and whether torch.cuda.is_available() printed True.

  • If torch.version.cuda prints None, or is_available() prints False, you have a CPU-only PyTorch (or no GPU visible). bitsandbytes will then load its CPU library and cannot run its CUDA code. Fix PyTorch first, in Step 3.
  • If import torch fails, PyTorch is not installed in this environment yet.

Our deploy page lists “JupyterLab + CUDA pre-installed” for Raw GPU but does not list the driver version, whether nvcc is on PATH, or whether PyTorch is pre-installed, so read them from the commands above on your own instance.

Step 3: Get a CUDA build of PyTorch that matches your card

Use a fresh virtual environment so you don’t break an existing one:

python3 -m venv ~/venvs/bnb && source ~/venvs/bnb/bin/activate
python -m pip install --upgrade pip

Then install PyTorch with the command from the PyTorch install page. On 2026-11-01 the current stable release is 2.15.0, and the download.pytorch.org indexes have Linux builds of 2.15.0 for CUDA 13.2 (cu132) and CUDA 13.4 (cu134). Pick a build whose CUDA version is not newer than the “CUDA Version” in your nvidia-smi header. For example:

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu132

Two cards need a closer look, based on the table above:

  • RTX 5090. NVIDIA lists the GeForce RTX 5090 at compute capability 12.0 (sm120). Only the bitsandbytes CUDA 12.8 and CUDA 13.x builds include sm120, so use a PyTorch build on CUDA 12.8 or newer. A current 2.15.0 build on cu132 or cu134 qualifies. Confirm with python -m bitsandbytes on the card (Step 5) before you rely on it.
  • V100 32G. NVIDIA’s legacy GPU list gives the V100 compute capability 7.0 (sm70). The bitsandbytes CUDA 13.x builds start at sm75, and LLM.int8() needs 7.5 or newer, so on a V100 pick a PyTorch build for CUDA 12.x and expect 8-bit optimizers and 4-bit (NF4/FP4) only. Check the PyTorch install page and its previous-versions list for a build on CUDA 12.x.

If your project pins a different PyTorch, our guide How to install a specific PyTorch CUDA version on a cloud GPU walks through it. Run the last line of Step 2 again and confirm torch.cuda.is_available() is True before you go on.

Step 4: Install bitsandbytes

In the same environment:

pip install bitsandbytes

If you will load models through Hugging Face Transformers, the Transformers docs install it together with the libraries it works with:

pip install --upgrade transformers accelerate bitsandbytes

There is nothing to compile: on Linux x86-64, pip downloads the prebuilt wheel (the docs give glibc 2.24 as the minimum).

Step 5: Verify the install on the GPU

bitsandbytes ships its own diagnostic command:

python -m bitsandbytes

According to the bitsandbytes 0.50.2 source, it prints the bitsandbytes version, platform, PyTorch and CUDA details and related package versions, then runs a small optimizer check on the GPU. Look for:

  • SUCCESS! at the end: the CUDA library loaded and ran.
  • WARNING: bitsandbytes is running as CPU-only!: your PyTorch has no CUDA, or bitsandbytes could not load a CUDA library. See the fix below.
  • A line like No prebuilt binary for CUDA 13.4, loading CUDA 13.2 instead. Set BNB_CUDA_VERSION to override.: this is a warning, not an error (see the next section).

The diagnostic output can include paths and environment details, and the tool itself warns you to remove sensitive information before you post it anywhere.

To check 4-bit loading in Transformers, the docs use BitsAndBytesConfig; replace the model ID with one you are allowed to use:

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

nf4_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4")
model = AutoModelForCausalLM.from_pretrained(
    "<your-model-id>",
    device_map="auto",
    quantization_config=nf4_config,
)

The Transformers docs recommend NF4 for training 4-bit base models, which is what QLoRA does. If you still need to download the model, see How to download a Hugging Face model on a cloud GPU.

Fix: “WARNING: bitsandbytes is running as CPU-only!”

python -m bitsandbytes prints this when it cannot run its CUDA code. Run the last line of Step 2 in the same environment. If torch.version.cuda prints None or torch.cuda.is_available() prints False, install a CUDA build of PyTorch (Step 3), reinstall bitsandbytes in that environment and run python -m bitsandbytes again. If PyTorch sees the GPU but the warning stays, read the library error above it and check the next two sections.

Fix: “No compatible CUDA library found” (CUDA version mismatch and BNB_CUDA_VERSION)

bitsandbytes reads torch.version.cuda and looks for a library named after it, for example 12.8 → libbitsandbytes_cuda128. In 0.50.2 the order is:

  1. An exact match.
  2. Otherwise, the closest library below PyTorch’s version with the same CUDA major version, with the warning shown in Step 5.
  3. Otherwise, the closest library above it with the same CUDA major version.

It never falls back across major versions. So a PyTorch on cu134 loads the CUDA 13.2 library and logs a warning, which matches the automatic fallback the troubleshooting docs describe.

If you see No compatible CUDA library found, the docs give two fixes: compile bitsandbytes from source (next section), or force a specific prebuilt library with an environment variable, digits only:

export BNB_CUDA_VERSION=132
python -m bitsandbytes

Only set it to a CUDA version the wheel actually ships (see the table under “What bitsandbytes needs”) and that your driver supports. If the cextension error message says CUDA VERSION MISMATCH, check whether an old BNB_CUDA_VERSION is still set in your shell or ~/.bashrc, and unset BNB_CUDA_VERSION.

Fix: “CUDA error: no kernel image is available for execution on the device”

The full message in bitsandbytes GitHub issues #1152 and #1937 is RuntimeError: CUDA error: no kernel image is available for execution on the device. The bitsandbytes troubleshooting docs cover it under “No kernel image available”: it means the CUDA library bitsandbytes loaded does not support your GPU, or your PyTorch CUDA version does not match. Check, in order:

  1. compute_cap from Step 2 against the table at the top. On an RTX 5090 (12.0), a PyTorch on CUDA 12.6 or older makes bitsandbytes load a library without sm120.
  2. Several CUDA versions on your paths. The docs say to look for them in the LD_LIBRARY_PATH, CUDA_HOME and PATH variables, including conda paths, and compare them with nvidia-smi:
printenv LD_LIBRARY_PATH CUDA_HOME PATH
ls -l ${CONDA_PREFIX}/lib/*cudart* 2>/dev/null

For a fatbinwrap error, the docs say the directory that contains cudart must be on LD_LIBRARY_PATH. If you use conda, see How to set up a conda environment on a cloud GPU.

Build from source (only if no prebuilt library fits)

  • Try the prebuilt wheel first: it covers CUDA 11.8 to 13.2, and BNB_CUDA_VERSION can force one of its libraries.
  • Fix CUDA path conflicts before you compile; the troubleshooting docs warn that a build can still fail if your paths point at several CUDA versions.
  • The installation docs list CMake 3.22.1 or newer, GCC 11 or newer and a CUDA Toolkit 11.8 or newer (nvcc from Step 2) as build requirements.
  • On a Raw GPU you have root, so you can install the build tools yourself; run long builds inside tmux or screen so they keep going if your SSH session drops.
  • After the build, run python -m bitsandbytes again and look for SUCCESS!.

Follow the steps in the official bitsandbytes installation docs.

Data on local instance storage is permanently deleted when you destroy the instance (see /terms), so keep your install commands in a script you can re-run on the next instance. Billed per minute, rounded up, with a 1-minute minimum; billing stops when you destroy the instance (see /terms).

Troubleshooting

Symptom What to check Source
WARNING: bitsandbytes is running as CPU-only! torch.version.cuda is None or torch.cuda.is_available() is False; install a CUDA build of PyTorch (Step 3) bitsandbytes diagnostics/main.py (0.50.2)
No prebuilt binary for CUDA X.Y, loading CUDA A.B instead. A warning; bitsandbytes loaded the closest library with the same CUDA major version bitsandbytes cextension.py (0.50.2), troubleshooting docs
No compatible CUDA library found No library with your CUDA major version; set BNB_CUDA_VERSION to a shipped version or build from source troubleshooting docs
CUDA VERSION MISMATCH A BNB_CUDA_VERSION override that does not match; unset it bitsandbytes cextension.py (0.50.2)
CUDA error: no kernel image is available for execution on the device The loaded library lacks your card’s architecture, or several CUDA versions are on your paths troubleshooting docs, installation docs
fatbinwrap Add the directory with cudart to LD_LIBRARY_PATH troubleshooting docs
LLM.int8() fails on an older card LLM.int8() needs compute capability 7.5 or newer installation docs

FAQ

How do I check that bitsandbytes is using the GPU?

Run python -m bitsandbytes in the same environment. It prints the PyTorch and CUDA details, runs a small check on the GPU and ends with SUCCESS! if the CUDA library works. If it prints WARNING: bitsandbytes is running as CPU-only!, your PyTorch has no CUDA support or no CUDA library could be loaded.

What does “WARNING: bitsandbytes is running as CPU-only!” mean?

Usually because PyTorch in that environment is a CPU-only build: torch.version.cuda prints None or torch.cuda.is_available() prints False. Install a CUDA build of PyTorch from the PyTorch install page, then reinstall bitsandbytes in the same environment and run python -m bitsandbytes again.

How do I fix “CUDA error: no kernel image is available for execution on the device” in bitsandbytes?

The bitsandbytes troubleshooting docs say this happens when the CUDA library bitsandbytes loaded does not support your GPU, or your PyTorch CUDA version does not match. Compare your card’s compute capability with the architectures in the bitsandbytes installation docs, use a PyTorch build whose CUDA library includes your card (CUDA 12.8 or newer for an RTX 5090), check the LD_LIBRARY_PATH, CUDA_HOME and PATH variables for several CUDA versions, then run python -m bitsandbytes again.

What does “No compatible CUDA library found” mean?

The bitsandbytes troubleshooting docs say it means no compatible pre-compiled library could be found at all. Their two fixes are to compile bitsandbytes from source, or to set BNB_CUDA_VERSION to force a specific pre-compiled version, digits only, for example 132 for CUDA 13.2.

What does BNB_CUDA_VERSION do?

It forces bitsandbytes to load the prebuilt library for a specific CUDA version instead of the one picked from torch.version.cuda. The value is digits only, for example 132 for CUDA 13.2. Use it only with a version the wheel ships and your driver supports, and unset it when you no longer need it.

Does bitsandbytes work on an RTX 5090?

The bitsandbytes installation docs list sm120, the architecture of the RTX 5090 (compute capability 12.0), for the CUDA 12.8 and CUDA 13.0 to 13.2 builds. Use a PyTorch build on CUDA 12.8 or newer and confirm with python -m bitsandbytes on the card before you rely on it.

Do I need to compile bitsandbytes from source?

Usually not. pip install bitsandbytes downloads a Linux x86-64 wheel with prebuilt libraries for CUDA 11.8 to 13.2. Build from source only if no library with your CUDA major version exists; the docs list CMake 3.22.1 or newer, GCC 11 or newer and a CUDA Toolkit 11.8 or newer.

Sources

All read on 2026-11-01.