How do I install FlashAttention on a cloud GPU?

Run pip install flash-attn --no-build-isolation inside the environment that already has PyTorch; it downloads a prebuilt wheel only if one matches your exact Python, PyTorch, CUDA major version and C++ ABI, and on 2026-10-27 the latest release (2.8.3.post2) has Linux wheels only for PyTorch 2.4 to 2.8 on CUDA 12, so with a newer PyTorch pip compiles it from source, which needs nvcc from a CUDA toolkit whose major version matches torch.version.cuda, plus ninja.

Updated 2026-10-27

This guide follows the official flash-attention README and setup.py, the flash-attention release list on GitHub, the PyTorch install page and SDPA docs, the Hugging Face Transformers docs and the NVIDIA CUDA installation guide, all read on 2026-10-27. It is for people who hit nvcc was not found, CUDA_HOME environment variable is not set or a failing source build when a training repo asks for flash-attn on a rented GPU.

Do you need the flash-attn package at all?

Often not. According to the PyTorch docs, torch.nn.functional.scaled_dot_product_attention (SDPA) has three implementations, one of them FlashAttention-2, and “All implementations are enabled by default”; PyTorch picks one based on the inputs (PyTorch SDPA docs, read 2026-10-27). Many model libraries use SDPA when flash-attn is not installed.

According to the Hugging Face Transformers docs, you can also load a FlashAttention-2 kernel from the Hub “without installing the package” by passing attn_implementation="kernels-community/flash-attn2" to from_pretrained() (Transformers GPU inference docs, read 2026-10-27). That needs the kernels package (pip install kernels), which, per its PyPI page, requires torch>=2.5 and CUDA.

Install flash-attn itself when your code imports flash_attn directly, or when a repo’s requirements or attn_implementation="flash_attention_2" ask for it.

Step 1: Deploy a Raw GPU and connect

Open /deploy and choose Raw GPU. Our deploy page describes it as “a raw GPU with JupyterLab + CUDA pre-installed. SSH access, root on a dedicated container, install any Python/CUDA libs you want.”
Pick a card. Cards listed on /pricing on 2026-10-27 include the RTX 5090 32G, RTX 4090 48G, RTX 4090 24G, RTX 3090 24G, A100 80G, A800 80G, H20 96G and RTX PRO 6000 96G. See current rates on /pricing.
Open a terminal in JupyterLab, or connect over SSH with your instance’s connection details:

ssh -p <port> <user>@<host>

Step 2: Check the environment first

nvidia-smi                                                  # driver and "CUDA Version" in the header
nvidia-smi --query-gpu=name,compute_cap --format=csv        # card name and compute capability
nvcc --version                                              # CUDA toolkit (compiler), if one is on PATH
python3 --version
python3 -c "import torch, sys; print(torch.__version__, torch.version.cuda, torch._C._GLIBCXX_USE_CXX11_ABI, sys.version_info[:2])"
nproc; free -g                                              # CPU cores and RAM, for MAX_JOBS in Step 5

Write down four values: the PyTorch version, torch.version.cuda, the ABI flag (True or False) and the Python version. Together they decide whether a prebuilt wheel exists (Step 3). Also note the compute capability: flash-attention 2 checks at runtime that the GPU is Ampere or newer (“FlashAttention only supports Ampere GPUs or newer.” in its source).

Our deploy page lists “JupyterLab + CUDA pre-installed” for Raw GPU but does not list the driver version, whether nvcc is on PATH, or whether PyTorch is pre-installed, so read them from the commands above on your own instance. If import torch fails, install PyTorch first (the current pip commands are on the PyTorch install page).

Step 3: See whether a prebuilt wheel matches

According to the flash-attention setup.py, the installer first looks for a wheel on the GitHub release page named like this, and only compiles if that file does not exist:

flash_attn-<version>+cu<CUDA major>torch<PyTorch major.minor>cxx11abi<TRUE|FALSE>-cp<XY>-cp<XY>-linux_x86_64.whl

On 2026-10-27 the latest release, v2.8.3.post2, has Linux x86_64 wheels for CUDA 12 builds of PyTorch 2.4, 2.5, 2.6, 2.7 and 2.8 only. The current stable PyTorch on the PyTorch install page is 2.14.1, with CUDA 12.6 and CUDA 13.2 builds, so with a current PyTorch there is no matching official wheel and pip goes to a source build.

You have three options:

Your situation	Option
Your project can run on PyTorch 2.8 (CUDA 12 build) and your card is Ampere, Ada or Hopper	Pin PyTorch to 2.8 so the prebuilt wheel matches (Step 4)
You need a newer PyTorch, or an RTX 5090	Build from source (Step 5)
You only need FlashAttention inside Transformers or SDPA	Skip `flash-attn` (see the section above)

Step 4 (option A): Pin PyTorch so a prebuilt wheel matches

Use a fresh virtual environment so you don’t break an existing one:

python3 -m venv ~/venvs/fa2 && source ~/venvs/fa2/bin/activate
python -m pip install --upgrade pip
pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu126
pip install packaging psutil ninja
pip install flash-attn==2.8.3.post2 --no-build-isolation

Both torch-2.8.0+cu126 and torchvision-0.23.0+cu126 are on the download.pytorch.org cu126 index on 2026-10-27. Use a CUDA 12 index whose CUDA version is not newer than the CUDA Version in your nvidia-smi header. The flash-attention README lists Ampere, Ada and Hopper GPUs (for example A100, RTX 3090, RTX 4090, H100) for FlashAttention-2 with CUDA. On our list, that matches the RTX 3090, RTX 4090, A100, A800 and H20 (labelled Hopper on /pricing); check compute_cap from Step 2 to be sure.

If pip prints that it is building a wheel for flash-attn instead of downloading one, the combination did not match: compare your four values from Step 2 with the file names on the release page.

The latest prebuilt flash-attn wheels only go up to torch 2.8; for a newer torch, use Step 5 (option B): Build from source.

Step 5 (option B): Build from source

According to the flash-attention README, a source build needs a CUDA toolkit (CUDA 12.0 or newer), PyTorch 2.2 or newer, the packaging, psutil and ninja Python packages, and Linux.

5a. Make sure nvcc matches PyTorch’s CUDA major version. PyTorch’s extension builder stops with “The detected CUDA version … mismatches the version that was used to compile PyTorch” if the toolkit’s major version differs from torch.version.cuda; a minor-version difference only gives a warning (torch/utils/cpp_extension.py). If nvcc --version in Step 2 printed nothing or a different major version, install a matching toolkit. The NVIDIA CUDA installation guide documents conda packages for this, recommends a dedicated conda environment, and lists cuda-nvcc as one of the packages; previous releases use a label:

# example for the cu132 PyTorch build (torch.version.cuda 13.2); pick the label that matches yours
conda install -y cuda -c nvidia/label/cuda-13.2.0
which nvcc && nvcc --version
export CUDA_HOME=$(dirname $(dirname $(which nvcc)))

Pick a toolkit that is not newer than the CUDA Version in your nvidia-smi header. If you use conda for this, our conda guide shows how to set up Miniforge.

5b. Install the build tools and check ninja. The README says to check that ninja --version followed by echo $? returns exit code 0, and to reinstall ninja if it doesn’t; without a working ninja, the build does not use multiple CPU cores.

pip install packaging psutil ninja
ninja --version; echo $?

5c. Build with a job limit. The README warns that on a machine with many CPU cores but limited RAM, ninja can start too many parallel compile jobs and run out of memory, and shows MAX_JOBS to cap them. Pick a value from the nproc and free -g output in Step 2:

MAX_JOBS=4 pip install flash-attn --no-build-isolation -v

--no-build-isolation lets the build see the PyTorch you already installed. Run the build inside tmux or screen so it keeps going if your SSH session drops.

RTX 5090. Our /pricing page labels the RTX 5090 32G as Blackwell. The setup.py in the 2.8.3.post2 source package compiles for compute capabilities 8.0, 9.0, 10.0 and 12.0 by default, and adds 10.0 and 12.0 only when the toolkit is CUDA 12.8 or newer; the README’s FlashAttention-2 support list names Ampere, Ada and Hopper only. So on an RTX 5090 use a CUDA 12.8 or newer toolkit (with a matching PyTorch build), and check it yourself before you rely on it: run the Step 6 script and confirm that import flash_attn works, the capability prints as (12, 0), the last line reads ran on cuda:0, and there is no “no kernel image” error.

FlashAttention-3 and -4. According to the flash-attention README, FlashAttention-3 and FlashAttention-4 currently target data-center GPUs such as the H100, H800 and B200. This guide uses FlashAttention-2.

Step 6: Verify the install on the GPU

python - <<'PY'
import torch, flash_attn
from flash_attn import flash_attn_func
print("flash_attn", flash_attn.__version__, "| torch", torch.__version__, "CUDA", torch.version.cuda)
print("device:", torch.cuda.get_device_name(0), "capability:", torch.cuda.get_device_capability(0))
q = torch.randn(1, 128, 8, 64, device="cuda", dtype=torch.float16)
out = flash_attn_func(q, q, q, causal=True)
print("ran on", out.device, "shape", tuple(out.shape))
PY

Look for: a flash_attn version, your card’s name, and ran on cuda:0 on the last line. flash_attn_func takes tensors shaped (batch, sequence length, heads, head dimension) in fp16 or bf16. If you see undefined symbol on import, the wheel was built for a different PyTorch: reinstall flash-attn after PyTorch, in the same environment.

In Transformers, then load the model with attn_implementation="flash_attention_2", as in the Transformers docs.

Step 7: Keep the wheel for the next instance

Data on local instance storage is permanently deleted when you destroy the instance (see /terms), and that includes a wheel you just compiled. Build it once as a file and copy it off:

MAX_JOBS=4 pip wheel flash-attn --no-build-isolation --no-deps -w ~/wheelhouse
ls ~/wheelhouse

From your own computer:

scp -P <port> <user>@<host>:~/wheelhouse/*.whl .

On the next instance, upload it and run pip install ./flash_attn-*.whl. It only works if PyTorch version, CUDA major version, Python version and ABI flag are the same as on the machine that built it, so keep a copy of your Step 2 output next to the file.

Billing is per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance (see /terms). A saved wheel means you don’t need to rebuild on the next instance.

Related: if you use ComfyUI with 4-bit diffusion models, see How do I install Nunchaku on a cloud GPU?

Troubleshooting

Symptom	What to check	Source
`nvcc was not found` or `CUDA_HOME environment variable is not set`	No matching wheel, so pip is compiling. Install a CUDA toolkit with the same major version as `torch.version.cuda` and set `CUDA_HOME` (Step 5a), or pin PyTorch so a wheel matches (Step 4)	flash-attention issue #1736 (“Error when installing flash-attn with uv (CUDA_HOME environment variable is not set.)”: a source build printed “nvcc was not found” and stopped with this error), `torch/utils/cpp_extension.py`
`The detected CUDA version ... mismatches the version that was used to compile PyTorch`	Toolkit major version differs from `torch.version.cuda`; install the matching toolkit	`torch/utils/cpp_extension.py`
`ModuleNotFoundError: No module named 'torch'` during install	Install PyTorch first and keep `--no-build-isolation`	flash-attention README
Build uses only one CPU core	`ninja --version; echo $?` must return 0; reinstall ninja	flash-attention README
Build is killed or the machine stops responding	Too many parallel jobs for the RAM; lower `MAX_JOBS`	flash-attention README
`undefined symbol` when importing `flash_attn`	Wheel built for another PyTorch; reinstall `flash-attn` after PyTorch in the same environment	flash-attention releases
`FlashAttention only supports Ampere GPUs or newer.`	The GPU’s compute capability is below 8.0; use SDPA instead	flash-attention source (`csrc/flash_attn/flash_api.cpp`)
“no kernel image is available” on an RTX 5090	The build lacks compute capability 12.0; rebuild with a CUDA 12.8 or newer toolkit and a matching PyTorch	flash-attention `setup.py`

Which card fits your model is a separate question; see How much VRAM do I need to run an LLM?.

FAQ

Why does pip install flash-attn try to compile instead of downloading a wheel?

Because no prebuilt wheel matches your exact combination of flash-attn version, PyTorch version, CUDA major version, Python version and C++ ABI. On 2026-10-27 the latest release, 2.8.3.post2, has Linux wheels only for CUDA 12 builds of PyTorch 2.4 to 2.8, so a newer PyTorch makes pip build from source.

Do I need nvcc to install flash-attn?

Only for a source build. A matching prebuilt wheel installs without a compiler. A source build needs nvcc from a CUDA toolkit whose major version matches torch.version.cuda, plus the packaging, psutil and ninja Python packages.

How do I stop the flash-attn build from running out of memory?

Limit the number of parallel compile jobs with the MAX_JOBS environment variable, for example MAX_JOBS=4 pip install flash-attn --no-build-isolation, as the flash-attention README shows. Pick a lower value if the build is still killed.

Does flash-attn work on an RTX 5090?

The 2.8.3.post2 source package compiles for compute capability 12.0 when the CUDA toolkit is 12.8 or newer, but the README’s FlashAttention-2 support list names only Ampere, Ada and Hopper GPUs. Build with a CUDA 12.8 or newer toolkit and a matching PyTorch, then run a short test on the GPU before you rely on it.

Can I use FlashAttention without installing the flash-attn package?

Yes. PyTorch’s scaled_dot_product_attention includes a FlashAttention-2 implementation that is enabled by default, and the Transformers docs show loading a FlashAttention-2 kernel from the Hub with attn_implementation set to kernels-community/flash-attn2 after pip install kernels.