How do I fix "CUDA out of memory" on a cloud GPU?

First check what is holding GPU memory with nvidia-smi and stop any leftover process, then make the job ask for less: a smaller batch (with gradient accumulation), a shorter sequence or context length, mixed precision and activation checkpointing; if the error says a lot of memory is “reserved but unallocated”, set PYTORCH_ALLOC_CONF=expandable_segments:True, and only move to a card with more VRAM if the job still doesn’t fit.

Updated 2026-10-27

This guide follows the official PyTorch 2.14 documentation, the Hugging Face Transformers training docs, the vLLM “Conserving Memory” page and NVIDIA’s nvidia-smi documentation, all read on 2026-10-27. It is written for people running their own PyTorch, Transformers or vLLM jobs on a rented GPU over SSH or JupyterLab.

What the error means

torch.OutOfMemoryError: CUDA out of memory (older code may show it as torch.cuda.OutOfMemoryError or RuntimeError: CUDA out of memory) means one allocation asked for more GPU memory than was free at that moment. The message usually has this shape:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate <X> GiB.
GPU 0 has a total capacity of <T> GiB of which <F> MiB is free.
Of the allocated memory <A> GiB is allocated by PyTorch, and <R> GiB is reserved by PyTorch but unallocated.

Read it before changing anything:

  • Total capacity is the card’s VRAM.
  • Allocated by PyTorch is memory your tensors were using at that moment.
  • Reserved but unallocated is memory PyTorch’s caching allocator holds but cannot hand out as one block. If this number is large, the problem is fragmentation (Step 5), not only size.
  • If free plus what your process holds is far below total capacity, something else on the GPU is using memory (Step 2).

Step 1: Check the environment

Run these first, in the terminal over SSH or in a JupyterLab terminal:

nvidia-smi
nvcc --version
python3 -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"
  • nvidia-smi shows the card, the driver, the total and used memory, and (where the system allows it) the processes on the GPU.
  • nvcc --version shows the CUDA toolkit compiler, if one is on your PATH. If it prints “command not found”, the toolkit may be installed outside your PATH; try ls /usr/local/cuda*/bin/nvcc.
  • The Python line confirms PyTorch can see the GPU. If import torch fails, install PyTorch with the command the official PyTorch install page gives for your setup; this guide does not say which version, if any, is pre-installed on our Raw GPU image.

To print only the memory numbers, NVIDIA’s nvidia-smi documentation describes the selective query option --query-gpu (run nvidia-smi --help-query-gpu for the field list):

nvidia-smi --query-gpu=name,memory.total,memory.used,memory.free --format=csv

Step 2: Stop whatever else is using the GPU

The most common surprise on a rented GPU is a second process you forgot: an earlier training run, a notebook kernel that still holds its tensors, or a model server you started and left running.

nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
ps -elf | grep python

NVIDIA’s documentation describes --query-compute-apps as the “List of currently active compute processes”. Inside a container, nvidia-smi may not list process names or IDs even when memory is in use, so also check ps. According to the official PyTorch FAQ, if GPU memory isn’t freed after Python quits, “it is very likely that some Python subprocesses are still alive. You may find them via ps -elf | grep python and manually kill them with kill -9 [pid].”

In JupyterLab, shut down kernels you no longer need; the official JupyterLab docs say kernels and terminals can be shut down from the Running panel. A running kernel keeps its tensors in GPU memory until it stops.

Step 3: Stop your own code from holding memory

According to the official PyTorch FAQ:

  • Don’t accumulate history across your training loop. A running total like total_loss += loss keeps the autograd graph of every step alive. The FAQ’s fix is total_loss += float(loss).
  • Don’t hold onto tensors you don’t need. Free a reference with del x; a tensor stored on an object lives until the object goes away.

For evaluation and inference, run the model under torch.no_grad() or torch.inference_mode(). According to the official PyTorch autograd notes, computations in these modes “are not recorded in the backward graph”, so no activations are kept for a backward pass you will never run.

Note that torch.cuda.empty_cache() is not a fix on its own. The official PyTorch CUDA semantics page says it “releases all unused cached memory from PyTorch so that those can be used by other GPU applications. However, the occupied GPU memory by tensors will not be freed so it can not increase the amount of GPU memory available for PyTorch.”

Step 4: Make the job ask for less memory

Change one thing at a time and rerun:

  1. Smaller batch, same effective batch. Lower the per-step batch size and add gradient accumulation. According to the official Hugging Face Transformers docs, “Large batches produce large activations that exhaust GPU memory. Gradient accumulation lets you train with a larger effective batch size by spreading gradient computation across multiple mini-batches.” With Trainer, set per_device_train_batch_size lower and gradient_accumulation_steps higher. In plain PyTorch, call optimizer.step() and optimizer.zero_grad() only every N batches, and divide the loss by N.
  2. Shorter sequences or context. The Hugging Face gradient checkpointing docs note that “activations scale with batch size and sequence length.” Truncate or bucket long inputs, or lower max_length / max_seq_length in your training config.
  3. Mixed precision. According to the official torch.amp docs, mixed precision runs some operations in torch.float32 and others in a lower-precision type, torch.float16 or torch.bfloat16. Wrap the forward pass in with torch.autocast(device_type="cuda", dtype=torch.bfloat16):, or set bf16=True in Transformers TrainingArguments. Check that loss and output quality stay acceptable.
  4. Activation checkpointing. The official torch.utils.checkpoint docs describe it as “a technique that trades compute for memory”: activations are recomputed in the backward pass instead of kept. In Transformers, set gradient_checkpointing=True (see the gradient checkpointing docs). Training steps take longer.
  5. LoRA or QLoRA for fine-tuning. Instead of updating every weight, train a small adapter with LoRA or QLoRA; the official Hugging Face PEFT docs say PEFT methods “only fine-tune a small number of (extra) model parameters”.

Step 5: Fix fragmentation with expandable_segments

If the error shows a large “reserved by PyTorch but unallocated” number, the memory exists but is split into pieces too small for the request. According to the official PyTorch CUDA semantics page, the caching allocator “can be controlled via the environment variable PYTORCH_ALLOC_CONF. PYTORCH_CUDA_ALLOC_CONF is its alias and is provided only for backward compatibility.” The expandable_segments option (marked experimental, default False) lets the allocator grow one segment instead of creating many fixed-size ones, which helps when allocation sizes keep changing, such as with a changing batch size.

Set it before Python starts:

export PYTORCH_ALLOC_CONF=expandable_segments:True
python3 train.py

If your PyTorch version is older and ignores PYTORCH_ALLOC_CONF, use the alias name PYTORCH_CUDA_ALLOC_CONF with the same value.

Step 6: Find out exactly what uses the memory

If you still can’t tell where the memory goes, ask PyTorch:

import torch
print(torch.cuda.memory_summary())

The official memory_summary docs describe it as “a human-readable printout of the current memory allocator statistics”, useful “periodically during training, or when handling out-of-memory exceptions”.

For a full timeline, the official Understanding CUDA Memory Usage page shows how to record a snapshot:

import torch
torch.cuda.memory._record_memory_history()
run_your_code()  # your training or inference steps
torch.cuda.memory._dump_snapshot("my_snapshot.pickle")

Copy my_snapshot.pickle to your laptop (for example with scp -P <port> <user>@<host>:~/my_snapshot.pickle .) and drag it into https://pytorch.org/memory_viz. PyTorch’s page says the visualizer “runs locally on your computer”.

Step 7: Out of memory when serving with vLLM

According to the official vLLM “Conserving Memory” docs:

  • “You can further reduce memory usage by limiting the context length of the model (max_model_len option) and the maximum batch size (max_num_seqs option).”
  • “Quantized models take less memory at the cost of lower precision.”
  • CUDA graphs “take up extra memory in the GPU”; you can turn graph capture off “completely via the enforce_eager flag”. The same page presents this as a trade-off against memory usage.

To run vLLM on a rented card, use the Raw GPU option described under “Doing this on CloudGPU” below.

Step 8: If it still doesn’t fit, use a card with more VRAM

If the model weights alone are larger than the card, no setting above will make it fit. How much VRAM a job needs depends on the model, precision and workflow; see How much VRAM do I need to run an LLM?.

Before you switch cards, back up what you need. Our /terms say that when an instance is destroyed, data on local instance storage is permanently deleted, so copy checkpoints, configs and logs to your laptop or your own storage first, for example:

scp -P <port> -r <user>@<host>:~/outputs ./outputs

Then start the new instance and copy them back.

Doing this on CloudGPU

Our /pricing page lists, on 2026-10-27, cards including the RTX 4090 24G, RTX 5090 32G, RTX 4090 48G, A100 80G and RTX PRO 6000 96G (VRAM as labelled on /pricing); see current rates there. For your own PyTorch, Transformers or vLLM environment, use the Raw GPU option on /deploy, which our deploy page describes as “a raw GPU with JupyterLab + CUDA pre-installed. SSH access, root on a dedicated container, install any Python/CUDA libs you want.”

GPU time is billed per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance; see /terms. You can top up by card (Visa or Mastercard), Alipay, Google Pay or Apple Pay, or with USDT, as listed on /pricing.

FAQ

What does “CUDA out of memory” mean?

It means one allocation asked for more GPU memory than was free at that moment. The error message lists the card’s total capacity, how much is free, how much PyTorch has allocated, and how much PyTorch has reserved but not allocated.

Does torch.cuda.empty_cache() fix CUDA out of memory?

Not on its own. The official PyTorch docs say it releases unused cached memory so other GPU applications can use it, but memory held by tensors is not freed, so it cannot increase the memory available to PyTorch. Delete tensors you no longer need first.

What does PYTORCH_ALLOC_CONF=expandable_segments:True do?

It tells PyTorch’s caching allocator to grow one memory segment instead of creating many fixed-size ones, which helps when the error shows a lot of memory reserved but unallocated. PyTorch marks the option experimental and off by default. Set it before Python starts; older versions read the alias PYTORCH_CUDA_ALLOC_CONF.

Why is GPU memory still used after my script stopped?

Usually another process still holds it, such as a leftover Python process, a running notebook kernel or a model server. Check with nvidia-smi and ps -elf | grep python, then stop the process or shut down the kernel.

When should I move to a GPU with more VRAM?

When the job still runs out of memory after you stop other processes, lower the batch size or context length, use mixed precision and checkpointing, and fix fragmentation, or when the model weights alone are larger than the card. VRAM need depends on the model, precision and workflow. Back up your checkpoints before you destroy the instance, because data on the instance is deleted.

How do I stop paying when I am done?

Destroy the instance. GPU time is billed per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance. See the terms page for details.

Sources

All read on 2026-10-27.