How do I save and resume training checkpoints on a cloud GPU?

Cover: Resume training from a checkpoint, PyTorch and Hugging Face Trainer on a cloud GPU

Save a checkpoint that holds both the model and the optimizer state (in plain PyTorch, a dictionary passed to torch.save; with the Hugging Face Trainer, the checkpoint-<step> folders it writes every save_steps), then resume with load_state_dict or trainer.train(resume_from_checkpoint=True), and copy the checkpoints off the instance before you destroy it, because data on the instance is deleted when it is destroyed.

Updated 2026-10-28

This guide follows the official PyTorch tutorial Saving and Loading Models, the Hugging Face Trainer docs and the TrainingArguments reference in the transformers source, all read on 2026-10-28. It is for people who fine-tune or train on a rented NVIDIA GPU such as the RTX 5090 or the other NVIDIA cards listed on /pricing and want to stop, move or restart a run without losing progress.

Step 1: Deploy a Raw GPU and connect

On /deploy, pick an NVIDIA GPU and the Raw GPU option (JupyterLab + CUDA pre-installed, root SSH). Note the SSH host, port and user from your dashboard; you need them in Step 6. Connect over SSH or open a terminal in JupyterLab.

VRAM need depends on model, precision and workflow; see How much VRAM do I need to run an LLM?. Checkpointing does not change which card you need; it changes how much work you keep if a run stops.

Step 2: Check the environment first

nvidia-smi          # GPU name, driver version and the "CUDA Version" the driver supports
nvcc --version      # CUDA toolkit compiler, if installed
python3 --version
python3 -c "import torch; print(torch.__version__, torch.cuda.is_available())"
which tmux screen   # a terminal multiplexer keeps training alive if SSH drops

If torch.cuda.is_available() prints False or PyTorch is missing, see How to install a specific PyTorch CUDA version on a cloud GPU. Keep training code and its packages in one environment; see How to set up a conda environment on a cloud GPU. Resume on the same PyTorch and transformers versions you trained with, and write them down (pip freeze > requirements.txt) next to your checkpoints.

Step 3: Save and resume a checkpoint in plain PyTorch

According to the official PyTorch tutorial, a checkpoint used for resuming training “must save more than just the model’s state_dict”; the optimizer’s state_dict matters too, because it “contains buffers and parameters that are updated as the model trains”. The tutorial saves these items in one dictionary with torch.save and notes that a common PyTorch convention is the .tar file extension:

import os
import torch

CKPT = "checkpoints/last.tar"
os.makedirs("checkpoints", exist_ok=True)

model = torch.nn.Linear(10, 1).cuda()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
start_epoch = 0

# resume if a checkpoint exists
if os.path.exists(CKPT):
    checkpoint = torch.load(CKPT, weights_only=True)
    model.load_state_dict(checkpoint["model_state_dict"])
    optimizer.load_state_dict(checkpoint["optimizer_state_dict"])
    start_epoch = checkpoint["epoch"] + 1

model.train()
for epoch in range(start_epoch, 20):
    x = torch.randn(64, 10, device="cuda")
    loss = model(x).pow(2).mean()
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    torch.save({
        "epoch": epoch,
        "model_state_dict": model.state_dict(),
        "optimizer_state_dict": optimizer.state_dict(),
        "loss": loss.item(),
    }, CKPT)

Points from the same tutorial:

  • Call model.train() before you continue training, so dropout and batch normalization layers are in training mode. Use model.eval() only for inference.
  • Load across devices with map_location. For example, torch.load(CKPT, map_location="cpu", weights_only=True) loads a checkpoint saved on a GPU onto a machine without one.
  • Checkpoints are larger than the model. The tutorial says such a checkpoint “is often 2~3 times larger than the model alone”, so plan where you will keep them.

If you use a learning-rate scheduler, save scheduler.state_dict() in the same dictionary and load it back the same way.

Step 4: Save and resume with the Hugging Face Trainer

According to the official TrainingArguments reference, the Trainer saves checkpoints into output_dir with save_strategy="steps" and save_steps=500 by default, and save_total_limit “deletes older checkpoints in output_dir”. Each checkpoint is a folder named checkpoint-<step>:

from transformers import Trainer, TrainingArguments

args = TrainingArguments(
    output_dir="out",
    save_strategy="steps",
    save_steps=500,
    save_total_limit=2,   # keep only the two most recent checkpoint-* folders
)
trainer = Trainer(model=model, args=args, train_dataset=train_ds)

# first run
trainer.train()

# after an interruption: pick up the last checkpoint in out/
trainer.train(resume_from_checkpoint=True)

# or a specific folder
# trainer.train(resume_from_checkpoint="out/checkpoint-1000")

The Trainer docs say that with resume_from_checkpoint=True it loads “the last checkpoint in args.output_dir”, and that training resumes from the model, optimizer and scheduler states loaded there. Two settings to watch:

  • Do not set save_only_model=True if you want to resume. The reference says it saves only the model weights and “prevents resuming training from the checkpoint”.
  • Push checkpoints to the Hub as you go. With push_to_hub=True and hub_strategy="checkpoint", the reference says the latest checkpoint is also pushed to a last-checkpoint subfolder “for easy resuming”. On a new instance, download that folder (see How to download a Hugging Face model on a cloud GPU) and pass its local path to resume_from_checkpoint.

Step 5: Run training so it survives a dropped SSH session

Start the job inside tmux (or screen) so it keeps running if your SSH connection drops, then check on it from another window:

tmux new -s train
python train.py          # detach with Ctrl+B then D; reattach with: tmux attach -t train
ls -lt out/              # newest checkpoint-* folder first
nvidia-smi               # is the GPU still busy?

See How to check GPU utilization on a cloud GPU if the GPU looks idle. Keep your account balance positive while a run is going: according to /terms, instances are destroyed automatically when the balance reaches zero, and the checkpoints on the instance go with them.

Step 6: Copy checkpoints off the instance, then destroy it

According to /terms, data on the instance is permanently deleted when you destroy it, so copy the checkpoints you want to keep first, for example from your computer:

scp -P <ssh-port> -r <ssh-user>@<ssh-host>:~/out/checkpoint-1000 ./checkpoint-1000
scp -P <ssh-port> <ssh-user>@<ssh-host>:~/checkpoints/last.tar ./last.tar

Copy your requirements.txt and training config too. To continue later, deploy a new Raw GPU, rebuild the environment, upload the checkpoint folder back into the same path and run the resume command from Step 3 or Step 4. Billing stops when you destroy the instance; see /terms.

FAQ

What should a PyTorch checkpoint contain to resume training?

According to the official PyTorch tutorial, save more than the model’s state_dict: also save the optimizer’s state_dict, plus items such as the epoch you left off on and the latest training loss, in one dictionary with torch.save. Add the scheduler’s state_dict if you use one.

How do I resume a Hugging Face Trainer run from the last checkpoint?

Call trainer.train(resume_from_checkpoint=True) with the same output_dir. The official Trainer docs say this loads the last checkpoint in args.output_dir and resumes from the model, optimizer and scheduler states saved there. To pick one folder, pass its path, such as out/checkpoint-1000.

Why can’t I resume from my Trainer checkpoint?

Check that output_dir points at the folder with the checkpoint-* subfolders and that save_only_model was not set to True. The official TrainingArguments reference says save_only_model saves only the model weights and prevents resuming training from the checkpoint.

What happens to my checkpoints when I destroy the instance?

They are deleted. The CloudGPU terms say data on the instance is permanently deleted when the instance is destroyed, so copy the checkpoints you want to keep to your computer or push them to the Hugging Face Hub before you destroy it.

How is a Raw GPU billed while I train with checkpoints?

Billing stops when you destroy the instance. See https://cloudgpu.app/terms for the full billing terms.