Save Your Work Before You Destroy a Cloud GPU: Checkpoints, rsync and Hub Uploads (2026)

On CloudGPU, destroying an instance deletes its disk, and there is no persistent storage across instances yet, so anything you want to keep — checkpoints, LoRA adapters, generated images, logs — must be copied off the machine before you destroy it. An instance is also destroyed automatically if your balance runs out. This post is the checklist and the commands we use.

What happens to your files, and when

Event	What happens to the disk
You click Destroy	The instance stops, billing settles, the disk is deleted
Your balance cannot cover the next hour	The instance is destroyed and its disk deleted
You want to “pause” and resume tomorrow	Not available yet: stopping releases the GPU; you start a new instance next time
The JupyterLab tab or SSH session closes	Nothing — the instance keeps running (and billing) until destroyed

Because billing is per minute, the economical pattern is: work, copy outputs off, destroy. The minutes spent copying are billed like any other minute, so plan the copy rather than leaving the instance up “just in case”.

Pick a destination

Destination	Good for	Command family
Your own computer	Small to medium outputs, anything private	rsync / scp over SSH
Hugging Face Hub (private repo)	LoRA adapters, merged models, datasets you will reuse	huggingface-cli upload
S3-compatible object storage (S3, R2, B2…)	Large checkpoints, image batches, logs	rclone copy
JupyterLab download button	A handful of small files	Browser

Do not push large files through the instance’s public API endpoint: the tunnel is capped at 3 MB/s and is meant for API responses, not file transfer. Use SSH or an upload from the machine.

Outbound speed from the supplier data centre to a given destination varies. Test with a small file first and add the copy time to your budget.

Commands

1. Pull with rsync from your laptop (resumable)

# run on your laptop; user, host and port are on the instance page
rsync -avP -e "ssh -p <ssh-port>" <ssh-user>@<ssh-host>:/workspace/out/ ./run-2026-10-23/

-P shows progress and resumes partial files if the connection drops. Re-run the same command at the end; rsync only copies what changed.

2. Keep pulling during a long run

# laptop: pull new checkpoints every 10 minutes while training runs
while true; do
  rsync -aP -e "ssh -p <ssh-port>" <ssh-user>@<ssh-host>:/workspace/out/ ./run/ && date
  sleep 600
done

Pair it with frequent saves in your trainer (for LLaMA-Factory, save_steps in the YAML; for Hugging Face Trainer, save_steps and save_total_limit).

3. Push a LoRA adapter to a private Hugging Face repo

pip install -U "huggingface_hub[cli]"
huggingface-cli login            # paste a fine-grained write token
huggingface-cli upload <your-user>/qwen3-8b-qlora-run1 /workspace/out/qwen3-8b-qlora . --private

Use a fine-grained token scoped to the one repo and revoke it after the run.

4. Copy a large output folder to object storage

curl https://rclone.org/install.sh | bash          # you are root on the instance; or: apt-get install -y rclone
rclone config                                       # add your S3 / R2 / B2 remote
rclone copy /workspace/out remote:my-bucket/run1 --progress --transfers 8

5. Tar first when there are many small files

tar -C /workspace -czf /workspace/outputs.tgz out/

Thousands of small PNGs or log files copy much faster as one archive.

The pre-destroy checklist

#	Check	Why
1	Training or batch has finished (or you saved a checkpoint)	Partial runs are lost on destroy
2	Outputs copied and verified (file count, sizes, or sha256sum)	A failed copy looks like a successful one until you need it
3	Configs, scripts and the exact command line saved	So you can reproduce the run on a fresh instance
4	Environment captured (pip freeze > requirements.txt)	Next instance starts clean
5	Tokens you pasted on the machine revoked	Good hygiene, even though the disk is deleted
6	Click Destroy	Unused minutes of the pre-authorised hour return to your balance

Budget for the copy, and for the balance

Two numbers keep long runs safe:

• Balance buffer. An instance is destroyed when the balance runs out. For an 8-hour run on a card listed at $0.65/h (RTX 4090 48G, 2026-10-23 23:58 Beijing time), the run itself is $5.20; keep that plus a margin for setup and copying.
• Copy time. Estimate size ÷ measured upload speed, then add it to the job. On a card listed at $0.59/h, 10 extra minutes of copying cost about $0.10.

Stock and prices change during the day; check the Deploy page. Billing rules are on per-minute billing.

A note on where your data goes

If your data must not leave a particular jurisdiction, do not train on it here, and choose your copy destination with the same care.

FAQ

What happens to my files when I destroy a cloud GPU instance on CloudGPU? The disk is deleted. Copy anything you need off the machine before you destroy it.

Is there persistent storage between instances? Not yet. Stopping releases the GPU and the next session starts on a new instance, so keep checkpoints and outputs elsewhere.

What if my balance runs out during training? The instance is destroyed and its disk deleted. Keep enough balance for the whole run plus a margin, and copy checkpoints off as you go.

How do I download checkpoints from a rented GPU? Use rsync or scp over SSH, or upload from the machine to a model hub or object storage. Do not use the public API endpoint, which is capped at 3 MB/s.

Am I billed while I copy files off? Yes. Billing runs per minute until you destroy the instance, so plan the copy and destroy promptly afterwards.

CloudGPU (cloudgpu.app) is an independent service. It has no partnership with, authorization from, or endorsement by the model vendor. Product names are trademarks of their owners. Prices are list prices read on 2026-10-23 23:58 Beijing time (UTC+8); live prices and stock on /deploy take precedence.