How do I keep a training or inference job running after my SSH session disconnects on a cloud GPU?

Start the job inside tmux (or GNU screen), or launch it with nohup ... &, and send its output to a log file; the job then keeps running on the GPU after SSH drops, and you reconnect later with tmux attach or screen -r and check it with tail -f and nvidia-smi.

Updated 2026-10-27

This guide follows the tmux man page, the GNU Screen manual, the GNU Coreutils manual for nohup, the ssh_config man page and the Python command-line docs, all read on 2026-10-27. It is written for people who run their own training or inference scripts on a rented GPU over SSH.

Why the job stops when SSH drops

A command you start in a plain SSH shell belongs to that terminal session. When the connection closes, the session gets a hangup signal (SIGHUP) and processes tied to it usually end. Closing SSH does not stop the instance itself: the GPU keeps running, and billing continues until you destroy the instance (see /terms).

There are three common ways to keep the job alive:

Method What it does Reattach to the live terminal? Use it when Source
tmux Runs your shell inside a session that stays up when you detach or disconnect Yes: tmux attach -t <name> You want to watch the job, scroll back and type into it tmux(1) man page
GNU screen Same idea; detaches automatically on hangup by default Yes: screen -r <name> tmux is not installed and screen is GNU Screen manual
nohup ... & Starts one command with hangup signals ignored No; you read its log file A script that never needs keyboard input GNU Coreutils: nohup

Step 1: Deploy a Raw GPU and connect over SSH

  1. Open /deploy and choose Raw GPU. Our deploy page describes it as “a raw GPU with JupyterLab + CUDA pre-installed. SSH access, root on a dedicated container, install any Python/CUDA libs you want.”
  2. Pick a card. Cards listed on /pricing on 2026-10-27 include the RTX 5090 32G, RTX 4090 48G, RTX 4090 24G, RTX 3090 24G, A100 80G, A100 40G, A800 80G, H20 96G and RTX PRO 6000 96G. To size the card for a model, see How much VRAM do I need to run an LLM?.
  3. When the instance is running, find its SSH connection details (user, host and port) for that instance in the dashboard.
  4. Connect from your laptop:
ssh -p <port> <user>@<host>

Step 2: Check the environment

Before you start a long job, confirm the GPU and toolkit are visible in this shell:

nvidia-smi              # GPU name, driver version, VRAM, running processes
nvcc --version          # CUDA toolkit version
python3 --version
which tmux screen       # prints a path for each tool that is installed

If nvcc is not found, the CUDA toolkit is not on PATH in that shell; find where the toolkit is installed on your instance and add its bin folder to PATH.

If which prints nothing for the tool you want, it isn’t installed, so install it first. On a Debian or Ubuntu based system, for example (as root; add sudo in front if you are not root):

apt-get update && apt-get install -y tmux     # or: apt-get install -y screen

Run cat /etc/os-release to see which Linux distribution you are on, and use that distribution’s package manager if it is not Debian or Ubuntu based.

According to the official tmux docs, tmux “may be detached from a screen and continue running in the background, then later reattached” (tmux(1) man page, read 2026-10-27).

  1. Start a named session and move to the folder that holds your project:
tmux new -s train
cd <your-project-folder>
  1. Inside the session, start the job and write its output to a log file as well as the screen:
python3 -u train.py 2>&1 | tee -a train.log

According to the official Python docs, -u forces the stdout and stderr streams to be unbuffered (Python command line, read 2026-10-27), so the log keeps up with the screen. 2>&1 puts error messages in the same log, which is where you look if the job stopped while you were away.

  1. Detach: press Ctrl-b, then d. The tmux docs list Ctrl-b as the default prefix key and d as “Detach the current client.” You can now close the laptop or lose the connection; the session and the job keep running on the instance.

  2. Later, reconnect with ssh and reattach:

tmux ls                 # list sessions (alias of list-sessions)
tmux attach -t train    # reattach to the session named "train"

Useful extras from the same man page:

  • Ctrl-b then [ enters copy mode, so you can scroll back through output you missed. Press q to leave it.
  • tmux new -A -s train attaches to train if it exists and creates it if not, which is handy once you reconnect.
  • tmux kill-session -t train ends the session and everything running in it, so use it only after the job is done and its outputs are copied off.

Start the job from a shell inside the session, as above, rather than passing the command directly to tmux new. When the command passed to a window exits, the window closes by default (the man page’s remain-on-exit option changes this), and its scrollback goes with it; the tee log keeps a copy either way.

Step 3, alternative A: GNU screen

According to the official GNU Screen docs, autodetach is on by default, so when the connection drops, screen detaches and your programs keep running until you resume them with screen -r (GNU Screen manual: Detach, read 2026-10-27).

screen -S train                                   # new session named "train"
python3 -u train.py 2>&1 | tee -a train.log       # run the job inside it
# detach: Ctrl-a, then d
screen -ls                                        # list sessions after you reconnect
screen -r train                                   # resume the session

The screen manual also documents -L, which turns on output logging for the windows, with the default log file name screenlog.0; -Logfile "file" sets another name (Invoking Screen). The tee line above works the same in screen and in tmux.

Step 3, alternative B: nohup for a script that needs no input

According to the official GNU Coreutils docs, nohup “runs the given command with hangup signals ignored, so that the command can continue running in the background after you log out”, and it “does not automatically put the command it runs in the background; you must do that explicitly, by ending the command line with an ‘&’” (GNU Coreutils: nohup invocation, read 2026-10-27).

nohup python3 -u train.py > train.log 2>&1 &
echo $!                 # the process ID; note it down
tail -f train.log       # follow the log; Ctrl-c stops tail, not the job

If you leave out > train.log, the same docs say output that would go to the terminal is appended to nohup.out (or $HOME/nohup.out). nohup gives you no terminal to come back to, so a script that waits for keyboard input will not work this way; use tmux or screen for that.

Step 4: After you reconnect, check the job is still running

tmux ls                                  # or: screen -ls
pgrep -af train.py                       # is the Python process still alive?
tail -n 50 train.log                     # last lines of output; is it still growing?
nvidia-smi                               # is the GPU busy? it may also list the process

According to NVIDIA’s nvidia-smi documentation (read 2026-10-27), you can also ask for just the compute processes:

nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
watch -n 10 nvidia-smi                   # refresh the full view every 10 seconds; Ctrl-c to exit

nvidia-smi may list your job’s process, but inside a container the process table can also come back empty while the GPU shows memory in use. If it does, rely on pgrep and the log file instead.

Step 5: Write checkpoints as you go

tmux, screen and nohup only protect the job from an SSH disconnect. They do not bring back a job that crashes, and nothing survives once the instance is destroyed.

  • Checkpoint regularly. Have your training script write model and optimizer state to a checkpoint file every N steps (for example with torch.save in PyTorch) and accept a resume path, so a crash costs you minutes of work instead of the whole run.
  • Keep the log next to the checkpoints in your project folder on the instance.
  • Keep a positive balance. According to our /terms, you must maintain a positive wallet balance to keep instances running, and instances are automatically destroyed when your balance reaches zero.

Step 6: Copy outputs off, then destroy the instance

When the job finishes, copy what you need to your laptop or push it elsewhere before you destroy the instance. Run this on your laptop, with your instance’s SSH details:

rsync -avP -e "ssh -p <port>" <user>@<host>:<your-project-folder>/checkpoints/ ./checkpoints/
scp -P <port> <user>@<host>:<your-project-folder>/train.log .

Our /docs say a destroyed instance releases its GPU and you create a new instance to resume, and our /terms say data on the instance is permanently deleted when it is destroyed. Check the copied files open on your side, then destroy the instance from the dashboard. GPU time is billed per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance (see /terms).

Troubleshooting

Symptom What to check Source
tmux: command not found or screen: command not found Install it (Step 2), or use nohup Step 2
tmux attach says there is no session Run tmux ls; if the list is empty, the session ended, usually because the job exited. Read train.log for the last error tmux(1)
screen -r train does not resume the session Run screen -ls; sessions marked “attached” still have a terminal, so use screen -d -r train, which the manual describes as “Reattach a session and if necessary detach it first” Invoking Screen
The log stays empty while the job runs Python is buffering output; start it with python3 -u Python command line
A nohup.out file appeared You did not redirect output; add > train.log 2>&1 GNU Coreutils: nohup
Idle SSH sessions drop Add -o ServerAliveInterval=30 to your ssh command (the default is 0, which sends no keepalive messages). This keeps an idle session open; it does not keep a job alive when the connection really breaks ssh_config(5)

This guide is based on the official tool documentation above. We have not run this flow end to end on a CloudGPU instance, so check each step on your own instance.

FAQ

Does closing my laptop stop a training job on a cloud GPU?

It does if the job was started in a plain SSH shell, because the shell gets a hangup signal when the connection closes. Start the job inside tmux or screen, or with nohup and a trailing &, and it keeps running on the instance after SSH drops.

Should I use tmux, screen or nohup?

Use tmux or screen when you want to come back to the live terminal, scroll through output or type into the job. Use nohup for a script that needs no keyboard input and writes everything to a log file. All three let the job continue after an SSH disconnect.

How do I get back to my job after I reconnect?

SSH into the instance again, then run tmux ls and tmux attach -t train for tmux, or screen -ls and screen -r train for screen. For a nohup job, run pgrep -af train.py to see that it is alive and tail -f train.log to follow its output.

Are tmux and screen installed on a CloudGPU Raw GPU instance?

We have not checked whether either is pre-installed on our Raw GPU image. Run which tmux screen after you connect. If the one you want isn’t installed, install it first, for example with apt-get update && apt-get install -y tmux on a Debian or Ubuntu based system.

Does the GPU keep billing while I am disconnected?

Yes. Closing SSH or detaching tmux does not stop the instance, and GPU time is billed per minute, rounded up, with a 1-minute minimum. Billing stops when you destroy the instance; see /terms.

What happens to my checkpoints when I destroy the instance?

Data on the instance is permanently deleted when it is destroyed, and to resume you create a new instance. Copy checkpoints, logs and outputs to your laptop with rsync or scp, or push them elsewhere, before you destroy it.

Sources

Related: /deploy · /pricing · /docs · How much VRAM do I need to run an LLM?