How do I train a Wan 2.2 LoRA with musubi-tuner on a cloud GPU?

Cover: Train a Wan 2.2 LoRA with musubi-tuner on a cloud GPU

Deploy a Raw GPU instance, install PyTorch and kohya-ss musubi-tuner in a Python virtual environment, download the Wan 2.2 14B fp16 high-noise and low-noise DiT files plus the T5 encoder and the Wan2.1 VAE, cache latents and text encoder outputs for your captioned clips, then run wan_train_network.py with --task t2v-A14B.

Updated 2026-10-30

This guide follows the official musubi-tuner README, Wan docs, dataset config docs and block swap docs, all read on 2026-10-30. It pins musubi-tuner v0.3.6, the latest release on the musubi-tuner releases page on that date (published 2026-09-27). The README states that the musubi-tuner code is licensed under Apache-2.0, and the Wan 2.2 repository ships an Apache 2.0 LICENSE.txt. The official Wan docs mark Wan training as experimental and say that musubi-tuner supports the Wan 2.2 14B models only.

Step 1: Deploy a Raw GPU and connect

On /deploy, pick a GPU such as the RTX 5090 32G or another card listed on /pricing, and choose the Raw GPU option (JupyterLab + CUDA pre-installed). Connect over SSH, or open a terminal in JupyterLab. VRAM need depends on model, precision and workflow; see How much VRAM do I need to run an LLM? and the official hardware text in the “Official hardware notes” section below before you choose a card. For other LoRA work on CloudGPU, see /rent-gpu-for-lora-fine-tuning.

Step 2: Check the environment first

nvidia-smi          # GPU name, driver version, and the "CUDA Version" the driver supports
nvcc --version      # CUDA toolkit compiler (PyTorch brings its own CUDA runtime; note the version anyway)
python3 --version   # musubi-tuner needs Python 3.10 or newer
free -h             # main RAM on the instance (see the official hardware notes)
which git tmux rsync

Write down the GPU name and the CUDA Version at the top right of the nvidia-smi output; Step 3 picks the PyTorch build from it. If git, tmux or rsync is missing, install it with your distribution’s package manager (on Debian/Ubuntu: sudo apt-get install -y git tmux rsync python3-venv; drop sudo if you are root).

Step 3: Install PyTorch and musubi-tuner in tmux

Run the install and the training inside tmux so they keep running if your SSH connection drops (see also How to keep a job running after SSH disconnects):

tmux new -s musubi
git clone --branch v0.3.6 https://github.com/kohya-ss/musubi-tuner.git ~/musubi-tuner
cd ~/musubi-tuner
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
pip install -e .
pip install ascii-magic matplotlib tensorboard prompt-toolkit

According to the official README, musubi-tuner needs PyTorch 2.6.0 or later, and you install the PyTorch build that matches your CUDA version before pip install -e . (the README example uses cu124). Replace cu128 with the build that fits the CUDA Version from Step 2: the musubi-tuner pyproject.toml at v0.3.6 lists cu124, cu128, cu130 and cu132 options. The last pip install line is optional in the README; it adds TensorBoard logging and the training tools’ extras. Detach from tmux with Ctrl-b then d; reattach with tmux attach -t musubi.

Check that PyTorch sees the GPU:

python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available(), torch.cuda.get_device_name(0), torch.cuda.get_arch_list())"

If this prints False or an error, compare torch.version.cuda with the CUDA Version from Step 2. On an RTX 5090, check that sm_120 is in the arch list; if it is not, reinstall PyTorch from a newer CUDA index.

Step 4: Configure Accelerate for one GPU

accelerate config

The official README gives these answers for a single-GPU machine:

  • In which compute environment are you running?: This machine
  • Which type of machine are you using?: No distributed training
  • Do you want to run your training on CPU only (even if a GPU / Apple Silicon / Ascend NPU device is available)?[yes/NO]: NO
  • Do you wish to optimize your script with torch dynamo?[yes/NO]: NO
  • Do you want to use DeepSpeed? [yes/NO]: NO
  • What GPU(s) (by id) should be used for training on this machine as a comma-seperated list? [all]: all
  • Would you like to enable numa efficiency? (Currently only supported on NVIDIA hardware). [yes/NO]: NO
  • Do you wish to use mixed precision?: bf16

The README notes that if you see ValueError: fp16 mixed precision requires a GPU, answer 0 to the GPU question. The launch command in Step 8 passes --mixed_precision explicitly, so the training precision follows that flag.

Step 5: Download the Wan 2.2 model files

According to the official Wan docs, musubi-tuner trains Wan 2.2 with the 14B fp16 DiT files from the Comfy-Org repackaged repository (fp8_scaled files are not supported), the models_t5_umt5-xxl-enc-bf16.pth T5 encoder, and the Wan2.1 VAE (Wan2.1_VAE.pth; the Wan2.2 VAE is for the 5B model, which musubi-tuner does not support). Wan 2.2 needs no CLIP model. The hf command comes with the huggingface-hub package that musubi-tuner installs:

mkdir -p ~/models/wan22
hf download Comfy-Org/Wan_2.2_ComfyUI_Repackaged \
  split_files/diffusion_models/wan2.2_t2v_high_noise_14B_fp16.safetensors \
  split_files/diffusion_models/wan2.2_t2v_low_noise_14B_fp16.safetensors \
  --local-dir ~/models/wan22
hf download Wan-AI/Wan2.1-I2V-14B-720P \
  models_t5_umt5-xxl-enc-bf16.pth Wan2.1_VAE.pth \
  --local-dir ~/models/wan22

For image-to-video training, download the wan2.2_i2v_high_noise_14B_fp16.safetensors and wan2.2_i2v_low_noise_14B_fp16.safetensors files from the same split_files/diffusion_models folder instead. Check each model card’s licence before you use the outputs; the Wan 2.2 repository lists Apache 2.0.

Step 6: Prepare the dataset and its TOML file

Put your clips and captions in one folder. According to the official dataset config docs, each caption is a .txt file with the same name as its video (clip001.mp4 → clip001.txt). Use only clips you have the right to train on.

mkdir -p ~/data/videos ~/data/cache
# copy your clips and .txt captions into ~/data/videos, for example from your computer:
# rsync -avP -e "ssh -p <ssh-port>" ./videos/ <user>@<instance-host>:~/data/videos/

Create ~/data/dataset.toml, based on the video sample in the official docs:

[general]
resolution = [960, 544]
caption_extension = ".txt"
batch_size = 1
enable_bucket = true
bucket_no_upscale = false

[[datasets]]
video_directory = "/root/data/videos"
cache_directory = "/root/data/cache"
target_frames = [1, 25, 45]
frame_extraction = "head"
source_fps = 30.0

Change /root to your home folder (echo $HOME). The official docs say that for Wan2.1 each target_frames value must be N*4+1 (1, 5, 9, 13…), suggest 16fps clips for Wan2.1, and use source_fps to resample clips recorded at another frame rate. Each dataset needs its own cache_directory. To check what the trainer will read, the docs offer --debug_mode video on the caching script.

Step 7: Cache latents and text encoder outputs

cd ~/musubi-tuner && source .venv/bin/activate
python src/musubi_tuner/wan_cache_latents.py \
  --dataset_config ~/data/dataset.toml \
  --vae ~/models/wan22/Wan2.1_VAE.pth
python src/musubi_tuner/wan_cache_text_encoder_outputs.py \
  --dataset_config ~/data/dataset.toml \
  --t5 ~/models/wan22/models_t5_umt5-xxl-enc-bf16.pth \
  --batch_size 16

According to the official Wan docs, add --i2v to the latent caching command for image-to-video training, --vae_cache_cpu to reduce VRAM use while caching latents, and --fp8_t5 to the text encoder command when VRAM is short.

Step 8: Train the LoRA on both experts

Wan 2.2 A14B uses a high-noise expert and a low-noise expert. According to the official Wan docs, passing the low-noise file to --dit and the high-noise file to --dit_high_noise trains one LoRA across both; --timestep_boundary defaults to 0.875 for T2V and 0.9 for I2V. The docs’ dtype table pairs an fp16 model with --mixed_precision fp16 or --fp8_base, so this command uses fp16. The other values come from the docs’ example command; the docs say these settings need experimentation.

cd ~/musubi-tuner && source .venv/bin/activate
accelerate launch --num_cpu_threads_per_process 1 --mixed_precision fp16 \
  src/musubi_tuner/wan_train_network.py \
  --task t2v-A14B \
  --dit ~/models/wan22/split_files/diffusion_models/wan2.2_t2v_low_noise_14B_fp16.safetensors \
  --dit_high_noise ~/models/wan22/split_files/diffusion_models/wan2.2_t2v_high_noise_14B_fp16.safetensors \
  --dataset_config ~/data/dataset.toml \
  --sdpa --mixed_precision fp16 --fp8_base \
  --optimizer_type adamw8bit --learning_rate 2e-4 --gradient_checkpointing \
  --max_data_loader_n_workers 2 --persistent_data_loader_workers \
  --network_module networks.lora_wan --network_dim 32 \
  --timestep_sampling shift --discrete_flow_shift 3.0 \
  --max_train_epochs 16 --save_every_n_epochs 1 --seed 42 \
  --output_dir ~/output --output_name my-wan22-lora

Options from the official docs for lower VRAM use:

  • --blocks_to_swap N keeps only part of the model on the GPU and moves N blocks to main RAM. The block swap docs say a larger N saves more VRAM, uses more main RAM, and may slow training.
  • --offload_inactive_dit moves the expert that is not in use to the CPU. The Wan docs say it does not work together with --blocks_to_swap.
  • To train one expert only, pass that expert’s file to --dit (no --dit_high_noise) and limit the timesteps with --min_timestep / --max_timestep plus --preserve_distribution_shape. The official ranges are:
Expert (official Wan docs) min_timestep max_timestep
T2V low noise 0 875
T2V high noise 875 1000
I2V low noise 0 900
I2V high noise 900 1000

While training runs, check the GPU from a second tmux window (Ctrl-b then c) with nvidia-smi: a python process from the venv should be listed with GPU memory in use.

Step 9: Use the LoRA

The trainer writes my-wan22-lora .safetensors files to ~/output, one per saved epoch. According to the official Wan docs, LoRAs trained with musubi-tuner seem to be compatible with ComfyUI, depending on the nodes you use; the docs also give wan_generate_video.py for inference and note that the official implementation uses a shift of 12.0 for T2V and 5.0 for I2V. To test the LoRA in ComfyUI on CloudGPU, see /rent-gpu-for-comfyui.

Official hardware notes

The official musubi-tuner README, read 2026-10-30, says: “VRAM: 12GB or more recommended for image training, 24GB or more for video training”, “Actual requirements depend on resolution and training settings”, and “Main Memory: 64GB or more recommended, 32GB + swap may work”. These are the project’s own notes, not CloudGPU measurements. The Wan docs add that --offload_inactive_dit and --blocks_to_swap trade VRAM use for main RAM use.

Measured on CloudGPU

Measurements on CloudGPU coming soon; until then, verify with the nvidia-smi check in Step 8 (a python process from the venv listed with GPU memory in use).

Step 10: Save what you need, then destroy the instance

Data on the instance is deleted when you destroy it. Copy the LoRA files, your dataset.toml and any logs to your computer first, for example with rsync -avP -e "ssh -p <ssh-port>" <user>@<instance-host>:~/output ./wan22-lora-output. CloudGPU bills per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance; see /terms. You can pay by card, Alipay, Google Pay, Apple Pay or USDT.

Troubleshooting

  • Model loading fails with an fp8_scaled file: the official Wan docs say fp8_scaled DiT files are not supported; use the _fp16 files from Step 5.
  • Error about the VAE: use Wan2.1_VAE.pth; the Wan2.2 VAE is for the 5B model.
  • target_frames error: for Wan each value must be N*4+1, such as 1, 25 or 45.
  • ValueError: fp16 mixed precision requires a GPU during accelerate config: answer 0 to the GPU question, as the README says.
  • torch.cuda.is_available() prints False or no kernel image is available for execution on the device: reinstall PyTorch from the CUDA index that matches nvidia-smi (Step 3) and check the arch list.
  • CUDA out of memory: add --blocks_to_swap N or --offload_inactive_dit (not both), train one expert at a time, or lower resolution and target_frames in the TOML.

FAQ

Which Wan 2.2 models can musubi-tuner train?

According to the official musubi-tuner Wan docs, read on 2026-10-30, musubi-tuner supports the Wan 2.2 14B models only: t2v-A14B and i2v-A14B. The 5B model and its Wan2.2 VAE are not supported.

Which model files do I need for Wan 2.2 LoRA training in musubi-tuner?

You need the fp16 high-noise and low-noise DiT files from the Comfy-Org Wan 2.2 repackaged repository, the models_t5_umt5-xxl-enc-bf16.pth T5 encoder and the Wan2.1_VAE.pth VAE. The official docs say fp8_scaled DiT files are not supported and Wan 2.2 needs no CLIP model.

How do I train both Wan 2.2 experts in one run?

According to the official musubi-tuner Wan docs, pass the low-noise model to --dit and the high-noise model to --dit_high_noise with --task t2v-A14B or i2v-A14B. The --timestep_boundary option defaults to 0.875 for T2V and 0.9 for I2V.

How much VRAM do I need to train a Wan 2.2 LoRA?

VRAM need depends on model, precision and workflow. The official musubi-tuner README, read on 2026-10-30, says 24GB or more for video training and that actual requirements depend on resolution and training settings. See https://cloudgpu.app/guides/how-much-vram-do-i-need-to-run-an-llm for more detail.

How is a Wan 2.2 LoRA training session billed on CloudGPU?

CloudGPU bills GPU instances per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance. See https://cloudgpu.app/terms for the full terms.