How to fine-tune an LLM with LLaMA-Factory on a cloud GPU

Rent a GPU on Deploy, then open a terminal in JupyterLab or over SSH. Install LLaMA-Factory from the official repository, run the LoRA or QLoRA SFT example, or open LlamaBoard, and copy the adapter off the instance before you destroy it. Destroying the instance deletes its disk.

The commands below are copied from the upstream files named in each step. They were not run on a CloudGPU instance for this page. Anything this page adds around them is marked.

The same install, with Axolotl or Unsloth instead of LLaMA-Factory, is how to fine-tune an LLM with Axolotl and how to fine-tune an LLM with Unsloth. An earlier write-up of this tool is the LLaMA-Factory blog post.

Memory the README table publishes

The LLaMA-Factory README hardware table is marked estimated (read 2026-11-12). The rows this page uses are:

Method Bits 7B 14B 30B 70B xB
Freeze / LoRA / GaLore / APOLLO / BAdam / OFT 16 16GB 32GB 64GB 160GB 2x GB
QLoRA / QOFT 8 10GB 20GB 40GB 80GB x GB
QLoRA / QOFT 4 6GB 12GB 24GB 48GB x/2 GB

The same table also prints full-tuning rows and a 2-bit QLoRA row. The quickstart below is LoRA SFT, not those rows. The table has no 4B column. The quickstart model id is Qwen/Qwen3-4B-Instruct-2507. This page does not substitute 4 into the x formula.

GET /v1/public/compute-capabilities does not list GPUs. It returns feature flags. The memory column is vram_gb from gpu-prices.json, updated_at 2026-11-12T10:02:26.669Z. GET /v1/public/gpu-catalog the same day (generated_at 2026-11-12T10:02:27.380Z) listed the same names and the same vram_gb values, plus Ascend 910B, which is not in gpu-prices.json and is not listed here. A cell is met when vram_gb is greater than or equal to the printed GB. Which listing you can start is on Deploy. This page does not say which listings are available.

Catalog name vram_gb 16-bit LoRA 8-bit QLoRA 4-bit QLoRA
RTX 2080 8 below 7B below 7B 7B
RTX 3080 10 below 7B 7B 7B
RTX 2080 Ti 11 below 7B 7B 7B
RTX 3080 Ti 12 below 7B 7B 7B, 14B
V100 16G 16 7B 7B 7B, 14B
A30 24 7B 7B, 14B 7B, 14B, 30B
RTX 3090 24 7B 7B, 14B 7B, 14B, 30B
RTX 4090 24 7B 7B, 14B 7B, 14B, 30B
RTX 4090D 24 7B 7B, 14B 7B, 14B, 30B
RTX 4080 SUPER 32G 32 7B, 14B 7B, 14B 7B, 14B, 30B
RTX 5090 32 7B, 14B 7B, 14B 7B, 14B, 30B
RTX 5090D 32 7B, 14B 7B, 14B 7B, 14B, 30B
V100 32G 32 7B, 14B 7B, 14B 7B, 14B, 30B
A100 40G 40 7B, 14B 7B, 14B, 30B 7B, 14B, 30B
A800 40G 40 7B, 14B 7B, 14B, 30B 7B, 14B, 30B
A40 48 7B, 14B 7B, 14B, 30B 7B, 14B, 30B, 70B
RTX 4090 48G 48 7B, 14B 7B, 14B, 30B 7B, 14B, 30B, 70B
RTX 4090D 48G 48 7B, 14B 7B, 14B, 30B 7B, 14B, 30B, 70B
RTX A6000 48 7B, 14B 7B, 14B, 30B 7B, 14B, 30B, 70B
NVIDIA RTX PRO 5000 72G 72 7B, 14B, 30B 7B, 14B, 30B 7B, 14B, 30B, 70B
RTX PRO 5000 72G 72 7B, 14B, 30B 7B, 14B, 30B 7B, 14B, 30B, 70B
A100 80G 80 7B, 14B, 30B 7B, 14B, 30B, 70B 7B, 14B, 30B, 70B
A800 80G 80 7B, 14B, 30B 7B, 14B, 30B, 70B 7B, 14B, 30B, 70B
H800 80 7B, 14B, 30B 7B, 14B, 30B, 70B 7B, 14B, 30B, 70B
RTX 6000D 83 7B, 14B, 30B 7B, 14B, 30B, 70B 7B, 14B, 30B, 70B
H20 96G 96 7B, 14B, 30B 7B, 14B, 30B, 70B 7B, 14B, 30B, 70B
RTX PRO 6000 96G 96 7B, 14B, 30B 7B, 14B, 30B, 70B 7B, 14B, 30B, 70B
RTX PRO 6000 Max-Q 96 7B, 14B, 30B 7B, 14B, 30B, 70B 7B, 14B, 30B, 70B

No vram_gb in that file is 160 or above, so none meets the 70B 16-bit LoRA cell. The public catalog also lists a template id llama-factory, name “LLaMA Factory”, recommended_gpu “RTX 4090 48G”, min_vram_gb 24 (read 2026-11-12). Those two fields are the template record. They are not cells in the README table. The steps below are the upstream source install on Raw GPU.

Install from source

On Deploy, choose Raw GPU. The README requirement table says Python minimum 3.11, recommend >=3.11, torch minimum 2.0.0, recommend 2.6.0, and transformers minimum 4.49.0, recommend 4.50.0. pyproject.toml on main, read the same day, is what pip install -e . installs: requires-python >=3.11.0, torch>=2.4.0, and transformers>=4.55.0,<=5.8.0,!=4.57.0,!=5.6.0. This page does not pick one of those two ranges over the other. bitsandbytes is not in that core list. requirements/bitsandbytes.txt is the line bitsandbytes>=0.39.0, which matches the optional table minimum. The source-install block does not install that file.

The clone and the install are the README. They were not run.

git clone --depth 1 https://github.com/hiyouga/LlamaFactory.git
cd LlamaFactory
pip install -e .
pip install -r requirements/metrics.txt

The QLoRA example sets quantization_method: bnb. The source-install block does not install requirements/bitsandbytes.txt. This line is that file, not the install block. It was not run.

pip install -r requirements/bitsandbytes.txt

If conda is what you use to get Python 3.11, see how to set up a conda environment on a cloud GPU. This page did not check whether the Raw GPU image already has conda. pyproject.toml does not pin a CUDA wheel index. How to install a specific PyTorch CUDA version says to use an index that is not newer than the CUDA Version in the nvidia-smi header. A long train should sit in a session that survives a dropped SSH connection: how to keep a job running after SSH disconnects.

The README also documents docker run -it --rm --gpus=all --ipc=host hiyouga/llamafactory:latest. It says that image is Ubuntu 22.04, CUDA 12.4, Python 3.11, PyTorch 2.6.0, and Flash-attn 2.7.4. This page follows the source install. The image was not run.

Check that PyTorch sees the GPU before you train. This check is not in the README.

python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available(), torch.cuda.get_device_name(0))"

LoRA SFT, then QLoRA SFT

The README quickstart, copied. The three commands are fine-tuning, inference, and merging for the Qwen3-4B-Instruct model.

llamafactory-cli train examples/train_lora/qwen3_lora_sft.yaml
llamafactory-cli chat examples/inference/qwen3_lora_sft.yaml
llamafactory-cli export examples/merge_lora/qwen3_lora_sft.yaml

examples/train_lora/qwen3_lora_sft.yaml sets model_name_or_path: Qwen/Qwen3-4B-Instruct-2507, stage: sft, finetuning_type: lora, lora_rank: 8, dataset: identity,alpaca_en_demo, template: qwen3_nothink, cutoff_len: 2048, output_dir: saves/qwen3-4b/lora/sft, and bf16: true. data/dataset_info.json maps identity to identity.json and alpaca_en_demo to alpaca_en_demo.json. The Hugging Face model API for Qwen/Qwen3-4B-Instruct-2507 reported gated: false, pipeline_tag: text-generation, and license apache-2.0 (read 2026-11-12). The first train downloads that repository. How to download a Hugging Face model is the manual hf download reference. cloudgpu-net is not in the LLaMA-Factory README. The download speed guide says to leave it off during pip, then turn it on for a weight download.

The README license section says the repository is Apache-2.0 and that each model has its own license. Read the license on the model card you load. This page does not copy the license links in that section.

For your own data, the README says to update data/dataset_info.json. The format notes are in data/README.md. This page does not invent a dataset file.

The QLoRA command the examples README marks as recommended, copied:

llamafactory-cli train examples/train_qlora/qwen3_lora_sft_otfq.yaml

That yaml uses the same model id, the same output_dir, and adds quantization_bit: 4 and quantization_method: bnb. The comment in the file lists the choices: 8 for bnb/hqq/eetq, 4 for bnb/hqq, 3 for hqq, 2 for hqq. The merge yaml starts with “DO NOT use quantized model or quantization_bit when merging LoRA adapters” and sets export_dir: saves/qwen3_sft_merged. export_size help text in src/llamafactory/hparams/model_args.py is “The file shard size (in GB) of the exported model.” The example sets export_size: 5 and export_device: cpu.

LlamaBoard

The README command, copied:

llamafactory-cli webui

src/webui.py (read 2026-11-12) sets server_name from GRADIO_SERVER_NAME, default 0.0.0.0, and calls launch with share, server_name, and inbrowser=True. It does not pass a port. The print above that call uses http://127.0.0.1:7860 as the example. This page did not run the UI, so it did not confirm the port the process prints. How to use SSH port forwarding starts the app on 127.0.0.1. Setting the environment variable the file reads is a change to the README command. It was not run.

GRADIO_SERVER_NAME=127.0.0.1 llamafactory-cli webui

Forward local port 7860 only if the process prints 7860. Then open http://localhost:7860.

The LLaMA-Factory README does not document a ComfyUI route. This page does not add one.

When it fails

CUDA out of memory. The published cells are the estimated table above. A listing below the cell you are using does not meet that cell. Then use how to fix CUDA out of memory on a cloud GPU. Watch the run with how to check GPU utilization.

“CUDA error: no kernel image is available” on an RTX 5090. Catalog RTX 5090 and RTX 5090D are 32 GB, which meets the 14B 16-bit LoRA cell and the 30B 4-bit QLoRA cell, and is below the 70B 4-bit cell. That is the catalog comparison, not a measurement of this example. pyproject.toml does not pin a CUDA wheel. The README Windows section says RTX 50 Series GPUs need the CUDA 12.8–12.9 bitsandbytes builds. The example card in that sentence is not in the catalog, so this page does not name it. The RTX 5090 guide says that card needs a PyTorch build for CUDA 12.8 or newer. This page did not run the install on an RTX 5090.

Save the adapter, then destroy

Copy saves/qwen3-4b/lora/sft and, if you merged, saves/qwen3_sft_merged, before you destroy the instance. Save your work before you destroy a cloud GPU is the checklist. Checkpoint settings already on this site: how to save and resume training checkpoints. The terms say data on local instance storage is permanently deleted when the instance is destroyed. The billing rule is on the pricing page. Prices are not copied here.

Frequently asked questions

How much VRAM does the LLaMA-Factory README publish for LoRA and QLoRA?

The LLaMA-Factory README marks its hardware table as estimated (read 2026-11-12). The 16-bit LoRA row is 16GB for 7B, 32GB for 14B, 64GB for 30B, 160GB for 70B, and 2x GB. The 4-bit QLoRA row is 6GB, 12GB, 24GB, 48GB, and x/2 GB. The 8-bit QLoRA row is 10GB, 20GB, 40GB, 80GB, and x GB. The table has no 4B column. The quickstart model id is Qwen/Qwen3-4B-Instruct-2507. This page does not substitute 4 into the x formula.

Which rented GPUs meet the 7B 16-bit LoRA cell?

A catalog vram_gb meets a printed cell when it is greater than or equal to that cell. For the 16-bit LoRA 7B cell of 16GB, that is every name at 16 or above in https://cloudgpu.app/gpu-prices.json read 2026-11-12: V100 16G, A30, RTX 3090, RTX 4090, RTX 4090D, RTX 4080 SUPER 32G, RTX 5090, RTX 5090D, V100 32G, A100 40G, A800 40G, A40, RTX 4090 48G, RTX 4090D 48G, RTX A6000, NVIDIA RTX PRO 5000 72G, RTX PRO 5000 72G, A100 80G, A800 80G, H800, RTX 6000D, H20 96G, RTX PRO 6000 96G, and RTX PRO 6000 Max-Q. RTX 2080 (8), RTX 3080 (10), RTX 2080 Ti (11), and RTX 3080 Ti (12) are below 16GB. No catalog vram_gb is 160 or above, so none meets the 70B 16-bit LoRA cell. GET /v1/public/compute-capabilities returned feature flags and no GPU list. Which listing you can start is on https://cloudgpu.app/deploy. This page does not say which listings are available.

How do I run LoRA SFT and QLoRA SFT?

LoRA SFT is llamafactory-cli train examples/train_lora/qwen3_lora_sft.yaml. The QLoRA command the examples README marks as recommended is llamafactory-cli train examples/train_qlora/qwen3_lora_sft_otfq.yaml. That file sets quantization_bit to 4 and quantization_method to bnb. LlamaBoard is llamafactory-cli webui. The README does not document ComfyUI.

Where are the adapter and the merged model saved?

The LoRA and QLoRA example yaml files set output_dir to saves/qwen3-4b/lora/sft. llamafactory-cli export examples/merge_lora/qwen3_lora_sft.yaml sets export_dir to saves/qwen3_sft_merged. The merge yaml says not to use a quantized model or quantization_bit when merging. Copy those directories before you destroy the instance. The terms say data on local instance storage is permanently deleted when the instance is destroyed.

Sources (read 2026-11-12, Asia/Shanghai)

Cards you can start are on the Deploy page. Prices are on https://cloudgpu.app/pricing and are not repeated here.

Last updated: 2026-11-12 (Asia/Shanghai).