How do I fine-tune an LLM with Axolotl on a cloud GPU?

Cover: Fine-tune with Axolotl on a cloud GPU, axolotl train step by step

Deploy a Raw GPU, confirm that nvidia-smi reports a driver that supports CUDA 13, install Axolotl with uv inside a Python 3.12 virtual environment, then run axolotl fetch examples and axolotl train examples/llama-3/lora-1b.yml; according to the official Axolotl docs, it needs an NVIDIA GPU (Ampere or newer for bf16 and Flash Attention), Python 3.12 or higher and PyTorch 2.13.0 or newer.

Updated 2026-10-28

This guide follows the official Axolotl docs home page, Quickstart and Inference and Merging pages, plus NVIDIA’s CUDA release notes and CUDA GPUs page, all read on 2026-10-28. Axolotl is a fine-tuning framework for large language models that is driven by one YAML config file, released under the Apache-2.0 license. This guide is for people who want to run LoRA, QLoRA or full fine-tuning on a rented NVIDIA GPU such as the RTX 5090 or the other NVIDIA cards listed on /pricing.

Step 1: Deploy a Raw GPU and connect

On /deploy, pick an NVIDIA GPU and the Raw GPU option (JupyterLab + CUDA pre-installed, root SSH). Connect over SSH or open a terminal in JupyterLab.

VRAM need depends on model, precision and workflow; see How much VRAM do I need to run an LLM? and Rent a GPU for LoRA fine-tuning before you choose a card. Training needs more memory than running the same model, so start with a small example.

Step 2: Check the environment first

nvidia-smi          # GPU name, driver version and the "CUDA Version" the driver supports
nvcc --version      # CUDA toolkit compiler, if installed
python3 --version   # Axolotl asks for Python 3.12 or higher (uv can install 3.12 for you)

What to look for:

Driver version. The install command in the Axolotl docs sets UV_TORCH_BACKEND=cu130, a PyTorch build for CUDA 13.0, and NVIDIA’s CUDA release notes list driver 580 or newer for CUDA 13.x. If your driver is older than 580, the docs’ comment on that line says to change it depending on your system, and the Axolotl Installation page gives cu128 as the other option; see How to install a specific PyTorch CUDA version on a cloud GPU for matching PyTorch to your driver. For Blackwell GPUs, which the Installation page says include the RTX 50-series, it asks for PyTorch 2.11.0 or newer and CUDA 13.0.
GPU generation. The docs ask for an NVIDIA GPU of the Ampere generation or newer for bf16 and Flash Attention. NVIDIA’s CUDA GPUs page lists compute capability 12.0 for the GeForce RTX 5090, 8.9 for the GeForce RTX 4090 and 8.0 for the NVIDIA A100; look up any other card on that page.
nvcc. Some optional packages build CUDA code during install. If one fails because it cannot find the CUDA toolkit, note the path from ls -d /usr/local/cuda*.

Step 3: Install Axolotl with uv

According to the official Axolotl docs, install uv, create a Python 3.12 environment, then install PyTorch and Axolotl:

# install uv if you don't already have it installed (restart shell after)
curl -LsSf https://astral.sh/uv/install.sh | sh

# change depending on system (cu130 = CUDA 13.0, needs driver 580 or newer)
export UV_TORCH_BACKEND=cu130

uv venv --python 3.12
source .venv/bin/activate

uv pip install torch==2.14.0 torchvision
uv pip install --no-build-isolation axolotl[deepspeed]

The docs also describe a Docker install (axolotlai/axolotl images). That path needs a Docker host with GPU access; a Raw GPU is a container, so this guide uses the uv path. Run the steps inside tmux or screen so a dropped SSH connection does not stop a long install or training run.

Check that PyTorch sees the GPU before you train:

python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0))"

Step 4: Run the official LoRA example

Download the example configs and train the 1B LoRA example from the Axolotl docs:

axolotl fetch examples
axolotl train examples/llama-3/lora-1b.yml

The Quickstart says this example “uses a 1B parameter model to ensure it runs on most GPUs”. When you run axolotl train, Axolotl downloads the base model, applies the LoRA adapter layers, loads and processes the dataset, runs the training loop and saves the trained model or LoRA weights. Base models and datasets come from Hugging Face; for a gated model, request access on its Hugging Face page and export your token as HF_TOKEN first. See How to download a Hugging Face model on a cloud GPU.

While it trains, run nvidia-smi in a second terminal to see GPU memory and utilization; How to check GPU utilization on a cloud GPU explains the columns.

Step 5: Train on your own data

The Quickstart’s custom example uses the alpaca format, one JSON object per line in a .jsonl file:

{"instruction": "Classify this text", "input": "I love this!", "output": "positive"}
{"instruction": "Classify this text", "input": "Not good at all", "output": "negative"}

Then create my_training.yml (this is the Quickstart’s example config):

base_model: NousResearch/Nous-Hermes-llama-1b-v1
load_in_8bit: true
adapter: lora

# Training settings
micro_batch_size: 2
num_epochs: 3
learning_rate: 0.0003

# Your dataset
datasets:
  - path: my_data.jsonl        # Your local data file
    type: alpaca               # Or other format

and train it:

axolotl train my_training.yml

The Quickstart explains the adapter lines: load_in_8bit: true with adapter: lora is LoRA; replace them with load_in_4bit: true and adapter: qlora for QLoRA; remove both lines for a full fine-tune. For large datasets, run axolotl preprocess my_training.yml first and set dataset_prepared_path in the config. Other formats are listed on the docs’ Dataset Formats page.

Step 6: Test and merge the result

From the Axolotl Quickstart and Inference and Merging pages, use the same YAML file you trained with:

# test the LoRA adapter
axolotl inference my_training.yml --lora-model-dir="./outputs/lora-out"

# multi-turn chat in the terminal
axolotl inference my_training.yml --lora-model-dir="./outputs/lora-out" --chat

# merge the LoRA weights into the base model
axolotl merge-lora my_training.yml --lora-model-dir="./outputs/lora-out"

Set --lora-model-dir to the output_dir of your config. The docs say the merged model is saved in the {output_dir}/merged directory.

Step 7: Save what you need, then destroy the instance

Data on the instance is deleted when you destroy it, so copy your adapter, merged model, config and logs to your computer or push them to a Hugging Face repo first. GPU time is billed per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance; see /terms.

FAQ

What does Axolotl need on a cloud GPU?

According to the official Axolotl docs, Axolotl needs an NVIDIA GPU (Ampere or newer for bf16 and Flash Attention) or an AMD GPU, Python 3.12 or higher and PyTorch 2.13.0 or newer. The docs’ uv install uses a CUDA 13.0 PyTorch build, and NVIDIA’s CUDA release notes list driver 580 or newer for CUDA 13.x, so check the driver version that nvidia-smi reports first.

What is the first command to test Axolotl?

Run axolotl fetch examples and then axolotl train examples/llama-3/lora-1b.yml. The Axolotl Quickstart says this LoRA example uses a 1B parameter model to ensure it runs on most GPUs.

How do I switch between LoRA, QLoRA and full fine-tuning in Axolotl?

In the YAML config, load_in_8bit: true with adapter: lora is LoRA. For QLoRA, use load_in_4bit: true with adapter: qlora. For a full fine-tune, remove both lines. This is how the Axolotl Quickstart describes it.

Where does Axolotl save the merged model?

Run axolotl merge-lora with the same config and --lora-model-dir pointing at your LoRA output. The Axolotl docs say the merged model is saved in the {output_dir}/merged directory.

How is a Raw GPU billed while Axolotl trains?

GPU time is billed per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance. See https://cloudgpu.app/terms for the full terms.