LabHub

Blog

LLM Fine-Tuning 2026 Deep Dive — LoRA · QLoRA · DoRA · GaLore · Unsloth · Axolotl · TRL · PEFT · MLX-LM Complete Guide

2026-05-16 · 24 min read · #llm-fine-tuning#lora#qlora#dora#galore#unsloth#axolotl#trl#peft#mlx#sft#dpo#deep-dive#english

한국어English日本語

Prologue — 2026, the Year Fine-Tuning Became a "User Tool"

In June 2021, Edward Hu (Microsoft) published the LoRA paper, showing that "training just a million parameters can give you GPT-3-level adaptation." Back then, fine-tuning was a supercomputer's job. In May 2023, Tim Dettmers released QLoRA and squeezed a 65B model onto a single 48GB GPU. From then on, fine-tuning shifted from "researcher's work" to "engineer's work."

And now, in May 2026, fine-tuning is a "user tool." A single M3 Ultra Mac Studio can LoRA-tune a 70B model. Unsloth handles 4-bit quantization automatically, and the TRL SFTTrainer wraps an SFT run in five lines. Axolotl and LLaMA-Factory have externalized every hyperparameter into a single YAML file. Apple's MLX-LM trains Mistral 7B LoRA on an M4 Max MacBook Pro. All of this happened in four years.

Fine-tuning no longer means "train the whole model from scratch." In over 90% of cases, the answer is a LoRA-family PEFT (Parameter-Efficient Fine-Tuning). And within PEFT, the variant you pick determines efficiency, quality, and hardware requirements.

What this article covers:

  1. Full fine-tuning vs PEFT — when to use which
  2. LoRA's math and intuition (Hu et al., 2021)
  3. QLoRA — 4-bit NF4 and double quantization (Dettmers, 2023)
  4. DoRA — Weight decomposition (NVIDIA, 2024)
  5. GaLore — Gradient projection (Zhao, 2024)
  6. PiSSA, LoRA+, rsLoRA, VeRA, LoftQ, OFT, BOFT
  7. PEFT 0.14 — Hugging Face's unified API
  8. TRL 0.13 — SFT, DPO, ORPO, KTO, IPO, GRPO, RLOO
  9. Unsloth — 2x faster training
  10. Axolotl 0.6 — YAML-based multi-GPU
  11. LLaMA-Factory — 100+ models, web UI
  12. MLX-LM and MLX-Tuner — Apple Silicon on-device
  13. Torchtune — Meta PyTorch recipes
  14. Datasets — ShareGPT, OpenHermes, Magpie, Tülu, Nectar
  15. DPO vs PPO — the RLHF alternative
  16. Synthetic data — Augmentoolkit, Distilabel, Self-Instruct
  17. Quantization for serving — GGUF, AWQ, GPTQ, EXL2
  18. Hardware — H100/H200, A100, 4090/5090, M3/M4, MI300X
  19. Cloud — Together, Modal, RunPod, vast.ai, Lambda Labs
  20. Fine-tuning Korean and Japanese models
  21. Which tool should you pick
  22. References

1. Full Fine-Tuning vs PEFT — The GPU Memory Math

The first decision in fine-tuning is "train all weights, or only some?" The answer is almost always only some. The GPU memory math makes it obvious.

Full Fine-Tuning Math

Suppose you full-fine-tune a 7B model. fp16 weights alone are 14GB. The Adam optimizer keeps two states (momentum and variance) per weight in fp32, so 4x weights = 56GB. Gradients are the same size as weights, 14GB. Activations scale with sequence length, adding tens of GB. Total: over 100GB for a 7B full fine-tune. A single H100 80GB cannot do it.

For 70B, multiply by 10. 1TB of memory. Even a single 8x H100 node is tight.

PEFT Math

LoRA on 7B? Base weights stay 14GB (or 3.5GB in 4-bit). LoRA adapters are usually 0.1~1% of the base — 7M~70M parameters or 14~140MB in fp16. Optimizer state covers only the adapter, so it stays small. Activations are similar, but if the base is 4-bit, activation memory shrinks too. Total: 8~20GB to train 7B. One RTX 4090 24GB is enough.

When Full Fine-Tuning Still Makes Sense

There are cases where full fine-tuning is the right call:

For the remaining 95% of cases, PEFT is the answer.

2. LoRA — Low-Rank Adaptation (Hu et al., 2021)

LoRA's idea in one sentence: "Weight updates are low-rank." When you full-fine-tune, W becomes W + dW. The observation is that dW typically has very low rank — not full rank.

The Math

For a base weight W (size d×k), LoRA approximates dW with the product of two small matrices A (d×r) and B (r×k), where r << min(d, k). During training, W is frozen and only A·B are updated. At inference, two options:

  1. Merge — precompute W + A·B as new weights. No inference overhead.
  2. Keep the adapter — keep W as-is, apply A·B as a separate adapter. Multiple adapters can be hot-swapped.

Three key hyperparameters:

Practical Guide (2026)

3. QLoRA — 4-bit NF4 and Double Quantization (Dettmers, 2023)

Tim Dettmers's QLoRA (May 2023), in one sentence: "Train LoRA on top of a 4-bit-quantized base, and memory drops 4x."

Three Key Contributions

  1. NF4 (NormalFloat 4-bit) — a 4-bit quantization optimized for normal distributions. Pretrained weights are roughly mean-0 normal, so QLoRA assigns 4-bit codes to the quantiles of a normal distribution. More accurate than INT4.
  2. Double Quantization — quantize the quantization constants (scales) themselves. The first quantization gives an fp32 scale per block; double-quantizing those scales saves additional memory. About 0.5 bits per parameter saved on average.
  3. Paged Optimizers — use NVIDIA Unified Memory to page optimizer state between CPU and GPU. Avoids OOM and improves training stability.

Memory Savings in Practice

7B fp16 = 14GB to 7B NF4 = 3.5GB. 4x reduction. Optimizer state stays small because it only covers the LoRA adapter. Outcome: train 7B on a single RTX 3090 24GB.

QLoRA has become the de facto default for LoRA training in 2026. Unsloth, Axolotl, and LLaMA-Factory all offer QLoRA as a first-class option.

QLoRA's Minor Tradeoffs

A 4-bit base can be slightly less stable to train on than fp16. So very large distribution shifts (language transfer, deep domain specialization) are safer with an fp16 base. Also, after training an adapter on a 4-bit base, merging that adapter back into the fp16 base can introduce small quality loss. LoftQ (chapter 7) tries to solve this.

4. DoRA — Weight-Decomposed LoRA (NVIDIA, 2024)

NVIDIA's Shih-Yang Liu introduced DoRA in 2024. One sentence: "Decompose the weight change into direction and magnitude, and LoRA does better."

Intuition

Decompose the weight matrix W into two parts:

DoRA applies LoRA to the direction and treats the magnitude m as a separately trainable parameter. The DoRA paper observes that full fine-tuning changes both magnitude and direction significantly, but LoRA tangles the two awkwardly.

Experimental Results

PEFT Support

Hugging Face PEFT 0.10 added use_dora=True as a single-flag activation. As of 2026, an informal estimate says about 30% of new LoRA training jobs use DoRA.

5. GaLore — Gradient Low-Rank Projection (Zhao, 2024)

Jiawei Zhao (Meta) published GaLore in March 2024. One sentence: "Weights are full rank, but gradients are low rank — project gradients into a low-rank subspace to save memory."

How It Differs

LoRA trains adapters (low-rank) and freezes the base weights. The result is few trainable parameters — an expressiveness ceiling. GaLore trains all the weights, but keeps the optimizer state (Adam's momentum and variance) in a low-rank subspace. Every N steps, GaLore runs SVD to find the principal direction of the gradient and keeps only that subspace as optimizer state.

The result: full fine-tuning's expressiveness with memory close to LoRA.

Memory Savings

For 7B full fine-tuning, the optimizer state (fp32 Adam) is 56GB. GaLore drops it to 7~14GB. The weights and activations are still full-training-sized, so total memory exceeds LoRA but is less than half of full fine-tuning.

Tradeoffs

GaLore targets the niche of "LoRA isn't enough, but full fine-tuning is too memory-hungry." A niche slice as of 2026, but growing.

6. PEFT Variants — PiSSA, LoRA+, rsLoRA, VeRA, LoftQ, OFT, BOFT

A swarm of LoRA variants have appeared. Short summary of the major ones supported by PEFT 0.14.

These variants give small marginal gains over vanilla LoRA. For 90% of cases, plain LoRA is enough. Try variants when chasing the top of an evaluation benchmark.

7. Hugging Face PEFT 0.14 — The Unified API

Hugging Face's PEFT library is the unified API for every variant above. As of May 2026, 0.14 is stable and 0.15 is in beta.

Basic Usage

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B")

config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules="all-linear",
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    use_dora=False,  # Set True for DoRA
)

model = get_peft_model(model, config)
model.print_trainable_parameters()
# trainable params: 24,313,856 || all params: 3,236,000,000 || trainable%: 0.75

Save and Load Adapters

model.save_pretrained("./my-lora-adapter")

# Later, reload
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B")
model = PeftModel.from_pretrained(base, "./my-lora-adapter")

Merge the Adapter

merged = model.merge_and_unload()
merged.save_pretrained("./merged-model")

Multi-Adapter

PEFT can hold multiple adapters on the same base and hot-swap them — switch the same model instance between English, Japanese, and Korean adapters instantly. Very useful for inference servers.

8. TRL 0.13 — SFT, DPO, ORPO, KTO, IPO, GRPO, RLOO

TRL (Transformer Reinforcement Learning) is Hugging Face's RLHF and alignment library. As of May 2026, 0.13 is stable, and the newer algorithms like GRPO and RLOO are first-class citizens.

SFTTrainer (Supervised Fine-Tuning)

The most-used tool. Train SFT on a chat dataset.

from trl import SFTTrainer, SFTConfig

config = SFTConfig(
    output_dir="./sft-output",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    bf16=True,
    packing=True,
    max_seq_length=4096,
)

trainer = SFTTrainer(
    model=model,
    args=config,
    train_dataset=dataset,
    peft_config=lora_config,  # Integrates with PEFT
)
trainer.train()

DPOTrainer (Direct Preference Optimization)

Align with pair data (prompt, chosen, rejected). Much simpler and more stable than PPO.

ORPOTrainer (Odd Ratio Preference Optimization)

A newer algorithm (2024) that does SFT and DPO together. No reference model needed, saving memory.

KTO (Kahneman-Tversky Optimization)

Inspired by prospect theory. Allows alignment with single labels (good/bad) instead of pairs.

GRPO (Group Relative Policy Optimization)

The algorithm DeepSeek-R1 used. Strong for reasoning training. As of 2026, the standard for reasoning models.

RLOO (REINFORCE Leave-One-Out)

Uses leave-one-out for baseline estimation. Simpler than PPO yet effective.

9. Unsloth — 2x Faster Training, 50% Less Memory

Unsloth (the Han brothers — Daniel and Michael, 2024) is the single-GPU fine-tuning game changer. One sentence: "Hand-written Triton kernels make LoRA/QLoRA training 2x faster."

How It's Fast

Example Usage

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3.2-3b-instruct-bnb-4bit",
    max_seq_length=4096,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=32,
    use_rslora=False,
    use_dora=False,
)

Then train via SFTTrainer. Result: train Llama 3.2 8B in 4 hours on an RTX 4090 24GB (vs 8~10 hours with vanilla Transformers).

Constraints

10. Axolotl 0.6 — Everything in YAML

Axolotl (OpenAccess AI Collective, 2023~) in one sentence: "Externalize every hyperparameter into a single YAML file."

Why YAML

Python training scripts are hard to reproduce. Storing learning rate, batch size, and adapter config in git means the code itself is a variable. YAML makes a single source of truth.

Example Config

base_model: meta-llama/Llama-3.2-3B
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer

load_in_4bit: true
adapter: qlora
lora_r: 16
lora_alpha: 32
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj

datasets:
  - path: tatsu-lab/alpaca
    type: alpaca

sequence_len: 4096
sample_packing: true
gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 3
learning_rate: 2e-4
bf16: auto
optimizer: paged_adamw_32bit

Strengths

Weaknesses

11. LLaMA-Factory — 100+ Models, Web UI

HiYouga's LLaMA-Factory (Zheng et al., 2023) in one sentence: "The GUI era of fine-tuning."

Features

Usage

pip install llamafactory
llamafactory-cli webui

Pick model, dataset, training mode in the browser, hit start. Non-coders can use it. Spread fast in Korean, Japanese, and Chinese in-house LLM teams.

Who Uses It

12. MLX-LM and MLX-Tuner — Apple Silicon On-Device

Apple released MLX (Machine Learning eXperience) in December 2023. PyTorch-like API but optimized for the Unified Memory Architecture (UMA) — CPU and GPU share the same memory pool on M1~M4 chips.

What MLX-LM Means

Example Usage

pip install mlx-lm
python -m mlx_lm.lora \
    --model mistralai/Mistral-7B-v0.3 \
    --train \
    --data ./my_data \
    --iters 1000 \
    --lora-layers 16 \
    --batch-size 4

Constraints

Still, "fine-tune on my laptop" has weight. Small teams in Korea and Japan can experiment without cloud bills.

13. Torchtune — Meta PyTorch's Official Recipes

Torchtune is the library the Meta PyTorch team released in April 2024. One sentence: "PyTorch native, memory efficient, recipe-based."

Philosophy

Usage

pip install torchtune
tune download meta-llama/Llama-3.2-3B-Instruct \
    --output-dir /tmp/Llama-3.2-3B \
    --hf-token YOUR_TOKEN

tune run lora_finetune_single_device \
    --config llama3_2/3B_lora_single_device

Strengths

Weaknesses

14. Datasets — ShareGPT, OpenHermes, Magpie, Tülu, Nectar

70% of good fine-tuning is the data. The popular open SFT datasets in 2026.

Quality Trumps Quantity

The LIMA paper (Meta, 2023) showed 1,000 well-curated examples beat 50,000 mediocre ones for SFT. By 2026, "more" has yielded to "better" in training data curation.

15. DPO vs PPO — Evolution of RLHF

The era of InstructGPT (2022) and ChatGPT (2022) used PPO (Proximal Policy Optimization). PPO requires: base model, reward model, reference model, actor, critic. Four models on GPU at once — a memory explosion.

The Arrival of DPO (2023)

Rafailov et al. (Stanford) released DPO (Direct Preference Optimization) in May 2023. One sentence: "Skip the reward model and optimize the policy directly from pair data."

Mathematically, DPO transforms PPO's optimization into a simple classification loss on pair data. The result:

The 2026 Competition

When do we still use PPO? Almost never. As of 2026, PPO survives only in large labs with mature training infrastructure (e.g., OpenAI internal). For a new project, DPO or its successors are the answer.

16. Synthetic Data — Augmentoolkit, Distilabel, Self-Instruct

No good SFT data? Make it. Synthetic data generation has become a first-class citizen of fine-tuning in 2026.

Self-Instruct (Wang et al., 2022)

The original. Give 175 seed instructions to an LLM, have it generate new instructions, then have the LLM generate answers. The bootstrap technique behind Alpaca, Wizard, and Magpie.

Augmentoolkit (e-p-armstrong, 2024)

A pipeline that produces QA pairs from source text. Converts PDFs, books, internal docs into SFT-ready format.

Distilabel (Argilla, now Hugging Face, 2024)

A synthetic data framework from the Argilla team (now at Hugging Face). Define a step-by-step pipeline — generate instruction, generate answer, AI evaluation, filtering. UltraFeedback was built with Distilabel.

NeMo Curator (NVIDIA)

Large-scale data curation — deduplication, quality filtering, PII removal. Used for both pretraining and SFT.

Magpie Technique

Give Llama-3-Instruct an empty prompt and the model generates "the user's turn" by itself — use that as the instruction and have the model answer. Zero cost (self-generation), 1M+ samples. Downside: the model's biases come along.

Synthetic Data Pitfalls

17. Quantization for Serving — GGUF, AWQ, GPTQ, EXL2

To serve a fine-tuned model, you quantize. Separate from QLoRA's 4-bit during training, inference has its own quantization formats.

Workflow

  1. Train LoRA in fp16/bf16, then merge the adapter into the base.
  2. Convert merged model to GGUF (CPU/Mac), AWQ (server GPU), or EXL2 (RTX inference).
  3. Serve via vLLM, TGI, llama.cpp, or Ollama.

The most common 2026 flow: train LoRA in PyTorch, merge, quantize to GGUF, serve with Ollama.

18. Hardware — H100/H200, A100, 4090/5090, M3/M4, MI300X

Fine-tuning hardware in 2026:

NVIDIA Data Center

NVIDIA Consumer

Apple Silicon

AMD

How to Pick

19. Cloud — Together, Modal, RunPod, vast.ai, Lambda Labs, Replicate

Most people rent GPUs instead of buying. The main 2026 options.

Price Sense (rough as of May 2026)

20. Fine-Tuning Korean and Japanese Models

Korea and Japan have active base-model builders and fine-tuning communities.

Korea

Japan

KO/JA Fine-Tuning Patterns

21. Which Tool Should You Pick — Decision Tree

Final summary. Recommended combos per scenario.

First Time Trying LoRA

In-House Model From Company Data

Korean/Japanese-Specialized Model

Reasoning Model

Small On a Laptop

Fast Prototype

22. References

Core Papers

Library Docs

Korean and Japanese LLM Ecosystem

Epilogue — Fine-Tuning Is No Longer a Secret

One-sentence summary of this article: In 2026, LLM fine-tuning is a "user tool." While LoRA evolved from a simple adapter to DoRA and GaLore, Unsloth doubled single-GPU training speed and Axolotl and LLaMA-Factory dissolved the entry barrier. We live in an age where one M3 Ultra Mac Studio can LoRA-tune a 70B model.

The remaining question is not "can I do it" but "what should I train it on." Good data, good evaluation, good use cases — that is the real work of a 2026 fine-tuning engineer.

"Tools are no longer the bottleneck. Data is."

— LLM Fine-Tuning 2026, end.

Read the original: https://www.youngju.dev/blog/culture/2026-05-16-llm-fine-tuning-2026-lora-qlora-dora-galore-unsloth-axolotl-trl-peft-mlx-lm-deep-dive.en

Comments

No comments yet.

Sign in to leave a comment