LLM Fine-Tuning SOP: A LoRA/QLoRA End-to-End Walkthrough

General-purpose LLMs can hold a conversation about almost anything, but the moment they hit your business they fall flat. Ask them to draft customer-service replies in your house tone and they come back with "hey there :)". Ask them to triage tickets into your thirty-odd internal business lines and they shove everything into generic buckets. Ask them to emit a fixed JSON schema and they drop a field two times out of ten. You can twist the prompt until you're blue in the face and still not hold it down, and RAG can only inject knowledge, not change behavior. That's when fine-tuning earns its keep. But full fine-tuning of a 7B model eats tens of gigabytes of VRAM just for Adam optimizer states, far beyond a single consumer GPU. This SOP walks through parameter-efficient fine-tuning with LoRA / QLoRA end to end: first the decision of when to fine-tune at all and how LoRA saves VRAM, then six concrete steps with copy-paste code (data -> base model -> config -> training -> evaluation -> deployment), and finally pitfalls and FAQ. Unlike our site's RAG Evaluation SOP and AI Agent Evaluation SOP, which cover how to evaluate, this one covers how to train, but the evaluation section reuses the same evaluation thinking.

1. Decide First: Fine-Tune, RAG, or Prompt

Fine-tuning is not a universal hammer. Run this decision table before you touch any code, because picking the wrong path wastes everything that follows.

One-line rule: use RAG to add knowledge, fine-tuning to change behavior, and prompts to cover ad-hoc needs. The most common misjudgment is fine-tuning to inject knowledge that changes. Model weights are a static snapshot, so by the time you finish training the facts are already stale and you trained for nothing. The other misjudgment is fine-tuning with only a few dozen examples, where prompt few-shot is the better deal, since fine-tuning at that scale either overfits or learns nothing. The real threshold for fine-tuning is hundreds to thousands of high-quality labeled examples, a stable behavior requirement, and a held-out set that can prove the model actually got better.

2. LoRA / QLoRA Principles: Why Not Full Fine-Tuning

Full fine-tuning updates every weight matrix W. For a 7B model that's 7 billion parameters each getting gradients and optimizer state, and Adam stores an additional 2x parameters for momentum and variance. A single 24GB GPU cannot hold that.

LoRA (Low-Rank Adaptation) rests on the assumption that the weight change during fine-tuning is low-rank and can be decomposed into two small matrices: W = W₀ + B·A, where B is d×r and A is r×d, and the rank r is far smaller than the dimension d (say r=16 with d=4096). Training freezes W₀ and only learns B and A. Parameter count drops from d×d to 2×d×r, which for a 4096-dim single layer is roughly 16.78M down to 0.13M, under 1%. VRAM pressure collapses, and the exported adapter file is only tens of megabytes.

Key hyperparameters to remember for the config step:

• r (rank): the rank of the low-rank matrices. Higher r means more expressive capacity but also easier overfitting. A common starting point is 8-16, scaling to 32-64 for harder tasks with more data.
• alpha: a scaling factor, the effective update is ΔW × (alpha/r). A common heuristic is alpha = 2×r (so r=16 pairs with alpha=32), which keeps the learning rate in a sane range.
• target_modules: which layers get LoRA adapters. Attaching to all attention linear projections (q/k/v/o_proj) is standard; MLP projections (gate/up/down_proj) are optional.
• dropout: dropout on the adapter, 0.05-0.1 is typical to curb overfitting.

QLoRA takes LoRA one step further by quantizing the frozen base W₀ to 4-bit (nf4 quantization plus double quantization), computing in bfloat16 during forward and backward passes, and backpropagating gradients only into the LoRA adapters. The result is that a 7B model trains on a single 16-24GB consumer GPU, and 13B is within reach. This is why LoRA + QLoRA is effectively the default for individual developers and small teams.

3. Prepare Data: Instruction Format and Cleaning

Data sets the ceiling, the model only decides how fast you approach it. Eight times out of ten, a fine-tuning failure is a data problem, not a config problem.

The standard instruction-tuning sample is three fields: instruction, optional input, and output:

import json
from datasets import Dataset

# Raw data: instruction / input / output three-field format
raw = [
    {"instruction": "Classify the support ticket below into a standard business type, output only the category name.",
     "input": "Customer: It's been 3 days since I ordered and it still hasn't shipped, please rush it.",
     "output": "Shipping escalation"},
    {"instruction": "Classify the support ticket below into a standard business type, output only the category name.",
     "input": "Customer: The item I received is damaged, I want a replacement.",
     "output": "After-sales replacement"},
    # ... Rule of thumb: 500-2000 samples for style/format tasks,
    # more for domain behavior injection; under 200 -> use prompt few-shot
]


def format_prompt(ex):
    """Build an alpaca-style prompt (use the base model's official template)"""
    text = f"### Instruction:\n{ex['instruction']}\n"
    if ex.get("input"):
        text += f"### Input:\n{ex['input']}\n"
    text += f"### Output:\n{ex['output']}"
    return {"text": text}


ds = Dataset.from_list(raw).map(format_prompt)
ds = ds.train_test_split(test_size=0.1, seed=42)   # hold out 10% for evaluation
print(ds)

Don't skip three cleaning steps: deduplicate (exact duplicates make the model memorize), filter by length (truncate or split anything over max_length), and validate format (should output carry punctuation, is the JSON valid). See the code comment for minimum sample counts: 500 samples can show results for style/format tasks, domain behavior injection usually needs 2000+, and you must reserve a held-out set the model never sees, otherwise evaluation is meaningless.

4. Pick a Base Model

Three rules for base selection: open-source and commercially usable, strong in your language (for Chinese use cases), and fits your VRAM.

Note: VRAM figures above are rough estimates and depend on sequence length, batch size, and whether gradient checkpointing is on. Model IDs and licenses are subject to the official repo; always confirm the license permits your commercial use case before choosing.

5. Configure LoRA and Load with QLoRA Quantization

Install dependencies, load the 4-bit quantized base, and attach the LoRA adapters:

# pip install peft transformers bitsandbytes accelerate
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

MODEL_ID = "Qwen/Qwen2.5-7B-Instruct"   # check the official repo

# QLoRA: load the base in 4-bit
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)
model = prepare_model_for_kbit_training(model)   # freeze quantized layers, enable grad checkpointing

# LoRA adapter config
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()   # confirm trainable params are <1%

prepare_model_for_kbit_training does several things: it freezes the quantized layers, enables gradient checkpointing, and casts LayerNorm to fp32 for numerical stability. Attaching LoRA to q/k/v/o_proj covers all attention linear projections; adding the MLP gate/up/down_proj is optional and doubles the trainable parameter count in exchange for more capacity.

6. Training

Using transformers.Trainer is the most stable path:

def tokenize(ex):
    tok = tokenizer(
        ex["text"], truncation=True, max_length=2048, padding=False,
    )
    tok["labels"] = tok["input_ids"].copy()
    return tok

train_ds = ds["train"].map(tokenize, remove_columns=ds["train"].column_names)

from transformers import TrainingArguments, Trainer, DataCollatorForSeq2Seq

training_args = TrainingArguments(
    output_dir="./output/lora-run1",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,      # effective batch = 16
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    logging_steps=10,
    save_strategy="epoch",
    bf16=True,
    optim="paged_adamw_8bit",           # QLoRA-specific, saves VRAM
    gradient_checkpointing=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_ds,
    data_collator=DataCollatorForSeq2Seq(tokenizer, padding=True),
)
trainer.train()
model.save_pretrained("./output/lora-adapter")   # only the adapter (tens of MB)

How to tune: epochs is typically 2-5, leaning higher (4-5) for small data while watching for overfitting, and lower (2-3) for larger data. Learning rate between 1e-4 and 3e-4 is the safe zone for LoRA, too high diverges and too low stalls. Batch size is VRAM-bound, so use gradient_accumulation_steps to reach an effective batch of 16-32. TRL's SFTTrainer wraps prompt templating and loss masking and is less work than a hand-rolled Trainer, but its parameters shift between releases, so refer to the official docs. The labels = input_ids.copy() line above computes loss over the entire prompt including the instruction; the stricter approach masks the prompt portion with -100 and only computes loss on the output, which SFTTrainer handles for you by default.

7. Evaluation: Did It Actually Get Better

Fine-tuning without evaluation is blind tuning. The evaluation thinking reuses the framework from our site's RAG Evaluation SOP and AI Agent Evaluation SOP: you must run on the held-out set reserved during train_test_split and look at three layers.

# Run inference on the held-out set and collect predictions
import torch

model.eval()
results = []
for ex in ds["test"]:
    prompt = ex["text"].rsplit("### Output:\n", 1)[0] + "### Output:\n"
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**inputs, max_new_tokens=64, do_sample=False)
    pred = tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
    gold = ex["text"].rsplit("### Output:\n", 1)[1]
    results.append({"pred": pred, "gold": gold})

Three evaluation approaches, pick any:

• Task metrics: format validation (JSON validity rate), classification accuracy, field hit rate. Automate whatever you can.
• LLM-as-judge: use a stronger model as a referee to score each prediction on faithfulness, format correctness, and relevance, using the same prompt template as in the RAG evaluation piece.
• Manual spot-check: sample at least 30 predictions by hand and hunt for bad cases. Metrics alone will miss "fluent but wrong direction" cases.

The critical line: always compare against the un-fine-tuned base model on the same held-out set. If scores don't improve or even drop after fine-tuning, your data or config has a problem, don't ship it.

8. Merge and Deploy

The adapter is an overlay, so deployment usually merges it back into full weights before quantizing for export:

from peft import AutoPeftModelForCausalLM

adapter_path = "./output/lora-adapter"
merged = AutoPeftModelForCausalLM.from_pretrained(
    adapter_path, device_map="auto", torch_dtype=torch.bfloat16,
)
merged = merged.merge_and_unload()   # fold LoRA weights into the base
merged.save_pretrained("./output/merged-model", safe_serialization=True)
tokenizer.save_pretrained("./output/merged-model")

The merged model can be deployed via vLLM, Ollama, or llama.cpp. Ollama uses the GGUF quantized format, while vLLM loads safetensors directly. Deployment cost is mostly VRAM-bound: a 7B model at 4-bit is about 5-6GB, runnable on a single consumer GPU or a 12GB cloud instance. Note that QLoRA trains on a quantized base, but at merge time you should load the original fp16/bf16 weights and merge the adapter into those, not the 4-bit weights, otherwise quantization errors stack.

9. Pitfalls

Pitfall 1: Data quality beats quantity. 1000 noisy samples are worse than 200 carefully labeled ones. Common noise: inconsistent output formatting (some with periods, some without), redundant information across instruction and input, and contradictory samples. Hand-audit 100 samples and only scale up once they all pass.

Pitfall 2: Catastrophic forgetting. Fine-tune too hard and the model loses general ability, nailing ticket classification but unable to hold a normal conversation. Mitigations: keep epochs under 5, learning rate under 3e-4, and mix in a portion of general instruction data as a safety net.

Pitfall 3: Overfitting. Loss keeps dropping but held-out scores stall or fall. The model has memorized training samples and breaks on rephrased inputs. Mitigations: watch held-out metrics and early-stop, lower rank, raise dropout, get more data.

Pitfall 4: Out-of-memory (OOM). Try these in order: drop batch to 1 and compensate with gradient accumulation, enable gradient_checkpointing, switch to QLoRA if you haven't. Long sequences over 2048 tokens are also a VRAM hog, truncate where you can.

Pitfall 5: Only watching training loss, not the eval set. A pretty training curve doesn't mean the model improved, since loss drops during overfitting too. The held-out metric is the only verdict. Don't train without an eval set.

Pitfall 6: Missing target_modules. Attaching only q_proj and skipping v_proj means the adapter can't learn what it needs to, and results suffer. The safe default is to attach all four attention projections (q/k/v/o_proj); when in doubt, attaching more beats attaching less.

FAQ

Q1: What's the real difference between LoRA and full fine-tuning, and when must you go full? LoRA only trains low-rank adapters (under 1% of parameters), saving VRAM, exporting as tens of MB, and allowing hot-swappable adapters. Full fine-tuning updates all weights, with a marginally higher performance ceiling but several to dozens of times the VRAM cost. For the vast majority of style/format/domain adaptation tasks, LoRA is enough. Full fine-tuning is only worth considering when the task requires the model to learn an entirely new capability, you have ample data (tens of thousands plus), and compute is not a constraint. Individuals and small teams default to LoRA/QLoRA.

Q2: How much data is enough? Can a few hundred samples work? Style/format tasks can show results with 500-2000 samples; injecting domain behavior usually needs 2000+. A few hundred can get the format right but won't generalize. Under 200, fall back to prompt few-shot. Quality beats quantity, 200 clean labels beat 2000 noisy ones.

Q3: Which base model, and is 7B enough? For Chinese use cases, Qwen2.5-7B-Instruct is a solid starting point with strong Chinese and lots of community fine-tuning references. Llama is the most mature English ecosystem. 7B is the sweet spot, trainable and deployable on a single GPU under QLoRA. Going too small (under 3B) caps what fine-tuning can achieve, and too large (70B) is out of reach for individuals without cloud compute. Always verify the license permits your commercial use case via the official repo.

Q4: How do I evaluate fine-tuning results, and do I need a framework? Minimum viable: run inference on the held-out set, manually spot-check 30 samples, and compute task metrics (format validity rate, accuracy). For more, use LLM-as-judge to batch-score, following the approach in our RAG Evaluation SOP. The key is to always compare against the un-fine-tuned base on the same set, otherwise you can't prove fine-tuning helped. You don't need a framework like Ragas or DeepEval, a hand-rolled script works.

Q5: What's the deployment cost, and how do I go live after fine-tuning? A 7B model at 4-bit takes about 5-6GB of VRAM, runnable on a single 12GB consumer GPU or a cloud instance costing a few dozen dollars a month. The flow is: merge the LoRA adapter, optionally quantize to GGUF/AWQ, then load with vLLM or Ollama. vLLM has high throughput for online concurrency, Ollama is better for local single-machine use. Training cost depends on duration: a 7B QLoRA run, 3 epochs over 2000 samples on a single 24GB card, finishes in a few hours.

References

• HuggingFace PEFT docs - LoRA
• HuggingFace PEFT docs - QLoRA
• HuggingFace Transformers - BitsAndBytesConfig
• HuggingFace TRL - SFTTrainer
• QLoRA paper: QLoRA: Efficient Finetuning of Quantized LLMs
• LoRA paper: LoRA: Low-Rank Adaptation of Large Language Models