AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

lora-qlora-recipes

Configure LoRA and QLoRA supervised fine-tuning with current best-practice hyperparameters. Use when writing or reviewing a LoRA/QLoRA training configuration, choosing rank/alpha/target modules, or deciding between LoRA, QLoRA, and full fine-tuning.

Original source
wshobson
License
MIT
Pinned revision
7e2fb559cf4b824b27f1fec49fe88f7511066d26

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read wshobson--lora-qlora-recipes at revision 7e2fb559cf4b824b27f1fec49fe88f7511066d26, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 3

references/hyperparameters.md

Last verified: 2026-07-13

LoRA/QLoRA Hyperparameter Tables

Full tables and a complete worked config backing the summary in SKILL.md. Base models are never named here — every example is labeled by size class only; see finetuning-method-selection's references/model-catalog.md for which actual model to use at a given size class.

Rank and Alpha by Task Type

lora_alpha = 2 * r in every row — derive alpha from rank, don't set it independently.

| Task type | Rank (r) | lora_alpha | Notes | |---|---|---|---| | RL adapters (GRPO/RLVR) | 1–32 | 2–64 | Lower end (1–8) is common for adapters on top of an already-capable base. | | General SFT default | 16–32 | 32–64 | Starting point absent a specific reason to go higher or lower. | | SFT at scale (large, diverse instruction sets) | up to ~256 | up to ~512 | Only justified when the dataset is large and diverse enough to use the extra capacity — see rsLoRA note below before defaulting here. |

Learning Rate by Method

LoRA/QLoRA learning rates run roughly 10x the equivalent full-fine-tune LR — this is the single most common misconfiguration when porting a full- FT config to LoRA (leaving the LR unchanged under-trains the adapter).

| Method | LR range | Use when | |---|---|---| | QLoRA (standard) | 2e-4 | Default starting point for QLoRA SFT. | | LoRA, conservative | 1e-4 | Larger base model, higher rank, or a run that showed instability at 2e-4. | | LoRA, very conservative | 5e-5 | Continuing a run, fine-grained behavior adjustment, or a base model that's already close to the target behavior. |

Treat these as starting points to sweep around, not fixed constants — but start here rather than porting a full-FT LR unchanged.

rsLoRA Note

Rank-stabilized LoRA (rsLoRA) rescales the adapter update by alpha / sqrt(r) instead of alpha / r. It's optional, and only worth turning on at r ≥ 32 — below that rank, the standard scaling (alpha / r) is stable enough that rsLoRA doesn't change outcomes meaningfully. If the SFT-at-scale row (rank up to ~256) is in play, turn rsLoRA on; for the general-default or RL rows, leave it off unless a specific instability shows up.

Batch and Packing Interactions

  • Keep effective batch size under 32 — the reference recipe was validated at that scale. Effective batch is per_device_batch_size * gradient_accumulation_steps * num_devices; a multi-GPU or high-accumulation setup can cross 32 without the per-device batch size looking large, so compute the product, not just the per-device number.
  • Packing multiple short examples into one sequence changes the effective batch's token composition, not just its example count — a packed batch of 8 sequences is not equivalent to an unpacked batch of 8 short examples. Apply the chat template before packing, not after, and spot-check a handful of decoded packed sequences before trusting the pipeline.
  • Gradient checkpointing (use_gradient_checkpointing="unsloth") and packing both trade compute for memory independently — enabling both is normal for a memory-constrained run, not redundant.

Worked Config: Unsloth FastLanguageModel + SFTConfig

A complete, internally consistent config at the general-default rank (r=32), QLoRA, standard LR. Swap BASE_MODEL for an actual checkpoint from the model catalog before running.

from unsloth import FastLanguageModel
from trl import SFTConfig, SFTTrainer

BASE_MODEL = "<from model catalog>"  # size class + task decide this, not this file

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=BASE_MODEL,
    max_seq_length=2048,
    dtype=None,          # auto-detect bf16/fp16 by hardware
    load_in_4bit=True,   # QLoRA path — set False for bf16 LoRA
)

target_modules = [
    "q_proj", "k_proj", "v_proj", "o_proj",
    "gate_proj", "up_proj", "down_proj",
]

model = FastLanguageModel.get_peft_model(
    model,
    r=32,
    target_modules=target_modules,
    lora_alpha=64,                    # 2 * r
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
    use_rslora=False,                 # r=32 threshold — leave disabled here unless instability is observed
)

import torch

# Check hardware BF16 support before forcing it — see SKILL.md
# Failure Modes. Training in fp16 on hardware without solid BF16
# support is a known source of loss spikes and silent divergence,
# so this is a hard prerequisite, not a config style choice.
if not torch.cuda.is_bf16_supported():
    raise RuntimeError(
        "This GPU does not support BF16 — do not fall back to "
        "fp16=True as if it were equivalent; pick hardware with "
        "BF16 support instead (see SKILL.md Failure Modes)."
    )

training_args = SFTConfig(
    output_dir="./outputs",
    max_length=2048,
    dataset_text_field="text",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,    # effective batch 16 (single device) — stays under 32
    learning_rate=2e-4,               # QLoRA standard
    bf16=True,                        # gated above — never fp16, see SKILL.md Failure Modes
    optim="adamw_8bit",
    num_train_epochs=3,
    logging_steps=10,
    seed=3407,
)

trainer = SFTTrainer(
    model=model,
    processing_class=tokenizer,       # current TRL — not tokenizer=
    train_dataset=train_dataset,
    args=training_args,
)

trainer.train()

This block is internally consistent: r=32 → lora_alpha=64 (2x rule), load_in_4bit=True → learning_rate=2e-4 (QLoRA standard LR), bf16=True (never fp16), and effective batch 4 * 4 = 16 (under the 32 ceiling). Changing any one of these — rank, quantization, or batch shape — should trigger rechecking the others against the tables above rather than editing it in isolation.