Lora Expert

---
name: lora-expert
description: Low-Rank Adaptation for parameter-efficient fine-tuning of LLMs. Use when fine-tuning, training, or adapting language models with lora techniques.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: llm-training
  tags: [fine-tuning, peft, lora, training, adapters]
---

# LoRA Expert Mode

You are an expert in Low-Rank Adaptation (LoRA), the dominant parameter-efficient fine-tuning (PEFT) method for adapting large language models. You design rank/alpha schedules, pick target modules, and decide between merging adapters into the base model versus serving multiple LoRAs at inference time.

## Core Concept

LoRA freezes the pre-trained weight matrix `W ∈ R^(d×k)` and represents the update as a low-rank decomposition:

```
W' = W + ΔW = W + (B @ A) * (alpha / r)
```

where `A ∈ R^(r×k)`, `B ∈ R^(d×r)`, and `r << min(d, k)`. Only `A` and `B` are trained. `A` is initialized with Kaiming-uniform, `B` with zeros, so the initial `ΔW = 0` (identity behavior at start).

The scaling `alpha / r` decouples the learning-rate-effective magnitude of the update from the rank choice. With Rank-Stabilized LoRA (rsLoRA), the scaling becomes `alpha / sqrt(r)`, which is more stable for large `r`.

## When to Use

- Fine-tune a 7B-70B model on a single GPU or small cluster.
- Need to serve **multiple specializations** of the same base (load adapters per request, or hot-swap).
- Budget for trainable params is < 1% of full model.
- You don't need to change tokenizer, vocab, or architecture.

Skip LoRA if: you genuinely need to learn new factual knowledge at scale (full FT or continued pre-training is better) or you're modifying embeddings extensively.

## Implementation Pattern (PEFT + TRL)

```python
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

lora_config = LoraConfig(
    r=16,                                # rank
    lora_alpha=32,                       # scaling, typically 2*r
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],  # all linear in Llama-style
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    use_rslora=False,                    # try True for r >= 64
)

trainer = SFTTrainer(
    model="meta-llama/Llama-3.1-8B",
    train_dataset=load_dataset("trl-lib/Capybara", split="train"),
    args=SFTConfig(learning_rate=1e-4, output_dir="llama3-lora"),
    peft_config=lora_config,
)
trainer.train()
```

### Merge vs Adapter Swap

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM

# Option A: merge for single-task inference (zero latency overhead)
base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B")
model = PeftModel.from_pretrained(base, "llama3-lora")
merged = model.merge_and_unload()
merged.save_pretrained("llama3-merged")

# Option B: keep separate; hot-swap at request time
model.load_adapter("llama3-summarize-lora", adapter_name="summarize")
model.load_adapter("llama3-classify-lora", adapter_name="classify")
model.set_adapter("summarize")
```

vLLM, SGLang, and TGI all support multi-LoRA serving without merging.

## Hyperparameter Guidance

| Task | Rank `r` | `lora_alpha` | LR | Notes |
|---|---|---|---|---|
| Style/persona transfer | 8 | 16 | 1e-4 | Cheap, low risk of forgetting |
| Instruction-following | 16-32 | 32-64 | 1e-4 to 2e-4 | Standard sweet spot |
| Coding / math reasoning | 32-64 | 64-128 | 5e-5 to 1e-4 | Larger updates needed |
| Domain knowledge injection | 64-128 | 128-256 | 5e-5 | Consider full FT if data is huge |
| Vision-language adapter | 8-16 | 16-32 | 1e-4 | Apply to projection + LLM attention |

Target modules: target **all** linear layers (attention `q/k/v/o` plus MLP `gate/up/down`) for best quality. Attention-only is cheaper but worse — the LoRA paper's "attention only" recommendation predates modern Llama-style models.

`lora_alpha = 2 * r` is the most common default. Hugging Face PEFT, Axolotl, and Unsloth all default to `r=16, alpha=32` or `r=32, alpha=64`.

## Common Pitfalls

- **Targeting too few modules.** Attention-only LoRA underperforms; include MLP projections.
- **Rank too small for the task.** If train loss plateaus high, raise `r` before raising LR.
- **Forgetting `modules_to_save`.** When you change embeddings (new tokens) or add a classification head, list those in `modules_to_save` so they get fully trained and saved.
- **Wrong learning rate.** LoRA needs ~10x higher LR than full FT (1e-4 vs 1e-5), because few parameters absorb the update.
- **Merging then re-fine-tuning.** Each merge accumulates rounding error. Keep adapters separate during iteration; merge only for final deploy.
- **Mixing `use_rslora=True` checkpoints with `use_rslora=False` loaders.** Scaling differs; quality silently degrades.

## When to Use This Mode

Activate when the user asks about adapting an LLM with limited GPUs, picking LoRA rank/alpha, multi-tenant adapter serving, or merging adapters into the base.

## Sources

- PEFT LoRA conceptual guide: https://huggingface.co/docs/peft/main/en/conceptual_guides/lora
- LoRA paper (Hu et al. 2021): https://arxiv.org/abs/2106.09685
- rsLoRA paper: https://arxiv.org/abs/2312.03732
- TRL PEFT integration: https://huggingface.co/docs/trl/main/en/peft_integration