441 Lora Peft Techniques 2ffa441f

---
name: ml-lora-peft-techniques
description: Fine-tuning large models with limited GPU memory
---


# LoRA and PEFT Techniques

**Scope**: LoRA, QLoRA, adapter tuning, rank selection, merging, multi-adapter inference
**Lines**: ~340
**Last Updated**: 2025-10-18

## When to Use This Skill

Activate this skill when:
- Fine-tuning large models with limited GPU memory
- Training multiple task-specific adapters from same base model
- Reducing training time and storage costs
- Merging multiple LoRA adapters for multi-task models
- Optimizing fine-tuning hyperparameters (rank, alpha)
- Deploying memory-efficient models in production

## Core Concepts

### Parameter-Efficient Fine-Tuning (PEFT)

**What is PEFT**:
- Train only a small subset of model parameters
- Freeze base model weights
- Add small trainable "adapter" layers
- Achieves 99%+ of full fine-tuning quality
- Uses 10-100x less memory and storage

**PEFT Methods**:
- **LoRA**: Low-Rank Adaptation (most popular)
- **QLoRA**: Quantized LoRA (4-bit base model)
- **Prefix Tuning**: Learn task-specific prompts
- **Adapter Layers**: Small bottleneck layers
- **IA3**: Learned rescaling vectors

### LoRA (Low-Rank Adaptation)

**How LoRA Works**:
- Freezes original model weights W
- Adds trainable low-rank matrices A and B
- Updates: ΔW = B × A (where B is m×r, A is r×n)
- Final output: W + ΔW = W + BA
- Only trains A and B (<<< original parameters)

**Key Hyperparameters**:
- **Rank (r)**: Dimensionality of low-rank matrices (4-128)
- **Alpha**: Scaling factor for LoRA updates (usually = r)
- **Dropout**: Regularization (0-0.1, often 0 for LoRA)
- **Target Modules**: Which layers to apply LoRA to

**Memory Savings**:
- 7B model: ~28GB → ~6GB (4-bit QLoRA)
- 13B model: ~52GB → ~10GB (4-bit QLoRA)
- 70B model: ~280GB → ~48GB (4-bit QLoRA)

### QLoRA (Quantized LoRA)

**QLoRA Components**:
- 4-bit NormalFloat (NF4) quantization of base model
- Double quantization for constants
- Paged optimizers for memory efficiency
- LoRA adapters trained in bfloat16

**Benefits**:
- Train 70B models on single 48GB GPU
- Minimal quality loss vs full precision
- Faster training than 16-bit LoRA

---

## Patterns

### Basic LoRA Configuration

```python
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

# Load base model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3-8b",
    device_map="auto",
    torch_dtype="auto"
)

# Configure LoRA
lora_config = LoraConfig(
    r=16,  # Rank
    lora_alpha=16,  # Alpha (scaling)
    target_modules=[
        "q_proj",  # Query projection
        "k_proj",  # Key projection
        "v_proj",  # Value projection
        "o_proj",  # Output projection
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

# Apply LoRA
model = get_peft_model(model, lora_config)

# Print trainable parameters
model.print_trainable_parameters()
# Output: trainable params: 4,194,304 || all params: 8,030,261,248 || trainable%: 0.05%
```

### QLoRA with 4-bit Quantization

```python
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model
import torch

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",  # NormalFloat4
    bnb_4bit_compute_dtype=torch.bfloat16,  # Compute in bf16
    bnb_4bit_use_double_quant=True,  # Double quantization
)

# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3-8b",
    quantization_config=bnb_config,
    device_map="auto",
)

# Prepare for k-bit training
model = prepare_model_for_kbit_training(model)

# Apply LoRA
lora_config = LoraConfig(
    r=64,  # Higher rank for 4-bit to compensate
    lora_alpha=128,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"  # MLP layers
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)
```

### Target Module Selection

```python
# Attention-only (memory efficient)
target_modules = ["q_proj", "v_proj"]

# All attention (good balance)
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj"]

# Attention + MLP (best quality)
target_modules = [
    "q_proj", "k_proj", "v_proj", "o_proj",
    "gate_proj", "up_proj", "down_proj"
]

# Auto-detect all linear layers
import re

def find_all_linear_names(model):
    """Find all linear layer names in model."""
    cls = torch.nn.Linear
    lora_module_names = set()

    for name, module in model.named_modules():
        if isinstance(module, cls):
            names = name.split('.')
            lora_module_names.add(names[-1])

    # Remove output layer
    if "lm_head" in lora_module_names:
        lora_module_names.remove("lm_head")

    return list(lora_module_names)

target_modules = find_all_linear_names(model)
```

### Rank Selection Strategy

```python
# Small models (<3B) or simple tasks
lora_config = LoraConfig(r=8, lora_alpha=16)

# Medium models (7-13B) general tasks
lora_config = LoraConfig(r=16, lora_alpha=32)

# Large models (30-70B) or complex tasks
lora_config = LoraConfig(r=64, lora_alpha=128)

# Quality vs efficiency tradeoff
ranks = [8, 16, 32, 64]
for r in ranks:
    print(f"Rank {r}:")
    print(f"  Trainable params: ~{r * 2 * 4096 * 32 / 1e6:.1f}M")
    print(f"  Storage: ~{r * 2 * 4096 * 32 * 2 / 1e6:.1f}MB")
```

### Saving and Loading LoRA Adapters

```python
from peft import PeftModel

# Save LoRA adapter only (~100MB for r=16)
model.save_pretrained("./lora-adapters")

# Load LoRA adapter
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3-8b",
    device_map="auto"
)

model = PeftModel.from_pretrained(
    base_model,
    "./lora-adapters"
)

# Merge adapter into base model (for deployment)
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./merged-model")

# Push to Hub (adapter only)
model.push_to_hub("username/lora-adapter")

# Push merged model
merged_model.push_to_hub("username/merged-model")
```

### Multi-Adapter Inference

```python
from peft import PeftModel

# Load base model once
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3-8b",
    device_map="auto"
)

# Load first adapter
model = PeftModel.from_pretrained(
    base_model,
    "adapters/task1",
    adapter_name="task1"
)

# Add second adapter
model.load_adapter("adapters/task2", adapter_name="task2")

# Add third adapter
model.load_adapter("adapters/task3", adapter_name="task3")

# Switch between adapters
model.set_adapter("task1")
output1 = model.generate(**inputs)

model.set_adapter("task2")
output2 = model.generate(**inputs)

# Disable all adapters (use base model)
model.disable_adapters()
output_base = model.generate(**inputs)
```

### Merging Multiple Adapters

```python
from peft import PeftModel

# Load base and first adapter
model = PeftModel.from_pretrained(
    base_model,
    "adapters/task1",
    adapter_name="task1"
)

# Load additional adapters
model.load_adapter("adapters/task2", adapter_name="task2")
model.load_adapter("adapters/task3", adapter_name="task3")

# Merge adapters with weights
model.add_weighted_adapter(
    adapters=["task1", "task2", "task3"],
    weights=[0.5, 0.3, 0.2],
    adapter_name="merged",
    combination_type="linear"  # or "cat" for concatenation
)

# Save merged adapter
model.save_pretrained("./merged-adapter")
```

### Training with PEFT

```python
from transformers import Trainer, TrainingArguments
from peft import get_peft_model, LoraConfig

# Configure LoRA
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

# Apply to model
model = get_peft_model(model, lora_config)

# Training arguments
training_args = TrainingArguments(
    output_dir="./lora-output",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,  # Higher LR for LoRA
    num_train_epochs=3,
    logging_steps=10,
    save_strategy="epoch",
    fp16=True,
)

# Train
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
)

trainer.train()

# Save LoRA adapter
model.save_pretrained("./lora-final")
```

### Modal.com Deployment

```python
import modal

app = modal.App("lora-finetune")

image = (
    modal.Image.debian_slim()
    .pip_install(
        "torch",
        "transformers",
        "peft",
        "bitsandbytes",
        "accelerate"
    )
)

@app.function(
    gpu="l40s",
    image=image,
    timeout=3600
)
def train_lora(base_model: str, dataset_path: str, output_path: str):
    from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
    from peft import LoraConfig, get_peft_model
    from datasets import load_dataset
    import torch

    # Load model
    model = AutoModelForCausalLM.from_pretrained(
        base_model,
        device_map="auto",
        torch_dtype=torch.bfloat16
    )

    tokenizer = AutoTokenizer.from_pretrained(base_model)

    # Configure LoRA
    lora_config = LoraConfig(
        r=16,
        lora_alpha=32,
        target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
        lora_dropout=0.05,
        bias="none",
        task_type="CAUSAL_LM"
    )

    model = get_peft_model(model, lora_config)

    # Load dataset
    dataset = load_dataset("json", data_files=dataset_path)

    # Train
    training_args = TrainingArguments(
        output_dir=output_path,
        per_device_train_batch_size=4,
        num_train_epochs=3,
        learning_rate=2e-4,
        logging_steps=10,
        save_strategy="epoch",
        bf16=True,
    )

    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=dataset["train"],
    )

    trainer.train()

    # Save adapter
    model.save_pretrained(output_path)
    tokenizer.save_pretrained(output_path)

    return {"status": "complete", "path": output_path}
```

---

## Quick Reference

### LoRA Rank Guidelines

```
Rank (r) | Trainable Params | Storage  | Use Case
---------|------------------|----------|---------------------------
4-8      | ~1-2M            | ~10MB    | Simple tasks, proof-of-concept
16       | ~4M              | ~50MB    | General fine-tuning
32       | ~8M              | ~100MB   | Complex tasks
64       | ~16M             | ~200MB   | Very complex, QLoRA compensation
128+     | ~32M+            | ~400MB+  | Experimental, diminishing returns
```

### Target Module Recommendations

```
Configuration           | Modules                          | Quality | Memory
------------------------|----------------------------------|---------|--------
Minimal                 | q_proj, v_proj                   | Good    | Low
Recommended             | q_proj, k_proj, v_proj, o_proj   | Better  | Medium
Full Attention + MLP    | All attention + gate/up/down     | Best    | High
```

### Alpha Selection

```python
# Common patterns
lora_alpha = r          # 1:1 ratio (standard)
lora_alpha = r * 2      # 2:1 ratio (stronger updates)
lora_alpha = 32         # Fixed at 32 (some practitioners prefer)

# Alpha / r ratio controls update magnitude
# Higher ratio = stronger LoRA influence
```

### PEFT Methods Comparison

```
Method      | Memory  | Quality | Speed | Use Case
------------|---------|---------|-------|----------------------------
LoRA        | Low     | High    | Fast  | General fine-tuning
QLoRA       | Very Low| High    | Fast  | Large models, limited VRAM
Prefix      | Low     | Medium  | Fast  | Prompt engineering
Adapter     | Medium  | High    | Medium| Task-specific modules
Full FT     | High    | Highest | Slow  | Maximum quality, unlimited compute
```

---

## Anti-Patterns

❌ **Rank too low**: Poor adaptation to task
✅ Start with r=16, increase if quality insufficient

❌ **Rank too high**: Overfitting, wasted compute
✅ Rarely need r>64 even for complex tasks

❌ **Wrong alpha/r ratio**: Unstable training
```python
# ❌ Bad: Alpha much larger than rank
lora_config = LoraConfig(r=8, lora_alpha=128)

# ✅ Good: Alpha = r or 2*r
lora_config = LoraConfig(r=16, lora_alpha=32)
```

❌ **Training all layers**: Negates memory benefits
✅ Target only attention or attention+MLP

❌ **Not using gradient checkpointing**: OOM on large models
```python
# ✅ Enable gradient checkpointing
model.gradient_checkpointing_enable()
```

❌ **Merging without testing adapters separately**: Can't debug
✅ Test individual adapters before merging

❌ **Using LoRA for small models with plenty VRAM**: Unnecessary
✅ Use full fine-tuning for models <3B if VRAM allows

❌ **Not saving adapter separately**: Lose flexibility
✅ Save adapter before merging for reusability

❌ **Ignoring base model precision**: Quality/memory mismatch
```python
# ❌ Bad: 4-bit model but fp32 LoRA
# ✅ Good: Match compute dtype
bnb_config = BitsAndBytesConfig(
    bnb_4bit_compute_dtype=torch.bfloat16
)
```

---

## Related Skills

- `unsloth-finetuning.md` - Fast LoRA training with Unsloth
- `modal-gpu-workloads.md` - GPU selection for PEFT training
- `llm-dataset-preparation.md` - Preparing data for fine-tuning
- `huggingface-autotrain.md` - Automated LoRA training
- `model-merging.md` - Advanced adapter merging techniques
- `quantization-techniques.md` - Model quantization deep dive

---

**Last Updated**: 2025-10-18
**Format Version**: 1.0 (Atomic)