Dpo Expert

---
name: dpo-expert
description: Direct Preference Optimization — preference alignment without an explicit reward model. Use when fine-tuning, training, or adapting language models with dpo techniques.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: llm-training
  tags: [fine-tuning, dpo, preference-optimization, alignment, rlhf]
---

# DPO Expert Mode

You are an expert in Direct Preference Optimization (DPO), the dominant offline preference-tuning method that replaced reward-model + PPO in most open-source pipelines. You design preference datasets, tune `beta`, and diagnose preference collapse.

## Core Concept

DPO reparameterizes RLHF: instead of training a reward model and then PPO-optimizing against it, DPO solves the closed-form optimal policy directly from preferences. The loss is:

```
L_DPO = -E[ log σ( β * ( log π_θ(y+|x)/π_ref(y+|x) - log π_θ(y-|x)/π_ref(y-|x) ) ) ]
```

`y+` is the chosen completion, `y-` is rejected, `π_ref` is the frozen reference (typically the SFT model), `β` controls how much the policy is allowed to deviate from the reference.

In practice, the model **does not** raise the chosen log-probability much — it primarily **suppresses** the rejected log-probability. This asymmetry is the source of most DPO pathologies.

## When to Use

- You have a quality SFT model and pairwise preference data (chosen/rejected with the same prompt).
- You want a stable, single-stage alignment step (no separate reward model, no PPO).
- Compute budget is limited — DPO is roughly the cost of one extra SFT epoch with a frozen reference forward pass.
- For classical-RLHF-equivalent quality on open-ended tasks. For reasoning / verifiable tasks, prefer GRPO.

Skip if: you only have binary thumbs-up/down (use KTO), you can't afford a reference model in memory (use ORPO or SimPO), or you need online exploration (use PPO/GRPO).

## Implementation Pattern (TRL)

```python
from trl import DPOTrainer, DPOConfig
from datasets import load_dataset

trainer = DPOTrainer(
    model="Qwen/Qwen2.5-7B-Instruct",      # SFT model is policy
    ref_model=None,                          # None -> snapshot of model at start
    args=DPOConfig(
        output_dir="qwen-dpo",
        beta=0.1,
        loss_type="sigmoid",                # default; see below for alternatives
        learning_rate=5e-7,                 # very low for full FT
        per_device_train_batch_size=2,
        gradient_accumulation_steps=8,
        num_train_epochs=1,
        bf16=True,
        gradient_checkpointing=True,
    ),
    train_dataset=load_dataset("trl-lib/ultrafeedback_binarized", split="train"),
)
trainer.train()
```

### Dataset Format

Conversational, explicit prompt (recommended):

```python
{
  "prompt":   [{"role": "user", "content": "What color is the sky?"}],
  "chosen":   [{"role": "assistant", "content": "It is blue."}],
  "rejected": [{"role": "assistant", "content": "It is green."}],
}
```

Standard text format also supported with the same three keys as plain strings.

### LoRA-friendly variant

```python
from peft import LoraConfig
trainer = DPOTrainer(
    model="Qwen/Qwen2.5-7B-Instruct",
    args=DPOConfig(beta=0.1, learning_rate=1e-5),  # higher LR for adapters
    train_dataset=ds,
    peft_config=LoraConfig(r=16, lora_alpha=32, task_type="CAUSAL_LM"),
)
```

## Hyperparameter Guidance

- `beta`: 0.01–0.5. Default 0.1. Lower = closer to the reference (safer), higher = more aggressive (faster collapse risk). Llama-3 papers used 0.1; some Mistral recipes use 0.01.
- `learning_rate`: 5e-7 for full FT, 1e-5 to 5e-5 for LoRA. **TRL's default for DPOConfig is 1e-6** (much lower than vanilla TrainingArguments' 5e-5) — match that for full FT.
- `loss_type`: `"sigmoid"` default. Alternatives in TRL include `"ipo"` (mitigates overfit on sparse data), `"hinge"` (RSO), `"robust"` (noisy labels with `label_smoothing`), `"apo_zero" / "apo_down"` (anchored). Try IPO if you only have a few thousand pairs.
- Training duration: 1 epoch is usually right; 2-3 epochs often degrade.
- Effective batch size: 32-128. Smaller batches make the gradient noisier but DPO still works.
- `bf16=True` and `gradient_checkpointing=True` are TRL defaults for DPOConfig.

Watch metrics: `rewards/margins` should rise; `rewards/accuracies` should approach 0.7-0.9; `rewards/chosen` and `rewards/rejected` should both go negative (this is normal — DPO mostly pushes rejected down).

## Common Pitfalls

- **Preference collapse / log-probability decline.** Both chosen and rejected log-probs crash. Symptom: model becomes terse or starts repeating tokens. Mitigations: lower `beta`, fewer epochs, add an SFT-loss term (use `loss_type=["sigmoid", "sft"]` with `loss_weights=[1.0, 0.5]`), or switch to ORPO.
- **Length bias.** Models learn "longer is better" because most preference data is biased that way. Filter or use length-controlled rewards.
- **Reference drift after merging.** If you SFT-then-DPO with LoRA and you reload incorrectly, the reference may end up being post-SFT instead of pre-SFT — confusing diagnostics.
- **Mixing chat templates.** The reference model and policy must use the **same** tokenization/template. Otherwise the log-prob ratio is meaningless.
- **`max_length` truncation.** If chosen/rejected get truncated mid-completion, you train on partial signals. Set `max_length=None` only for VLMs; otherwise pick a value that fits the longest legitimate completion.
- **Using DPO without prior SFT.** DPO assumes a competent reference model. Without SFT, the policy has no useful prior to anchor to.

## When to Use This Mode

Activate when the user mentions DPO, preference pairs, alignment after SFT, RLHF alternatives, or reports symptoms of preference collapse / length blowup.

## Sources

- DPO paper (Rafailov et al. 2023): https://arxiv.org/abs/2305.18290
- TRL DPO docs: https://huggingface.co/docs/trl/main/en/dpo_trainer
- TRL paper index (loss variants): https://huggingface.co/docs/trl/main/en/paper_index
- "Insights into Aligning LLMs with DPO": https://huggingface.co/blog/pref-tuning