Off Policy Sft Eval

---
name: off-policy-sft-eval
description: Evaluates the trade-off between improving downstream mathematical reasoning capabilities and mitigating catastrophic forgetting on general-domain knowledge benchmarks after off-policy supervised fine-tuning. It measures how well a model retains pre-trained general knowledge while learning a new specialized task. Use when the user wants to benchmark on Math500, MinervaMath, AMC23, AGIEval-Math, IMO-Bench, MMLU, MMLU-Pro, AGIEval, or asks about evaluating this task. Reports OverallAvg.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.11220
  bibtex_key: wang2026patch
  confidence: high
---

# off-policy-sft-eval

> Patch the Distribution Mismatch: RL Rewriting Agent for Stable Off-Policy SFT — Wang et al. (2026) (arXiv:2602.11220, 2026)

## What this evaluates

Evaluates the trade-off between improving downstream mathematical reasoning capabilities and mitigating catastrophic forgetting on general-domain knowledge benchmarks after off-policy supervised fine-tuning. It measures how well a model retains pre-trained general knowledge while learning a new specialized task.

## Datasets

- **Math500** — total ?; splits: test (-1)
- **MinervaMath** — total ?; splits: test (-1)
- **AMC23** — total ?; splits: test (-1)
- **AGIEval-Math** — total ?; splits: test (-1)
- **IMO-Bench** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)
- **MMLU-Pro** — total ?; splits: test (-1)
- **AGIEval** — total ?; splits: test (-1)

## Metrics

- `MathAvg` — range: [0, 1]
  - Arithmetic mean of accuracy scores across all five mathematical reasoning benchmarks.
- `GeneralAvg` — range: [0, 1]
  - Arithmetic mean of accuracy scores across MMLU, MMLU-Pro, and AGIEval (with math-related subsets removed).
- `OverallAvg` **(primary)** — range: [0, 1]
  - Average of MathAvg and GeneralAvg: (MathAvg + GeneralAvg) / 2. Used as the headline metric for the gain-forgetting trade-off.
- `Math↑ (%)` — range: percent
  - Relative improvement in MathAvg over the instruct-tuned base model: (M - M_base) / M_base × 100.
- `Gen↓ (%)` — range: percent
  - Relative drop in GeneralAvg compared to the instruct-tuned base model: (G_base - G) / G_base × 100.

## Input / output format

**Input**: Instruction prompt for mathematical reasoning or general-domain multiple-choice/factual QA tasks.

**Output**: Model-generated answer text or selected option label.

## Scoring recipe

```python
def compute_metrics(preds, golds, base_preds):
    math_scores = [exact_match(p, g) for p, g in zip(preds['math'], golds['math'])]
    general_scores = [exact_match(p, g) for p, g in zip(preds['gen'], golds['gen'])]
    
    math_avg = mean(math_scores)
    general_avg = mean(general_scores)
    overall_avg = (math_avg + general_avg) / 2
    
    math_avg_base = mean([exact_match(p, g) for p, g in zip(base_preds['math'], golds['math'])])
    general_avg_base = mean([exact_match(p, g) for p, g in zip(base_preds['gen'], golds['gen'])])
    
    math_rel_imp = (math_avg - math_avg_base) / math_avg_base * 100
    gen_rel_drop = (general_avg_base - general_avg) / general_avg_base * 100
    return overall_avg, math_rel_imp, gen_rel_drop
```

## Common pitfalls

- Reporting relative improvement/drop percentages (Math↑, Gen↓) instead of absolute scores, which obscures baseline performance differences across model sizes.
- Failing to remove math-related subsets from AGIEval and MMLU when computing GeneralAvg, which artificially inflates the catastrophic forgetting metric.
- Using the pre-training checkpoint instead of the instruct-tuned base model as the reference for relative metric calculations, violating the paper's defined baseline.

## Evidence (verbatim from paper)

> Math↑ denotes the relative MathAvg improvement over the instruct-tuned base within the same block: (M-M_base)/M_base×100. Gen↓ denotes the relative GeneralAvg drop: (G_base-G)/G_base×100. OverallAvg =(MathAvg+GeneralAvg)/2.

## Citation

```bibtex
@misc{wang2026patch,
  title={Patch the Distribution Mismatch: RL Rewriting Agent for Stable Off-Policy SFT},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2602.11220}
}
```

- arXiv: 2602.11220