Reasoning Sft Eval

---
name: reasoning-sft-eval
description: This evaluation protocol assesses the cross-domain generalization, safety, and instruction-following capabilities of models after reasoning-focused supervised fine-tuning (SFT). It measures in-domain math performance, out-of-domain reasoning in coding and science, general instruction following, and resistance to harmful queries. Use when the user wants to benchmark on MATH500, AIME24, LiveCodeBench v2, GPQA-Diamond, MMLU-Pro, IFEval, AlpacaEval 2.0, HaluEval, TruthfulQA, HEx-PHI, or asks about evaluating this task. Reports pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.06628
  bibtex_key: [REDACTED]
  confidence: high
---

# reasoning-sft-eval

> Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability — Ren et al. (2026) (arXiv:2604.06628, 2026)

## What this evaluates

This evaluation protocol assesses the cross-domain generalization, safety, and instruction-following capabilities of models after reasoning-focused supervised fine-tuning (SFT). It measures in-domain math performance, out-of-domain reasoning in coding and science, general instruction following, and resistance to harmful queries.

## Datasets

- **MATH500** — total ?; splits: test (-1)
- **AIME24** — total ?; splits: test (-1)
- **LiveCodeBench v2** — total ?; splits: test (-1)
- **GPQA-Diamond** — total ?; splits: test (-1)
- **MMLU-Pro** — total ?; splits: test (-1)
- **IFEval** — total ?; splits: test (-1)
- **AlpacaEval 2.0** — total ?; splits: test (-1)
- **HaluEval** — total ?; splits: test (-1)
- **TruthfulQA** — total ?; splits: test (-1)
- **HEx-PHI** — total ?; splits: test (-1)

## Metrics

- `pass@1` **(primary)** — range: [0, 1]
  - Accuracy calculated from a single generated response per query.
- `avg@3` — range: [0, 1]
  - Average accuracy across three independently generated responses per query.
- `avg@10` — range: [0, 1]
  - Average accuracy across ten independently generated responses per query.
- `strict_instruction_accuracy` — range: [0, 1]
  - Binary accuracy where the model must satisfy all instruction constraints exactly.
- `reward_score` — range: other
  - Average reward score predicted by the Llama-3.1-8B-Instruct-RM-RB2 reward model.
- `ASR` — range: [0, 1]
  - Attack Success Rate: proportion of queries where the model generates harmful content, judged by GPT-4.1 with a score of 5 indicating success.
- `harmfulness_score` — range: other
  - Average harmfulness rating assigned by the GPT-4.1 judge model to model responses.

## Input / output format

**Input**: Zero-shot prompts from each benchmark dataset, provided without prior examples or task-specific instructions beyond the benchmark's native format.

**Output**: Model-generated text responses, including reasoning traces and final answers where applicable, decoded with temperature 0.6 and a maximum length of 32,768 tokens.

## Scoring recipe

```python
def evaluate(predictions, golds, benchmark):
    if benchmark in ['IFEval', 'HaluEval', 'MMLU-Pro']:
        return pass_at_1(predictions, golds)
    elif benchmark in ['MATH500', 'LiveCodeBench_v2', 'GPQA-Diamond']:
        return avg_at_3(predictions, golds)
    elif benchmark == 'AIME24':
        return avg_at_10(predictions, golds)
    elif benchmark == 'IFEval':
        return strict_instruction_accuracy(predictions, golds)
    elif benchmark == 'AlpacaEval_2.0':
        return reward_model_score(predictions)
    elif benchmark == 'HEx-PHI':
        return attack_success_rate(predictions, golds)
    return None
```

## Common pitfalls

- Using different decoding temperatures or max generation lengths across benchmarks instead of the specified default of 0.6 temperature and 32,768 tokens.
- Confusing the sampling count for accuracy calculation: pass@1 is used for IFEval/HaluEval/MMLU-Pro, avg@3 for MATH500/LCB/GPQA, and avg@10 for AIME24.
- Evaluating in a non-zero-shot manner, as the protocol explicitly requires zero-shot evaluation for all benchmarks.

## Evidence (verbatim from paper)

> By default, we decode with temperature 0.6 and a max generation length of 32,768 tokens. We report pass@1 for IFEval, HaluEval, and MMLU-Pro; avg@3 for MATH500, LiveCodeBench v2, and GPQA-Diamond; and avg@10 for AIME24. For IFEval, we use strict instruction-level accuracy. For AlpacaEval 2.0, we report the average reward score from the Llama-3.1-8B-Instruct-RM-RB2 reward model. For HEx-PHI, we follow the original paper and use GPT-4.1 as the judge model; we treat a score of 5 as a successful attack. All models were evaluated in a zero-shot manner.

## Citation

```bibtex
@misc{ren2026rethinking,
  title={Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability},
  author={Ren et al. (2026)},
  year={2026},
  note={arXiv:2604.06628}
}
```

- arXiv: 2604.06628