Code Reasoning Eval

---
name: code-reasoning-eval
description: This evaluation probes a model's ability to solve programming contest problems and reason through mathematics and science questions. It measures how well fine-tuning on synthesized data preserves or enhances in-domain coding capabilities while maintaining out-of-domain generalization across multiple reasoning benchmarks. Use when the user wants to benchmark on LiveCodeBench-V5, LiveCodeBench-V6, LiveCodeBench-Pro, OJBench, AIME-2024, AIME-2025, OlympiadBench, GPQA, or asks about evaluating this task. Reports pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.14164
  bibtex_key: huang2026tessy
  confidence: high
---

# code-reasoning-eval

> How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data — Huang et al. (2026) (arXiv:2604.14164, 2026)

## What this evaluates

This evaluation probes a model's ability to solve programming contest problems and reason through mathematics and science questions. It measures how well fine-tuning on synthesized data preserves or enhances in-domain coding capabilities while maintaining out-of-domain generalization across multiple reasoning benchmarks.

## Datasets

- **LiveCodeBench-V5** — total ?; splits: test (-1)
- **LiveCodeBench-V6** — total ?; splits: test (-1)
- **LiveCodeBench-Pro** — total ?; splits: test (-1)
- **OJBench** — total ?; splits: test (-1)
- **AIME-2024** — total ?; splits: test (-1)
- **AIME-2025** — total ?; splits: test (-1)
- **OlympiadBench** — total ?; splits: test (-1)
- **GPQA** — total ?; splits: test (-1)

## Metrics

- `pass@1` **(primary)** — range: percent
  - Computed as the average pass rate across multiple independent generation runs. For code tasks, a run passes if the generated code solves the problem (e.g., passes hidden test cases). For math/science, it checks exact match or logical equivalence against the gold answer.

## Input / output format

**Input**: Text-only programming contest questions, mathematics problems, and science QA prompts.

**Output**: Model-generated reasoning traces and final answers/code solutions.

## Scoring recipe

```python
def compute_pass_at_1(predictions_per_run, gold_answers):
    run_pass_rates = []
    for run_preds in predictions_per_run:
        correct = sum(1 for p in run_preds if check_solution(p, gold_answers))
        run_pass_rates.append(correct / len(run_preds))
    return sum(run_pass_rates) / len(run_pass_rates) * 100
```

## Common pitfalls

- pass@1 must be averaged over multiple independent generation runs rather than calculated from a single greedy or sampled output.
- For multimodal evaluation datasets, only the text-only components are used; visual inputs are stripped before evaluation.
- Training on fully teacher-generated data often causes performance degradation on the student model, contrary to naive distillation assumptions.

## Evidence (verbatim from paper)

> All evaluations were conducted on the OpenCompass *[opencompass]* platform, where pass@1 is computed as the average of results from multiple independent runs.

## Citation

```bibtex
@misc{huang2026tessy,
  title={How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data},
  author={Huang et al. (2026)},
  year={2026},
  note={arXiv:2604.14164}
}
```

- arXiv: 2604.14164