Resyn Eval

---
name: resyn-eval
description: Evaluates the reasoning capabilities of language models on synthetically generated, code-verifiable tasks. It measures performance on both the custom ReSyn dataset and standard reasoning benchmarks using zero-shot generation with specific sampling parameters. Use when the user wants to benchmark on ReSyn, or asks about evaluating this task. Reports mean@4.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.20117
  bibtex_key: he2026resyn
  confidence: high
---

# resyn-eval

> ReSyn: Autonomously Scaling Synthetic Environments for Reasoning Models — He et al. (2026) (arXiv:2602.20117, 2026)

## What this evaluates

Evaluates the reasoning capabilities of language models on synthetically generated, code-verifiable tasks. It measures performance on both the custom ReSyn dataset and standard reasoning benchmarks using zero-shot generation with specific sampling parameters.

## Datasets

- **ReSyn** — total 16500; splits: train (16000), val (500)

## Metrics

- `mean@4` **(primary)** — range: [0, 1]
  - Pass@k accuracy: the fraction of test instances where at least one of the top-k generated samples is judged correct by the verifier.
- `mean@128` — range: [0, 1]
  - Pass@k accuracy calculated over 128 generated samples per instance, used specifically for the AIME 2024 benchmark.

## Input / output format

**Input**: Natural language question Q, optionally prefixed with instructions to structure output using <think> and <answer> tags.

**Output**: Candidate solutions a_i containing intermediate reasoning inside <think>…</think> tags and a final answer inside <answer>…</answer> tags.

## Scoring recipe

```python
def compute_mean_k(predictions, verifiers, k=4):
    correct = 0
    for q, verifier in verifiers:
        answers = extract_answers(predictions[q])
        for ans in answers[:k]:
            if verifier(ans):
                correct += 1
                break
    return correct / len(verifiers)
```

## Common pitfalls

- Confusing the training reward signal (format score × answer score) with the zero-shot evaluation metric (pass@k accuracy).
- Assuming verifiers are always LLM-based; the paper specifies they are implemented in code, though LLM verifiers may occasionally error (assigned 0).
- Overlooking that evaluation is strictly zero-shot with temperature 0.8 and top-p 0.95, not fine-tuned or SFT.

## Evidence (verbatim from paper)

> Table 1: Evaluation of ReSyn versus the base Instruct model across reasoning and math benchmarks. All benchmarks are evaluated in zero-shot conditions using temperature 0.8 and top-p 0.95 sampling. | Model | #Params | BBH (zero-shot) | BBEH | GSM8K-test | AIME 2024 | | --- | --- | --- | --- | --- | --- | |  |  | mean@4 | mean@4 | mean@4 | mean@128 |

## Citation

```bibtex
@misc{he2026resyn,
  title={ReSyn: Autonomously Scaling Synthetic Environments for Reasoning Models},
  author={He et al. (2026)},
  year={2026},
  note={arXiv:2602.20117}
}
```

- arXiv: 2602.20117