Long Context Reasoning Eval

---
name: long-context-reasoning-eval
description: Evaluates a model's ability to perform multi-hop question answering and information retrieval over extremely long contexts (up to 128K tokens), while also measuring preservation of short-context reasoning and instruction-following capabilities. Use when the user wants to benchmark on LongBench v1, LongBench v2, MMLU, MATH-500, IFEval, Needle in a Haystack, RULER, or asks about evaluating this task. Reports pass@1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.19363
  bibtex_key: wang2025loongrl
  confidence: high
---

# long-context-reasoning-eval

> LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts — Siyuan Wang et al. (2025) (arXiv:2510.19363, 2025)

## What this evaluates

Evaluates a model's ability to perform multi-hop question answering and information retrieval over extremely long contexts (up to 128K tokens), while also measuring preservation of short-context reasoning and instruction-following capabilities.

## Datasets

- **LongBench v1** — total ?; splits: test (-1)
- **LongBench v2** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)
- **MATH-500** — total ?; splits: test (-1)
- **IFEval** — total ?; splits: test (-1)
- **Needle in a Haystack** — total ?; splits: test (-1)
- **RULER** — total ?; splits: test (-1)

## Metrics

- `pass@1 accuracy` **(primary)** — range: percent
  - For each problem, the model generates 8 candidate responses. The metric is the fraction of problems where at least one candidate contains the correct answer, reported as a percentage.

## Input / output format

**Input**: A multi-hop question paired with a long-context document or set of documents (up to 128K tokens).

**Output**: A reasoning trace followed by a final answer. Inference samples 8 responses per problem with temperature 0.6 and max 10K output tokens.

## Scoring recipe

```python
correct_count = 0
for problem in dataset:
    responses = model.generate(problem.context, problem.question, n=8, temperature=0.6)
    if any(extract_answer(r) == problem.gold_answer for r in responses):
        correct_count += 1
return (correct_count / len(dataset)) * 100
```

## Common pitfalls

- The paper uses a custom 'two-way substring exact match' verifier for RL training rewards, but evaluation uses standard pass@1 accuracy; do not confuse the two.
- Baselines may use single-pass inference, while LoongRL explicitly samples 8 solutions per problem to compute pass@1; ensure fair comparison by matching sampling strategies.
- Context length generalization is tested up to 128K tokens despite training only on 16K inputs; results degrade sharply for baselines at longer lengths.

## Evidence (verbatim from paper)

> For inference, reasoning models and our models use temperature 0.6, with up to 128K input tokens and 10K output tokens. We sample eight solutions per problem and report average pass@1 accuracy.

## Citation

```bibtex
@misc{wang2025loongrl,
  title={LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts},
  author={Siyuan Wang et al. (2025)},
  year={2025},
  note={arXiv:2510.19363}
}
```

- arXiv: 2510.19363