Redstar Reasoning Eval

---
name: redstar-reasoning-eval
description: Evaluates the model's ability to solve complex mathematical, coding, and general reasoning problems. It probes multi-step reasoning, domain-specific knowledge integration, and long-context handling across diverse benchmarks. Use when the user wants to benchmark on Math & Reasoning Benchmarks, Hellobench, SedarEval, Chinese Graduate Entrance Mathematics Test, or asks about evaluating this task. Reports AVG.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.11284
  bibtex_key: xu2025redstar
  confidence: high
---

# redstar-reasoning-eval

> RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems? — Xu et al. (2025) (arXiv:2501.11284, 2025)

## What this evaluates

Evaluates the model's ability to solve complex mathematical, coding, and general reasoning problems. It probes multi-step reasoning, domain-specific knowledge integration, and long-context handling across diverse benchmarks.

## Datasets

- **Math & Reasoning Benchmarks** — total ?; splits: test (-1); repo https://github.com/QwenLM/Qwen2.5-Math
- **Hellobench** — total ?; splits: test (-1)
- **SedarEval** — total ?; splits: test (-1)
- **Chinese Graduate Entrance Mathematics Test** — total ?; splits: test (-1); repo https://www.chinakaoyan.com/info/article/id/585019.shtml

## Metrics

- `AVG` **(primary)** — range: percent
  - Percentage of correctly solved problems. Calculated as (number of solved problems / total number of problems) * 100.

## Input / output format

**Input**: Natural language prompts containing mathematical problems, coding tasks, or general reasoning questions, often requiring multi-step solutions.

**Output**: Step-by-step reasoning traces (Long-CoT) followed by a final answer or solution.

## Scoring recipe

```python
def compute_avg(predictions, gold):
    correct = 0
    for pred, true_ans in zip(predictions, gold):
        if extract_final_answer(pred) == true_ans:
            correct += 1
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Models may output correct reasoning but fail to format the final answer correctly, leading to false negatives in exact-match scoring.
- Long-CoT outputs can exceed context windows or be truncated, causing evaluation scripts to miss the final answer if not properly parsed.
- Aggregating scores across heterogeneous benchmarks without weighting can mask domain-specific weaknesses.

## Evidence (verbatim from paper)

> All results are verified by human annotaters. “A/B” represents the ratio of the number of problems solved under this type of problem to the total number of problems. AVG represents the ratio of the solved problems under the total problems.

## Citation

```bibtex
@misc{xu2025redstar,
  title={RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?},
  author={Xu et al. (2025)},
  year={2025},
  note={arXiv:2501.11284}
}
```

- arXiv: 2501.11284