R Bench Eval

---
name: r-bench-eval
description: Evaluates complex reasoning capabilities of LLMs and MLLMs on graduate-level, multi-disciplinary academic questions in both English and Chinese. It probes the models' ability to handle rigorous, curriculum-based problems requiring extended chain-of-thought reasoning. Use when the user wants to benchmark on R-Bench-T, R-Bench-M, or asks about evaluating this task. Reports Top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.02018
  bibtex_key: guo2025rbench
  confidence: high
---

# r-bench-eval

> R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation — Guo et al. (2025) (arXiv:2505.02018, 2025)

## What this evaluates

Evaluates complex reasoning capabilities of LLMs and MLLMs on graduate-level, multi-disciplinary academic questions in both English and Chinese. It probes the models' ability to handle rigorous, curriculum-based problems requiring extended chain-of-thought reasoning.

## Datasets

- **R-Bench-T** — total 1094; splits: test (1094)
- **R-Bench-M** — total 665; splits: test (665)

## Metrics

- `Top-1 accuracy` **(primary)** — range: percent
  - Calculated as the percentage of questions where the model's predicted answer exactly matches the ground truth answer. Computed over the full test set and reported as a percentage.

## Input / output format

**Input**: Text-only prompts for R-Bench-T; text-and-image prompts for R-Bench-M. Both include a Chain-of-Thought (CoT) instruction. Available in English and Chinese.

**Output**: Model generates a final answer (and optionally reasoning steps). For evaluation, the final answer is extracted and compared against the gold label.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if normalize_answer(pred) == normalize_answer(gold):
        correct += 1
accuracy = (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Reasoning-focused models (e.g., o1-mini) do not benefit from explicit CoT prompting, unlike chat models, which can lead to misleading comparisons if prompting strategies are not standardized.
- Performance varies significantly across academic disciplines (up to 37.9% range), so aggregate scores may mask severe weaknesses in specific subjects.
- Multilingual consistency does not guarantee equivalent difficulty; models may overfit to specific linguistic patterns rather than demonstrating true cross-lingual reasoning transfer.

## Evidence (verbatim from paper)

> The values in the table represent the Top-1 accuracy, in %. In all tests, the CoT prompt is used by default. For details on the specific prompts, please refer to our appendix.

## Citation

```bibtex
@misc{guo2025rbench,
  title={R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation},
  author={Guo et al. (2025)},
  year={2025},
  note={arXiv:2505.02018}
}
```

- arXiv: 2505.02018