Mrceval Eval

---
name: mrceval-eval
description: This benchmark evaluates large language models' ability to comprehend passages and answer questions across multiple dimensions, including context understanding, external knowledge integration, and complex reasoning. It probes factual fidelity, counterfactual handling, commonsense, world knowledge, and multi-hop reasoning capabilities. Use when the user wants to benchmark on MRCEval, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.07144
  bibtex_key: ma2025mrceval
  confidence: high
---

# mrceval-eval

> MRCEval: A Comprehensive, Challenging and Accessible Machine Reading Comprehension Benchmark — Ma et al. (2025) (arXiv:2503.07144, 2025)

## What this evaluates

This benchmark evaluates large language models' ability to comprehend passages and answer questions across multiple dimensions, including context understanding, external knowledge integration, and complex reasoning. It probes factual fidelity, counterfactual handling, commonsense, world knowledge, and multi-hop reasoning capabilities.

## Datasets

- **MRCEval** — total 2100; splits: test (-1); repo https://github.com/THU-KEG/MRCEval

## Metrics

- `accuracy` **(primary)** — range: percent
  - The proportion of correctly answered questions out of the total number of questions.

## Input / output format

**Input**: A reading comprehension passage, a question, and multiple-choice options, with a fixed instruction appended at the beginning.

**Output**: A single selected option corresponding to the correct answer.

## Scoring recipe

```python
correct = 0
total = len(gold)
for pred, gold_label in zip(predictions, gold):
    if pred.strip().lower() == gold_label.strip().lower():
        correct += 1
accuracy = (correct / total) * 100 if total > 0 else 0
```

## Common pitfalls

- Models are evaluated with different sampling settings (DeepSeek uses temperature 0.6 and top-p 0.95, while all others use greedy/temperature 0.0), which may confound performance comparisons.
- Evaluation relies on a single run without averaging over multiple random seeds or temperature variations, making results sensitive to stochastic decoding.
- The fixed prompt is appended to every instance regardless of task type, which may artificially boost or suppress performance on certain sub-tasks.

## Evidence (verbatim from paper)

> We report accuracy as the metric from a single run result.

## Citation

```bibtex
@misc{ma2025mrceval,
  title={MRCEval: A Comprehensive, Challenging and Accessible Machine Reading Comprehension Benchmark},
  author={Ma et al. (2025)},
  year={2025},
  note={arXiv:2503.07144}
}
```

- arXiv: 2503.07144