Crbench Eval

---
name: crbench-eval
description: Evaluates whether multimodal large language models can perform genuine visual reasoning on charts by inferring values from axes and scales, rather than relying on OCR or pre-existing annotations. It probes the model's ability to interpret complex visual structures and perform multi-step estimation on both synthetic and real-world charts. Use when the user wants to benchmark on CRBench, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.04457
  bibtex_key: zhang2025domllmsreallyunderstandthecharts
  confidence: high
---

# crbench-eval

> Do MLLMs Really Understand the Charts? — Xiao Zhang et al. (2025) (arXiv:2509.04457, 2025)

## What this evaluates

Evaluates whether multimodal large language models can perform genuine visual reasoning on charts by inferring values from axes and scales, rather than relying on OCR or pre-existing annotations. It probes the model's ability to interpret complex visual structures and perform multi-step estimation on both synthetic and real-world charts.

## Datasets

- **CRBench** — total ?; splits: Synthetic (-1), Real (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers out of the total number of questions. Calculated as (number of correct predictions / total predictions) * 100. Standard exact-match evaluation is used for chart value estimation tasks.

## Input / output format

**Input**: Chart images (synthetic or real, non-annotated) paired with a natural language question asking for value estimation or chart interpretation.

**Output**: A direct answer or a Chain-of-Thought reasoning trace followed by the final answer.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_answers):
    if normalize(pred) == normalize(gold):
        correct += 1
accuracy = (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Models often bypass visual reasoning by relying on OCR to read axis labels or pre-existing annotations, leading to inflated scores on annotation-heavy benchmarks.
- Applying Chain-of-Thought prompting without underlying visual reasoning capabilities frequently degrades performance, as the model is forced to articulate flawed reasoning steps.

## Evidence (verbatim from paper)

> | Methods | Accuracy | | | 
 | --- | --- | --- | 
 | Synthetic | Real | Overall | The results in Table[1] underscore the significant challenge our CRBench presents to current models. The generally low scores across the board reveal a critical weakness: the widespread inability to reason over non-annotated charts.

## Citation

```bibtex
@misc{zhang2025domllmsreallyunderstandthecharts,
  title={Do MLLMs Really Understand the Charts?},
  author={Xiao Zhang et al. (2025)},
  year={2025},
  note={arXiv:2509.04457}
}
```

- arXiv: 2509.04457