Chartmuseum Eval

---
name: chartmuseum-eval
description: This benchmark evaluates the visual reasoning capabilities of Large Vision-Language Models (LVLMs) on real-world charts. It specifically probes the model's ability to perform visual extraction, object picking, visual comparisons, and trajectory tracking, while distinguishing these from purely textual inference tasks. Use when the user wants to benchmark on CHARTMUSEUM, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.13444
  bibtex_key: [REDACTED]
  confidence: high
---

# chartmuseum-eval

> ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models — Tang et al. (2025) (arXiv:2505.13444, 2025)

## What this evaluates

This benchmark evaluates the visual reasoning capabilities of Large Vision-Language Models (LVLMs) on real-world charts. It specifically probes the model's ability to perform visual extraction, object picking, visual comparisons, and trajectory tracking, while distinguishing these from purely textual inference tasks.

## Datasets

- **CHARTMUSEUM** — total 1162; splits: Visual (510), Synthesis (133), Visual/Text (234), Text (123), Overall (1000)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Accuracy is computed as the percentage of questions answered correctly. Answers are evaluated using an LLM-as-a-Judge approach to handle paraphrasing, as all questions are designed to have unique, objective answers.

## Input / output format

**Input**: A chart image paired with a natural language question requiring visual or textual reasoning.

**Output**: A unique, objective answer string (e.g., a specific value, category name, or short phrase).

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        # LLM-as-a-Judge evaluates semantic match using Appendix Fig 27 prompt
        if llm_judge_match(pred, gold):
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Models heavily over-rely on OCR/textual extraction rather than actual visual reasoning, leading to artificially high scores on 'Text' splits but poor performance on 'Visual' splits.
- Applying chain-of-thought or extended reasoning prompts yields minimal gains for chart understanding compared to math/code tasks, contrary to recent LLM trends.
- LLM-as-a-Judge evaluation requires a carefully crafted prompt to avoid bias; naive exact-match fails due to paraphrasing.

## Evidence (verbatim from paper)

> Table 3: Accuracy performance comparison of models on the test set of the CHARTMUSEUM benchmark. Humans achieve an overall accuracy of 93.0% on a randomly sampled subset consisting 100 examples.

## Citation

```bibtex
@misc{tang2025chartmuseum,
  title={ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models},
  author={Tang et al. (2025)},
  year={2025},
  note={arXiv:2505.13444}
}
```

- arXiv: 2505.13444