Chart Reasoning Eval

---
name: chart-reasoning-eval
description: Evaluates a model's ability to understand complex chart visualizations and perform fine-grained visual grounding and numerical reasoning. It probes both in-domain chart comprehension across real-world and synthetic datasets, and out-of-domain generalization to visual mathematical reasoning tasks. Use when the user wants to benchmark on CharXiv, ChartQAPro, ChartQA, ChartBench, ChartX, ReachQA, MathVista, WeMath, MathVerse, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.02794
  bibtex_key: [REDACTED]
  confidence: high
---

# chart-reasoning-eval

> CharTool: Tool-Integrated Visual Reasoning for Chart Understanding — Situo Zhang et al. (arXiv:2604.02794, 2026)

## What this evaluates

Evaluates a model's ability to understand complex chart visualizations and perform fine-grained visual grounding and numerical reasoning. It probes both in-domain chart comprehension across real-world and synthetic datasets, and out-of-domain generalization to visual mathematical reasoning tasks.

## Datasets

- **CharXiv** — total ?; splits: test (-1)
- **ChartQAPro** — total ?; splits: test (-1)
- **ChartQA** — total ?; splits: test (-1)
- **ChartBench** — total ?; splits: test (-1)
- **ChartX** — total ?; splits: test (-1)
- **ReachQA** — total ?; splits: test (-1)
- **MathVista** — total ?; splits: test (-1)
- **WeMath** — total ?; splits: test (-1)
- **MathVerse** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard exact-match or normalized string matching accuracy calculated per sub-task (reasoning, description, recognition, etc.) and averaged across benchmarks. Formula: (Number of correct predictions / Total number of predictions) * 100.

## Input / output format

**Input**: A chart image (bar, line, pie, scatter, or multi-subplot scientific figures) paired with a natural language question or query.

**Output**: Natural language text containing step-by-step reasoning and a final answer.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Benchmarks report multiple sub-task scores (e.g., reasoning, description, recognition); averaging them without explicit weighting can obscure performance on specific capabilities.
- Out-of-domain benchmarks (MathVista, WeMath, MathVerse) test general visual/math reasoning rather than chart-specific skills, so direct score comparison with chart benchmarks is misleading.
- The paper uses GPT-5.2 for dataset quality evaluation, which is separate from the model evaluation protocol and should not be conflated with the main accuracy metrics.

## Evidence (verbatim from paper)

> We conduct a comprehensive evaluation of our models across a diverse set of standard chart benchmarks, categorized into: (1) Real-world Benchmarks, comprising charts extracted from real-world sources, including CharXiv, ChartQAPro, and ChartQA; and (2) Synthetic Benchmarks, containing synthesized images or queries: ChartBench, ChartX, and ReachQA. Furthermore, to assess the out-of-domain generalization capabilities of our approach, we include three visual math reasoning benchmarks: MathVista, WeMath, and MathVerse.

## Citation

```bibtex
@misc{zhang2026chartool,
  title={CharTool: Tool-Integrated Visual Reasoning for Chart Understanding},
  author={Situo Zhang et al.},
  year={2026},
  note={arXiv:2604.02794}
}
```

- arXiv: 2604.02794