Chart Understanding Eval

---
name: chart-understanding-eval
description: Evaluates multimodal language models' ability to comprehend diverse chart types, extract underlying numerical data, and answer questions across varying complexity levels. It distinguishes between OCR-dependent recognition on annotated charts and true data reasoning on unannotated or raw-data-requiring charts. Use when the user wants to benchmark on ChartQA, PlotQA, ChartDQA, MMC, ChartX, Chart-to-Table, Chart-to-Text, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.14298
  bibtex_key: [REDACTED]
  confidence: medium
---

# chart-understanding-eval

> In-Depth and In-Breadth: Pre-training Multimodal Language Models Customized for Comprehensive Chart Understanding — Fan et al. (2025) (arXiv:2507.14298, 2025)

## What this evaluates

Evaluates multimodal language models' ability to comprehend diverse chart types, extract underlying numerical data, and answer questions across varying complexity levels. It distinguishes between OCR-dependent recognition on annotated charts and true data reasoning on unannotated or raw-data-requiring charts.

## Datasets

- **ChartQA** — total ?; splits: human (-1), augmented (-1)
- **PlotQA** — total ?; splits: test (-1)
- **ChartDQA** — total ?; splits: test (-1)
- **MMC** — total ?; splits: VQA (-1)
- **ChartX** — total ?; splits: VQA (-1)
- **Chart-to-Table** — total ?; splits: test (-1)
- **Chart-to-Text** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Exact match accuracy between the model's generated text answer and the ground truth answer. Computed separately for the human and augmented splits on ChartQA.
- `F1 score` — range: [0, 1]
  - Token-level F1 score measuring the overlap between predicted and gold table structures/values on Chart-to-Table.
- `benchmark-specific metrics` — range: varies
  - MMC, ChartX, and ChartDQA follow their respective official evaluation protocols and scoring rules as defined in their original publications.

## Input / output format

**Input**: Chart image paired with a natural language question. For ChartDQA, models may also receive access to raw underlying data or JSON representations.

**Output**: Natural language text answer or structured JSON, conforming to each benchmark's specific output schema.

## Scoring recipe

```python
def evaluate(predictions, gold, benchmark):
    if benchmark == 'ChartQA':
        return exact_match_accuracy(predictions, gold)
    elif benchmark == 'Chart-to-Table':
        return compute_f1_score(predictions, gold)
    else:
        return run_official_evaluator(predictions, gold, benchmark)
```

## Common pitfalls

- High scores on annotated benchmarks (e.g., ChartQA) may primarily reflect OCR capability rather than true chart understanding or data reasoning.
- PlotQA evaluation is strictly zero-shot for most baselines because training them on PlotQA is infeasible, creating an asymmetric comparison.
- Underlying data evaluation (ChartDQA) requires explicit reasoning over raw numerical values, which models often fail at if they rely solely on visual annotations.

## Evidence (verbatim from paper)

> As shown in Table 2, ChartScope achieves on-par accuracy with the SOTA on ChartQA. Additionally, ChartScope achieves a competitive F1 score on Chart-to-Table... For benchmark details and evaluation metrics, we follow each benchmark’s protocol; please refer to Sec.B for more information.

## Citation

```bibtex
@misc{fan2025chartscope,
  title={In-Depth and In-Breadth: Pre-training Multimodal Language Models Customized for Comprehensive Chart Understanding},
  author={Fan et al. (2025)},
  year={2025},
  note={arXiv:2507.14298}
}
```

- arXiv: 2507.14298