Chartgalaxy Eval

---
name: chartgalaxy-eval
description: This benchmark evaluates multimodal large language models' ability to understand and reason about infographic charts. It probes capabilities in text-based data reasoning, visual-element association, and visual style analysis through structured question-answering tasks. Use when the user wants to benchmark on ChartGalaxy, or asks about evaluating this task. Reports relaxed accuracy (5% margin).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.18668
  bibtex_key: li2025chartgalaxy
  confidence: high
---

# chartgalaxy-eval

> ChartGalaxy: A Dataset for Infographic Chart Understanding and Generation — Li et al. (2025) (arXiv:2505.18668, 2025)

## What this evaluates

This benchmark evaluates multimodal large language models' ability to understand and reason about infographic charts. It probes capabilities in text-based data reasoning, visual-element association, and visual style analysis through structured question-answering tasks.

## Datasets

- **ChartGalaxy** — total 70248; splits: train (70248), test (2176); repo https://github.com/ChartGalaxy/ChartGalaxy

## Metrics

- `relaxed accuracy (5% margin)` **(primary)** — range: percent
  - Accuracy calculated by comparing model predictions to ground truth, allowing a 5% relative tolerance margin for numerical answers. Textual answers are evaluated using ANLS, and multiple-choice questions use exact matching.

## Input / output format

**Input**: An infographic chart image paired with a natural language question (covering data identification, comparison, extraction, fact-checking, visual-element association, or visual understanding).

**Output**: A natural language answer string, a selected multiple-choice option, or a numerical value.

## Scoring recipe

```python
def compute_vqa_accuracy(predictions, golds, question_types):
    correct = 0
    for pred, gold, qtype in zip(predictions, golds, question_types):
        if qtype == 'multiple_choice':
            if pred.strip().lower() == gold.strip().lower(): correct += 1
        elif qtype == 'text':
            if compute_anls(pred, gold) >= 0.5: correct += 1
        elif qtype == 'numerical':
            if abs(float(pred) - float(gold)) / float(gold) <= 0.05: correct += 1
    return correct / len(predictions) * 100
```

## Common pitfalls

- Numerical answers require a strict 5% relative margin tolerance; absolute differences can lead to false negatives.
- ANLS for textual answers uses a standard 0.5 threshold, but variations in tokenization or case sensitivity can affect scores if not normalized.
- The independent evaluation set only contains synthetic charts with bounding-box annotations, so results may not generalize to real-world infographic charts without visual elements.

## Evidence (verbatim from paper)

> For the evaluation metrics, we follow previous work on chart question answering*(Masry et al., [2025])*, using relaxed accuracy with a 5% margin for numerical answers, ANLS for textual answers, and exact matching for multiple-choice questions.

## Citation

```bibtex
@misc{li2025chartgalaxy,
  title={ChartGalaxy: A Dataset for Infographic Chart Understanding and Generation},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2505.18668}
}
```

- arXiv: 2505.18668