Chartab Eval

---
name: chartab-eval
description: This benchmark evaluates vision-language models on fine-grained chart understanding, specifically focusing on dense grounding of data and visual attributes, identifying precise differences between paired charts, and measuring robustness to stylistic perturbations like color or font changes. Use when the user wants to benchmark on ChartAB, or asks about evaluating this task. Reports SCRM.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.26781
  bibtex_key: [REDACTED]
  confidence: high
---

# chartab-eval

> ChartAB: A Benchmark for Chart Grounding & Dense Alignment — Bansal et al. (2025) (arXiv:2510.26781, 2025)

## What this evaluates

This benchmark evaluates vision-language models on fine-grained chart understanding, specifically focusing on dense grounding of data and visual attributes, identifying precise differences between paired charts, and measuring robustness to stylistic perturbations like color or font changes.

## Datasets

- **ChartAB** — total 9000; splits: test (-1); repo https://github.com/tianyi-lab/ChartAlignBench

## Metrics

- `SCRM` **(primary)** — range: [0, 1]
  - Precision of the predicted CSV against the ground-truth data table, computed using the structural similarity protocol from StructChart.
- `key-value alignment score` — range: [0, 1]
  - Ratio of correctly identified divergent elements (keys) and their associated values between two charts to the total number of ground-truth differences.
- `L2 error` — range: other
  - Euclidean distance between predicted and ground-truth RGB values for chart visual elements.
- `QA accuracy` — range: percent
  - Exact match for string answers; numerical answers are correct if within a 5% error margin of the ground truth.

## Input / output format

**Input**: Single or paired chart images accompanied by natural language instructions and a task-specific JSON template defining the expected output structure.

**Output**: Structured JSON or CSV representation of extracted data/attributes, or a JSON file listing identified differences between paired charts.

## Scoring recipe

```python
# 1. Data Grounding (SCRM)
pred_csv = parse_json_to_csv(prediction)
gold_csv = instance.gold_data
scrm = compute_structural_similarity(pred_csv, gold_csv) # per StructChart protocol

# 2. Alignment (Key-Value)
pred_diffs = parse_json_output(prediction)
gold_diffs = instance.gold_differences
matched = sum(1 for k, v in pred_diffs.items() if k in gold_diffs and v == gold_diffs[k])
alignment_score = matched / len(gold_diffs) if gold_diffs else 0

# 3. QA Accuracy
pred_ans = prediction.strip()
gold_ans = instance.gold_answer
if is_numeric(gold_ans):
    correct = abs(float(pred_ans) - float(gold_ans)) / abs(float(gold_ans)) <= 0.05
else:
    correct = pred_ans == gold_ans
```

## Common pitfalls

- VLMs often hallucinate visual attributes (colors, fonts) or misperceive layout structures when grounding single charts.
- Direct end-to-end alignment without intermediate grounding yields significantly weaker performance and higher ambiguity.
- Models are highly sensitive to attribute perturbations (e.g., color shifts), causing alignment scores to drop drastically under robustness tests.

## Evidence (verbatim from paper)

> Dense Grounding performance is evaluated by the precision of the detected semantic elements in a given chart, e.g., values of visualized data, color of bars, legend position, font size. In the experiments, we report (1) Legend position grounding’s confusion matrix in Figure[8]; (2) Text-style grounding accuracy in Figure[6]; (3) Color grounding’s L2 error of RGB values in Figure[7]; and (4) Data grounding performance in Figure[9(b)] is evaluated by the precision of predicted CSV using the SCRM metric introduced in StructChart (Xia et al., [2023]).

## Citation

```bibtex
@misc{bansal2025chartab,
  title={ChartAB: A Benchmark for Chart Grounding & Dense Alignment},
  author={Bansal et al. (2025)},
  year={2025},
  note={arXiv:2510.26781}
}
```

- arXiv: 2510.26781