Chartdiff Eval

---
name: chartdiff-eval
description: This benchmark evaluates a model's ability to perform cross-chart comparative reasoning by generating natural language summaries that identify and explain differences in trends, fluctuations, and anomalies between pairs of charts. It probes vision-language models on their capacity to synthesize visual information from multiple plots into coherent, human-aligned textual descriptions. Use when the user wants to benchmark on ChartDiff, or asks about evaluating this task. Reports GPT Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.28902
  bibtex_key: ye2026chartdiff
  confidence: high
---

# chartdiff-eval

> ChartDiff: A Large-Scale Benchmark for Comprehending Pairs of Charts — Ye (2026) (arXiv:2603.28902, 2026)

## What this evaluates

This benchmark evaluates a model's ability to perform cross-chart comparative reasoning by generating natural language summaries that identify and explain differences in trends, fluctuations, and anomalies between pairs of charts. It probes vision-language models on their capacity to synthesize visual information from multiple plots into coherent, human-aligned textual descriptions.

## Datasets

- **ChartDiff** — total 8541; splits: train (-1), test (-1)

## Metrics

- `ROUGE` — range: percent
  - Standard lexical-overlap metric measuring n-gram overlap between generated and reference summaries. Reported as ROUGE-1, ROUGE-2, and ROUGE-L.
- `GPT Score` **(primary)** — range: [1, 5]
  - Model-based evaluation using GPT-5.4 as a judge with a predefined grading prompt to assess summary quality. Scores range from 1 to 5.

## Input / output format

**Input**: Two charts concatenated horizontally into a single combined image, accompanied by a prompt instructing the model to generate a comparison summary.

**Output**: A natural language text summary describing the differences between the two charts.

## Scoring recipe

```python
def score_rouge(reference, prediction):
    return rouge_score(reference, prediction, rouge_types=['rouge1', 'rouge2', 'rougeL'])

def score_gpt(reference, prediction):
    prompt = f'Grade the quality of this summary against the reference: Pred="{prediction}", Ref="{reference}".'
    response = call_llm('gpt-5.4', prompt)
    return extract_numeric_score(response)
```

## Common pitfalls

- High ROUGE scores indicate strong lexical overlap but do not guarantee human-aligned quality or accurate difference detection.
- Pipeline-based extract-then-compare methods are highly sensitive to chart type (e.g., failing on pie charts) and plotting library rendering.
- Multi-series charts consistently yield lower scores across all model families compared to single-series charts.

## Evidence (verbatim from paper)

> We adopt two complementary evaluation metrics: ROUGE. We use ROUGE as a standard lexical-overlap metric to measure similarity between generated summaries and reference annotations. GPT Score. We further employ a model-based evaluation metric, GPT Score using GPT-5.4 as the judge model to assess the quality of generated summaries with a predefined grading prompt (Figure 16 and Figure 17).

## Citation

```bibtex
@misc{ye2026chartdiff,
  title={ChartDiff: A Large-Scale Benchmark for Comprehending Pairs of Charts},
  author={Ye (2026)},
  year={2026},
  note={arXiv:2603.28902}
}
```

- arXiv: 2603.28902