Chartnet Eval

---
name: chartnet-eval
description: Evaluates multimodal vision-language models on chart understanding tasks, including reconstructing plotting code from charts, extracting tabular data, summarizing chart content, and answering complex reasoning questions. Use when the user wants to benchmark on ChartNet Evaluation Set, or asks about evaluating this task. Reports ChartNet Evaluation Metrics.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.27064
  bibtex_key: [REDACTED]
  confidence: high
---

# chartnet-eval

> ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding — Kondic et al. (2026) (arXiv:2603.27064, 2026)

## What this evaluates

Evaluates multimodal vision-language models on chart understanding tasks, including reconstructing plotting code from charts, extracting tabular data, summarizing chart content, and answering complex reasoning questions.

## Datasets

- **ChartNet Evaluation Set** — total 2000; splits: test (2000)

## Metrics

- `ChartNet Evaluation Metrics` **(primary)** — range: percent
  - Exec.: fraction of generated scripts executing without error. Code-D: numeric correspondence between plotted values and ground truth. Code-S: structural/syntactic overlap. Img.: visual alignment. Data Extraction: CSV similarity score. Summarization: holistic GPT-4o judge score. QA: RapidFuzz fuzzy accuracy on extracted <answer>.

## Input / output format

**Input**: Chart image I; for reconstruction tasks, ground-truth plotting code/data table/summary may be provided as context or reference; for QA, a complex reasoning question paired with I.

**Output**: For reconstruction: executable plotting script C′. For extraction: CSV table. For summarization: textual summary. For QA: <think> and <answer> sections.

## Scoring recipe

```python
# 1. Chart Reconstruction
exec_rate = sum(1 for c in preds if run_script(c) == 0) / len(preds)
code_d, code_s, img_sim = gpt4o_judge(preds, gold_code, gold_img)
# 2. Data Extraction
data_sim = gpt4o_judge_csv(preds, gold_csv)
# 3. Summarization
summary_score = gpt4o_judge_text(preds, gold_summary)
# 4. QA
answer = extract_tag(preds, "<answer>")
qa_acc = rapidfuzz.fuzz.ratio(answer, gold_qa) / 100.0
```

## Common pitfalls

- Most metrics (except QA) rely on GPT-4o as an automated judge, which may introduce bias or inconsistency compared to exact programmatic metrics.
- Custom metrics like Code-D, Code-S, and Img. similarity lack standardized definitions and depend heavily on the specific GPT-4o prompt templates in Appendix B.4.
- The 'holistic score' for summarization combines multiple dimensions (coverage, faithfulness, correctness, clarity) into a single number without transparent weighting.

## Evidence (verbatim from paper)

> To rigorously evaluate the tasks in the core ChartNet dataset, we curate a held-out evaluation suite randomly drawn from ChartNet’s synthetic corpus. The set comprises 2000 chart tuples... We evaluate (a) execution rate (Exec.) — the fraction of generated scripts C′ that execute without error, (b) data fidelity (Code-D) — the correspondence between plotted numeric values and the data defined in ground-truth code, (c) code similarity (Code-S) — the structural and syntactic overlap between generated, C′, and source code, C, and (d) rendered image similarity (Img.) — the visual alignment between the rendered prediction and the input chart I... We report average fuzzy accuracy. All metrics are automatically computed using GPT-4o as a judge, except for the Chart QA with CoT Reasoning task.

## Citation

```bibtex
@misc{kondic2026chartnet,
  title={ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding},
  author={Kondic et al. (2026)},
  year={2026},
  note={arXiv:2603.27064}
}
```

- arXiv: 2603.27064