Vprochart Eval

---
name: vprochart-eval
description: Evaluates a model's ability to understand chart visuals and perform multi-step numerical and logical reasoning to answer natural language questions. It specifically probes visual perception alignment and programmatic solution reasoning over structured chart data. Use when the user wants to benchmark on ChartQA, PlotQA, DVQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.01667
  bibtex_key: huang2024vprochart
  confidence: high
---

# vprochart-eval

> VProChart: Answering Chart Question through Visual Perception Alignment Agent and Programmatic Solution Reasoning — Huang et al. (2024) (arXiv:2409.01667, 2024)

## What this evaluates

Evaluates a model's ability to understand chart visuals and perform multi-step numerical and logical reasoning to answer natural language questions. It specifically probes visual perception alignment and programmatic solution reasoning over structured chart data.

## Datasets

- **ChartQA** — total ?; splits: Human (-1), Augment (-1)
- **PlotQA** — total ?; splits: D1 (-1), D2 (-1)
- **DVQA** — total ?; splits: Familiar (-1), Novel (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Number of correctly predicted answers divided by the total number of questions, multiplied by 100. Answers are typically compared using exact match or normalized string matching.

## Input / output format

**Input**: A chart image paired with a natural language question.

**Output**: A natural language answer or numerical value.

## Scoring recipe

```python
def calculate_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if str(p).strip().lower() == str(g).strip().lower())
    return correct / len(gold) * 100
```

## Common pitfalls

- ChartQA-H (Human) and ChartQA-M (Augment) have vastly different difficulty levels due to manual curation versus machine generation; mixing them up or reporting only one skews results.
- PlotQA-D1 is often used for training or in-domain testing, while D2 is a held-out generalization set; reporting only D1 inflates perceived robustness.
- DVQA's Familiar vs Novel split tests generalization to unseen chart types and data distributions, which is frequently overlooked when only aggregate scores are reported.

## Evidence (verbatim from paper)

> In Table [1] and [2], we present a comprehensive summary of VProChart’s performance on three popular datasets.

## Citation

```bibtex
@misc{huang2024vprochart,
  title={VProChart: Answering Chart Question through Visual Perception Alignment Agent and Programmatic Solution Reasoning},
  author={Huang et al. (2024)},
  year={2024},
  note={arXiv:2409.01667}
}
```

- arXiv: 2409.01667