Charxiv Eval

---
name: charxiv-eval
description: Evaluates multimodal large language models' ability to understand real-world charts through descriptive and reasoning tasks. It probes capabilities like information extraction, pattern recognition, counting, compositional reasoning, and robustness to chart complexity (e.g., multiple subplots) and unanswerable queries. Use when the user wants to benchmark on CharXiv, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.18521
  bibtex_key: wang2024charxiv
  confidence: high
---

# charxiv-eval

> CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs — Wang et al. (2024) (arXiv:2406.18521, 2024)

## What this evaluates

Evaluates multimodal large language models' ability to understand real-world charts through descriptive and reasoning tasks. It probes capabilities like information extraction, pattern recognition, counting, compositional reasoning, and robustness to chart complexity (e.g., multiple subplots) and unanswerable queries.

## Datasets

- **CharXiv** — total 2323; splits: validation (-1), test (-1); repo https://github.com/princeton-nlp/CharXiv

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions out of the total number of questions in a category. Evaluated via exact match or automated grading consistent with human evaluation protocols.

## Input / output format

**Input**: A chart image (minimum resolution 448×448) paired with a natural language question (either descriptive or reasoning).

**Output**: A natural language text string containing the model's predicted answer.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, golds):
    if normalize(pred) == normalize(gold):
        correct += 1
accuracy = (correct / len(golds)) * 100
```

## Common pitfalls

- Models perform near-randomly on compositional tasks like counting axis ticks, despite humans succeeding easily.
- Performance degrades sharply (30-50% drop) for open-source models when charts contain 6 or more subplots.
- Weak models frequently fail to identify unanswerable questions, often hallucinating answers instead.

## Evidence (verbatim from paper)

> Although counting is easy for humans, this particular task causes 20 out of 24 models to achieve an accuracy below 10% (our random baseline achieves 5.35%).

## Citation

```bibtex
@misc{wang2024charxiv,
  title={CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2406.18521}
}
```

- arXiv: 2406.18521