Chartassistant Eval

---
name: chartassistant-eval
description: Evaluates a multimodal language model's ability to comprehend, summarize, and answer questions about various chart types (base and specialized). It probes chart-to-text generation, open-ended and numerical question answering, referring question answering, and chart-to-table translation. Use when the user wants to benchmark on ChartQA, Chart-to-Text, OpenCQA, MathQA, ReferQA, RealQA, or asks about evaluating this task. Reports relaxed_correctness.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.02384
  bibtex_key: meng2024chartassistant
  confidence: high
---

# chartassistant-eval

> ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning — Fanqing Meng et al. (2024) (arXiv:2401.02384, 2024)

## What this evaluates

Evaluates a multimodal language model's ability to comprehend, summarize, and answer questions about various chart types (base and specialized). It probes chart-to-text generation, open-ended and numerical question answering, referring question answering, and chart-to-table translation.

## Datasets

- **ChartQA** — total ?; splits: aug (-1), human (-1)
- **Chart-to-Text** — total ?; splits: Pew (-1), Statista (-1)
- **OpenCQA** — total ?; splits: test (-1)
- **MathQA** — total ?; splits: test (-1)
- **ReferQA** — total ?; splits: test (-1)
- **RealQA** — total ?; splits: Math (-1), Extract (-1)

## Metrics

- `relaxed_correctness` **(primary)** — range: percent
  - Exact match with a 5% numerical tolerance. A prediction is correct if it matches the gold answer exactly, or if the absolute relative error between the predicted and gold numerical values is ≤ 0.05.
- `BLEU` — range: percent
  - Standard n-gram based BLEU score used for chart summarization and open-ended QA generation tasks.
- `RMS_F1` — range: percent
  - Root Mean Square F1 score used to evaluate chart-to-table translation performance, following the DePlot protocol.

## Input / output format

**Input**: Chart image paired with a natural language question or instruction (e.g., summarization prompt, numerical QA, referring QA, or table extraction request).

**Output**: Text response containing the answer, summary, or structured table data.

## Scoring recipe

```python
def compute_relaxed_correctness(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        try:
            p, g = float(pred), float(gold)
            if abs(p - g) / max(abs(g), 1e-6) <= 0.05:
                correct += 1
        except ValueError:
            if pred.strip().lower() == gold.strip().lower():
                correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Relaxed correctness allows a 5% numerical tolerance, so strict exact-match evaluation will incorrectly penalize valid answers.
- BLEU evaluation on Chart-to-Text and OpenCQA is highly sensitive to reference wording due to limited ground-truth references, making scores volatile and heavily dependent on reference alignment.
- Baseline models are fine-tuned on the training split of each test dataset, whereas ChartAssistant is evaluated after a single unified training phase, creating an unfair comparison if not explicitly accounted for.

## Evidence (verbatim from paper)

> Metrics. For evaluating ChartQA, MathQA, and ReferQA, we adopt the approach used in previous studies [25, 31], which considers relaxed correctness (allowing for an exact match with tolerance for a 5% numerical error). As for Chart-to-Text and OpenCQA, we employ BLEU as the evaluation metric following previous works [25, 31]. For chart-to-table translation, we use RMS_F1 from DePlot [24].

## Citation

```bibtex
@misc{meng2024chartassistant,
  title={ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning},
  author={Fanqing Meng et al. (2024)},
  year={2024},
  note={arXiv:2401.02384}
}
```

- arXiv: 2401.02384