Chartqa Eval

---
name: chartqa-eval
description: Evaluates multimodal models' ability to answer questions about charts by extracting visual and textual information. It probes robustness to missing labels and geometric perturbations, distinguishing between simple pattern matching and true structural reasoning. Use when the user wants to benchmark on ChartQA, Charixv, or asks about evaluating this task. Reports Relaxed Accuracy (RA).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.09764
  bibtex_key: [REDACTED]
  confidence: high
---

# chartqa-eval

> Socratic Chart: Cooperating Multiple Agents for Robust SVG Chart Understanding — Ji et al. (2025) (arXiv:2504.09764, 2025)

## What this evaluates

Evaluates multimodal models' ability to answer questions about charts by extracting visual and textual information. It probes robustness to missing labels and geometric perturbations, distinguishing between simple pattern matching and true structural reasoning.

## Datasets

- **ChartQA** — total ?; splits: human (-1), augmented (-1)
- **Charixv** — total ?; splits: test (-1)

## Metrics

- `Relaxed Accuracy (RA)` **(primary)** — range: percent
  - Measures the accuracy of question answering on the dataset, considering both human-authored and augmented QA pairs. It is the standard metric for chart understanding, typically allowing for minor formatting or unit variations in the predicted answer compared to the ground truth.

## Input / output format

**Input**: Chart image (bar, pie, or line chart) paired with a natural language question.

**Output**: Natural language answer to the question.

## Scoring recipe

```python
def compute_ra(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- High scores on augmented questions often reflect memorization of programmatic patterns rather than genuine chart reasoning.
- Models frequently rely on OCR/text shortcuts instead of visual estimation, causing sharp performance drops under label removal or perturbation.
- Aggregating human and augmented sets can mask poor performance on complex, real-world chart questions.

## Evidence (verbatim from paper)

> In terms of evaluation metric, we use Relaxed Accuracy (RA), which measures the accuracy of question answering on the ChartQA dataset, considering both human-authored and augmented QA pairs. This is the standard evaluation metric adopted by previous chart understanding works, enabling direct and fair comparison across different approaches.

## Citation

```bibtex
@misc{ji2025socraticchart,
  title={Socratic Chart: Cooperating Multiple Agents for Robust SVG Chart Understanding},
  author={Ji et al. (2025)},
  year={2025},
  note={arXiv:2504.09764}
}
```

- arXiv: 2504.09764