Polychartqa Eval

---
name: polychartqa-eval
description: Evaluates multimodal language models' ability to answer questions that require reasoning across multiple charts or images. It probes visual decomposition, sub-chart localization, and handling of complex multi-visual contexts versus single-chart inputs. Use when the user wants to benchmark on PolyChartQA, MultiChartQA-RQ1, or asks about evaluating this task. Reports L-Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.21344
  bibtex_key: efat2026beyond
  confidence: high
---

# polychartqa-eval

> Beyond Single Plots: A Benchmark for Question Answering on Multi-Charts — Efat et al. (2026) (arXiv:2604.21344, 2026)

## What this evaluates

Evaluates multimodal language models' ability to answer questions that require reasoning across multiple charts or images. It probes visual decomposition, sub-chart localization, and handling of complex multi-visual contexts versus single-chart inputs.

## Datasets

- **PolyChartQA** — total ?; splits: test (-1); repo https://github.com/NRT-D4/PolyChartQA
- **MultiChartQA-RQ1** — total 365; splits: test (365)

## Metrics

- `L-Accuracy` **(primary)** — range: [0, 1]
  - LLM-based accuracy computed by prompting Claude-3.7-Sonnet as a judge to determine if the model's generated answer matches the ground truth.
- `H-Accuracy` — range: [0, 1]
  - Human-evaluation accuracy where human annotators verify model-predicted answers against ground truth for human-authored questions in a zero-shot setting.
- `BERTScore` — range: [0, 1]
  - Measures semantic similarity between model predictions and reference answers using BERT embeddings.

## Input / output format

**Input**: Multi-chart images (composite of multiple sub-charts) or single-chart images, paired with a natural language question.

**Output**: Natural language answer string corresponding to the question.

## Scoring recipe

```python
def compute_l_accuracy(pred, gt):
    prompt = f'Question: {pred}\nGround Truth: {gt}\nIs the answer correct?'
    return 1.0 if llm_judge(prompt) == 'Yes' else 0.0

def compute_bertscore(pred, gt):
    return bert_score.compute(predictions=[pred], references=[gt])[0]

def compute_h_accuracy(pred, gt):
    return 1.0 if human_evaluator(pred, gt) == 'Correct' else 0.0
```

## Common pitfalls

- Using an LLM judge without verifying alignment with human evaluation can produce misleading L-Accuracy scores.
- Evaluating MultiChartQA-RQ1 requires strict separation of single-chart vs multi-chart input configurations, as the questions are identical across conditions.
- Relying on a single evaluation run may obscure minor variability across model versions or prompting strategies.

## Evidence (verbatim from paper)

> Evaluation Metrics: We use H-Accuracy (Human-evaluation), L-Accuracy (LLM-based accuracy) introduced by the previous works *Pramanick et al. ([2024](#bib.bib30 "SPIQA: a dataset for multimodal question answering on scientific papers")); Liu et al. ([2024a](#bib.bib20 "MMC: advancing multimodal chart understanding with large-scale instruction tuning")); Tang et al. ([2025](#bib.bib35 "ChartMuseum: testing visual reasoning capabilities of large vision-language models")); Wang et al. ([2024](#bib.bib34 "CharXiv: charting gaps in realistic chart understanding in multimodal llms"))*, and BERTScore *Zhang et al. ([2020](#bib.bib31 "BERTScore: evaluating text generation with bert"))*. BERTScore measures semantic similarity between model predictions and references. H-accuracy was obtained through human evaluation of model-predicted answers for human-authored questions in a Zero-shot setting. As human evaluation is difficult to scale, for all other cases, we use L-accuracy by prompting a selected LLM as a judge to assess whether the ground-truth and model-generated answers are similar or not.

## Citation

```bibtex
@misc{efat2026beyond,
  title={Beyond Single Plots: A Benchmark for Question Answering on Multi-Charts},
  author={Efat et al. (2026)},
  year={2026},
  note={arXiv:2604.21344}
}
```

- arXiv: 2604.21344