Dvqa Eval

---
name: dvqa-eval
description: This benchmark evaluates a model's ability to perform visual reasoning and information extraction on bar chart data visualizations. It specifically probes whether systems can accurately read chart-specific labels, handle out-of-vocabulary terms, and answer natural language questions about quantitative relationships and chart structure. Use when the user wants to benchmark on DVQA, or asks about evaluating this task. Reports exact-match accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1801.08163
  bibtex_key: kafle2018dvqa
  confidence: high
---

# dvqa-eval

> DVQA: Understanding Data Visualizations via Question Answering — Kafle et al. (2018) (arXiv:1801.08163, 2018)

## What this evaluates

This benchmark evaluates a model's ability to perform visual reasoning and information extraction on bar chart data visualizations. It specifically probes whether systems can accurately read chart-specific labels, handle out-of-vocabulary terms, and answer natural language questions about quantitative relationships and chart structure.

## Datasets

- **DVQA** — total ?; splits: train (-1), Test-Familiar (-1), Test-Novel (-1)

## Metrics

- `exact-match accuracy` **(primary)** — range: [0, 1]
  - A prediction is counted as correct only if the generated string is character-for-character identical to the ground truth answer.
- `edit distance` — range: [0, 1]
  - A prediction is counted as correct if the Levenshtein edit distance between the generated string and the ground truth is less than or equal to 1.

## Input / output format

**Input**: A bar chart image and a natural language question.

**Output**: A single string representing the answer.

## Scoring recipe

```python
def exact_match_accuracy(predictions, golds):
    return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)

def edit_distance_accuracy(predictions, golds, max_dist=1):
    return sum(1 for p, g in zip(predictions, golds) if edit_distance(p, g) <= max_dist) / len(golds)
```

## Common pitfalls

- Fixed-vocabulary models fail on chart-specific labels because they cannot encode or generate out-of-vocabulary terms present in the visualization.
- Minor OCR or string generation errors cause complete failure under exact-match scoring, especially on the Test-Novel split where labels are unseen during training.

## Evidence (verbatim from paper)

> To measure performance, an algorithm gets a question correct only if it generates a string that is identical to the ground truth. To better assess MOM, we also measure its performance using edit distance, which is denoted MOM ($\pm 1$). This model is allowed to get a question correct as long as the answer it generates is within one edit distance or less compared to the correct answer.

## Citation

```bibtex
@misc{kafle2018dvqa,
  title={DVQA: Understanding Data Visualizations via Question Answering},
  author={Kafle et al. (2018)},
  year={2018},
  note={arXiv:1801.08163}
}
```

- arXiv: 1801.08163