Visual Tableqa Eval

---
name: visual-tableqa-eval
description: Probes multimodal visual reasoning capabilities over complex, LaTeX-rendered table images. It specifically tests multi-step inference, structural layout understanding, and the ability to extract and reason over tabular data from visual inputs rather than raw text. Use when the user wants to benchmark on Visual-TableQA, or asks about evaluating this task. Reports Relaxed Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.07966
  bibtex_key: lompo2025visualtableqa
  confidence: high
---

# visual-tableqa-eval

> Visual-TableQA: Open-Domain Benchmark for Reasoning over Table Images — Lompo et al. (2025) (arXiv:2509.07966, 2025)

## What this evaluates

Probes multimodal visual reasoning capabilities over complex, LaTeX-rendered table images. It specifically tests multi-step inference, structural layout understanding, and the ability to extract and reason over tabular data from visual inputs rather than raw text.

## Datasets

- **Visual-TableQA** — total 6000; splits: test (-1), train (-1); repo https://github.com/AI-4-Everyone/Visual-TableQA

## Metrics

- `Relaxed Accuracy` **(primary)** — range: percent
  - Fraction of correctly answered questions where the predicted answer matches the ground truth within a small numerical or semantic tolerance. The paper notes this metric typically inflates true accuracy by ~5%, which is subtracted for cross-benchmark ranking comparisons.

## Input / output format

**Input**: Image-question pairs formatted into a unified prompt with a system message designed to elicit chain-of-thought reasoning. A LaTeX-code variant (Visual-TableQA-CIT) is also supported.

**Output**: Free-form text response containing the predicted answer and reasoning steps.

## Scoring recipe

```python
def compute_relaxed_accuracy(predictions, gold_answers, jury_models):
    correct_count = 0
    for pred, gold in zip(predictions, gold_answers):
        votes = sum(1 for model in jury_models if model_judge(pred, gold, relaxed=True))
        if votes > len(jury_models) / 2:
            correct_count += 1
    return (correct_count / len(predictions)) * 100
```

## Common pitfalls

- Comparing raw Relaxed Accuracy scores across benchmarks without applying the paper's recommended 5% inflation adjustment for fair ranking.
- Confusing the image-based evaluation (Visual-TableQA) with the LaTeX-code variant (Visual-TableQA-CIT), which yields systematically higher scores (~+6.26%) and tests different capabilities.
- Assuming fine-tuning on this dataset guarantees transfer gains to other benchmarks; the paper shows transfer is asymmetric (e.g., fine-tuning on ReachQA degrades Visual-TableQA performance).

## Evidence (verbatim from paper)

> Performance metrics are measured using Relaxed Accuracy, which allows for small numerical deviations in the predicted answers. We assume that this accuracy inflates the actual accuracy by at least 5%. This margin is subtracted when selecting the best-performing results, which are shown in bold.

## Citation

```bibtex
@misc{lompo2025visualtableqa,
  title={Visual-TableQA: Open-Domain Benchmark for Reasoning over Table Images},
  author={Lompo et al. (2025)},
  year={2025},
  note={arXiv:2509.07966}
}
```

- arXiv: 2509.07966