Table Qa Eval

---
name: table-qa-eval
description: Evaluates a model's ability to answer questions about tabular data using various reasoning strategies. It probes factual retrieval, numerical reasoning, and complex multi-step table understanding across different difficulty levels. Use when the user wants to benchmark on Penguins in a Table, TableBench, or asks about evaluating this task. Reports Exact Match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.09642
  bibtex_key: hyeon2026mata
  confidence: high
---

# table-qa-eval

> MATA: Multi-Agent Framework for Reliable and Flexible Table Question Answering — Hyeon et al. (2026) (arXiv:2602.09642, 2026)

## What this evaluates

Evaluates a model's ability to answer questions about tabular data using various reasoning strategies. It probes factual retrieval, numerical reasoning, and complex multi-step table understanding across different difficulty levels.

## Datasets

- **Penguins in a Table** — total ?; splits: test (-1)
- **TableBench** — total ?; splits: test (-1)

## Metrics

- `Exact Match (EM)` **(primary)** — range: [0, 1]
  - 1 if the predicted answer string exactly matches the ground truth string, else 0.
- `Fuzzy matching` — range: [0, 1]
  - Textual similarity score based on Levenshtein distance, computed using the fuzzywuzzy library.
- `SQuAD-style token-level F1 score` — range: [0, 1]
  - Token-level overlap between prediction and ground truth, calculated as the harmonic mean of precision and recall at the token level.

## Input / output format

**Input**: A table (in tabular format) and a natural language question.

**Output**: A single textual answer string.

## Scoring recipe

```python
def score(prediction, ground_truth):
    em = 1.0 if prediction.strip() == ground_truth.strip() else 0.0
    fuzzy = fuzz.ratio(prediction, ground_truth) / 100.0
    pred_tokens = set(prediction.lower().split())
    gt_tokens = set(ground_truth.lower().split())
    if not pred_tokens and not gt_tokens:
        f1 = 1.0
    elif not pred_tokens or not gt_tokens:
        f1 = 0.0
    else:
        prec = len(pred_tokens & gt_tokens) / len(pred_tokens)
        rec = len(pred_tokens & gt_tokens) / len(gt_tokens)
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return {'EM': em, 'Fuzzy': fuzzy, 'F1': f1}
```

## Common pitfalls

- Exact Match (EM) is overly strict for TableQA and fails to capture semantically correct but syntactically different answers.
- Baselines like TabLaP and MixSC were originally tuned for closed-source GPT models and show significant performance drops when applied to open-source or smaller LLMs.
- Excessive LLM inference steps can degrade performance on simple tasks, so limiting reasoning paths is sometimes beneficial.

## Evidence (verbatim from paper)

> First, we used fuzzy matching222https://pypi.org/project/fuzzywuzzy/, a metric widely adopted in studies *King and Flanigan ([2024]); Cheng et al. ([2024]); Nekvinda and Dušek ([2021])* to measure textual similarity based on Levenshtein distance *Levenshtein ([1966])*. Second, we adopted the SQuAD-style token-level F1 score *Rajpurkar et al. ([2016])*, which evaluates token-level overlap between the prediction and the ground truth. By incorporating these two metrics, we complement the strictness of EM with more flexible and nuanced evaluations.

## Citation

```bibtex
@misc{hyeon2026mata,
  title={MATA: Multi-Agent Framework for Reliable and Flexible Table Question Answering},
  author={Hyeon et al. (2026)},
  year={2026},
  note={arXiv:2602.09642}
}
```

- arXiv: 2602.09642