Wikitablequestions Eval

---
name: wikitablequestions-eval
description: Evaluates a model's ability to perform semantic parsing over semi-structured tabular data by generating database queries or answers from natural language questions. It measures how well the model aligns textual utterances with table schemas and content to retrieve correct results. Use when the user wants to benchmark on WIKITABLEQUESTIONS, or asks about evaluating this task. Reports execution accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2005.08314
  bibtex_key: yin2020tabert
  confidence: high
---

# wikitablequestions-eval

> TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data — Pengcheng Yin et al. (2020) (arXiv:2005.08314, 2020)

## What this evaluates

Evaluates a model's ability to perform semantic parsing over semi-structured tabular data by generating database queries or answers from natural language questions. It measures how well the model aligns textual utterances with table schemas and content to retrieve correct results.

## Datasets

- **WIKITABLEQUESTIONS** — total ?; splits: dev (-1), test (-1)

## Metrics

- `execution accuracy` **(primary)** — range: percent
  - The percentage of test instances where the generated database query produces the exact same result as the gold query when executed against the target table.

## Input / output format

**Input**: A natural language question (utterance) paired with a database table represented via row linearization and content snapshots (sampled rows).

**Output**: A database query or direct answer string.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, golds):
    if execute_query(pred, table) == execute_query(gold, table):
        correct += 1
return (correct / len(predictions)) * 100
```

## Common pitfalls

- Execution accuracy requires actually running the generated query against the table, not just string-matching the SQL.
- The paper evaluates using 10 random runs and reports mean ± std and best dev-tuned test scores, which can inflate performance if not properly tracked.

## Evidence (verbatim from paper)

> Evaluation Metrics As standard, we report execution accuracy on WIKITABLEQUESTIONS and exact-match accuracy of DB queries on SPIDER.

## Citation

```bibtex
@misc{yin2020tabert,
  title={TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data},
  author={Pengcheng Yin et al. (2020)},
  year={2020},
  note={arXiv:2005.08314}
}
```

- arXiv: 2005.08314