Wikitq Eval

---
name: wikitq-eval
description: Evaluates a model's ability to perform complex tabular reasoning to answer open-ended questions based on a provided table. It probes the model's capacity to extract, aggregate, and filter information from structured data to produce short text span answers. Use when the user wants to benchmark on WikiTQ, or asks about evaluating this task. Reports denotation accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.04398
  bibtex_key: wang2024chainoftable
  confidence: high
---

# wikitq-eval

> Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding — Wang et al. (2024) (arXiv:2401.04398, 2024)

## What this evaluates

Evaluates a model's ability to perform complex tabular reasoning to answer open-ended questions based on a provided table. It probes the model's capacity to extract, aggregate, and filter information from structured data to produce short text span answers.

## Datasets

- **WikiTQ** — total ?; splits: test (-1)

## Metrics

- `denotation accuracy` **(primary)** — range: [0, 1]
  - Exact match between the predicted answer and the set of gold-standard answers. A prediction is correct only if it exactly matches one of the valid answers.

## Input / output format

**Input**: A tabular dataset and a natural language question.

**Output**: A short text span representing the answer.

## Scoring recipe

```python
def score(pred, gold):
    return 1.0 if pred in gold else 0.0
accuracy = sum(score(p, g) for p, g in zip(predictions, golds)) / len(golds)
```

## Common pitfalls

- Denotation accuracy requires exact string match against a predefined set of valid answers, not fuzzy or semantic similarity.
- The official evaluation pipeline must be used to handle case sensitivity and whitespace normalization correctly.

## Evidence (verbatim from paper)

> For WikiTQ evaluation, we use the official denotation accuracy*(Pasupat \& Liang, [2015](#bib.bib33 ""))*, and for TabFact, we employ the binary classification accuracy.

## Citation

```bibtex
@misc{wang2024chainoftable,
  title={Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2401.04398}
}
```

- arXiv: 2401.04398