Open Wikitable Eval

---
name: open-wikitable-eval
description: Evaluates open-domain table retrieval and end-to-end question answering over complex table reasoning tasks. It probes a model's ability to retrieve relevant table segments from a corpus and then answer questions using either direct reading or SQL generation. Use when the user wants to benchmark on Open-WikiTable, or asks about evaluating this task. Reports Top-k table retrieval accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.07288
  bibtex_key: kweon2023openwikitable
  confidence: high
---

# open-wikitable-eval

> Open-WikiTable: Dataset for Open Domain Question Answering with Complex Reasoning over Table — Kweon et al. (2023) (arXiv:2305.07288, 2023)

## What this evaluates

Evaluates open-domain table retrieval and end-to-end question answering over complex table reasoning tasks. It probes a model's ability to retrieve relevant table segments from a corpus and then answer questions using either direct reading or SQL generation.

## Datasets

- **Open-WikiTable** — total ?; splits: validation (-1), test (-1); repo https://github.com/seaon0042/Open_

## Metrics

- `Top-k table retrieval accuracy` **(primary)** — range: [0, 1]
  - Percentage of queries where the ground-truth table segment appears in the top-k retrieved results.
- `exact match (EM) accuracy` — range: [0, 1]
  - For readers: exact string match between predicted and gold answer. For parsers: exact match on the execution result of the generated SQL query against the gold answer.

## Input / output format

**Input**: Question text. For retrieval, compared against flattened table segments (tables split into 100-word chunks and appended with descriptions). For end-to-end QA, question concatenated with retrieved table segments.

**Output**: For retrieval: ranked list of table segments. For QA: textual answer or SQL query.

## Scoring recipe

```python
def score_retrieval(retrieved, gold):
    return 1.0 if gold in retrieved else 0.0

def score_reader_em(pred_answer, gold_answer):
    return 1.0 if pred_answer == gold_answer else 0.0

def score_parser_em(pred_sql, gold_answer, tables):
    exec_pred = execute_sql(pred_sql, tables)
    return 1.0 if exec_pred == gold_answer else 0.0
```

## Common pitfalls

- Table splitting into 100-word chunks means models must handle multiple segments per table, which significantly impacts retrieval and QA performance.
- Parser EM relies on SQL execution results, so syntactically different but semantically equivalent SQL queries are treated as correct, unlike strict string matching.
- Decontextualized questions make retrieval artificially easy due to direct information overlap, while paraphrased questions require deeper semantic understanding.

## Evidence (verbatim from paper)

> We use the exact match accuracy (EM) for the evaluation metric. For the parser, EM is computed on the execution result of generated SQLs, as they can be expressed in a diverse form.

## Citation

```bibtex
@misc{kweon2023openwikitable,
  title={Open-WikiTable: Dataset for Open Domain Question Answering with Complex Reasoning over Table},
  author={Kweon et al. (2023)},
  year={2023},
  note={arXiv:2305.07288}
}
```

- arXiv: 2305.07288