Ott Qa Eval

---
name: ott-qa-eval
description: This benchmark evaluates a model's ability to perform open-domain question answering by retrieving and fusing evidence from both tabular and textual sources. It specifically probes multi-hop reasoning capabilities where answers require bridging information across separate table segments and text passages. Use when the user wants to benchmark on OTT-QA, or asks about evaluating this task. Reports EM.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2010.10439
  bibtex_key: chen2020ottqa
  confidence: high
---

# ott-qa-eval

> Open Question Answering over Tables and Text — Chen et al. (2020) (arXiv:2010.10439, 2020)

## What this evaluates

This benchmark evaluates a model's ability to perform open-domain question answering by retrieving and fusing evidence from both tabular and textual sources. It specifically probes multi-hop reasoning capabilities where answers require bridging information across separate table segments and text passages.

## Datasets

- **OTT-QA** — total ?; splits: dev (-1), test (-1); repo https://github.com/wenhuchen/OTT-QA

## Metrics

- `EM` **(primary)** — range: percent
  - Exact match accuracy; the predicted answer string must exactly match the ground truth answer string.
- `F1` — range: percent
  - Token-level F1 score measuring the overlap between predicted and ground truth answer tokens.

## Input / output format

**Input**: A natural language question and a set of retrieved evidence blocks (table segments and text passages), typically truncated to 4096 subword tokens for the reader model.

**Output**: A single text string representing the predicted answer.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    em_scores = [1.0 if pred.strip() == gold.strip() else 0.0 for pred, gold in zip(predictions, golds)]
    f1_scores = []
    for pred, gold in zip(predictions, golds):
        p_tokens = set(pred.lower().split())
        g_tokens = set(gold.lower().split())
        if not p_tokens or not g_tokens:
            f1_scores.append(0.0)
            continue
        intersection = len(p_tokens & g_tokens)
        precision = intersection / len(p_tokens)
        recall = intersection / len(g_tokens)
        f1_scores.append(2 * precision * recall / (precision + recall))
    return sum(em_scores) / len(em_scores) * 100, sum(f1_scores) / len(f1_scores) * 100
```

## Common pitfalls

- The evaluation jointly measures retrieval and reading performance; poor entity linking or block retrieval drastically lowers the final EM/F1, masking reader capabilities.
- Using predicted hyperlinks instead of oracle links causes a significant performance drop (~7% EM), highlighting that link prediction is a major bottleneck in this task.
- Single-hop retrieval often fails to capture bridging evidence, so models must use iterative or fusion retrieval strategies to achieve competitive scores.

## Evidence (verbatim from paper)

> By combining the two strategies, the final EM score can reach 28%, with an 18% absolute improvement, which is greater than the sum of individual improvements.

## Citation

```bibtex
@misc{chen2020ottqa,
  title={Open Question Answering over Tables and Text},
  author={Chen et al. (2020)},
  year={2020},
  note={arXiv:2010.10439}
}
```

- arXiv: 2010.10439