Tanq Eval

---
name: tanq-eval
description: Evaluates open-domain, multi-hop question answering by requiring models to aggregate data from multiple sources and generate structured answer tables. It probes capabilities in multi-hop reasoning, data normalization, unit conversion, and table construction. Use when the user wants to benchmark on TANQ, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.07765
  bibtex_key: akhtar2024tanq
  confidence: high
---

# tanq-eval

> TANQ: An open domain dataset of table answered questions — Akhtar et al. (2024) (arXiv:2405.07765, 2024)

## What this evaluates

Evaluates open-domain, multi-hop question answering by requiring models to aggregate data from multiple sources and generate structured answer tables. It probes capabilities in multi-hop reasoning, data normalization, unit conversion, and table construction.

## Datasets

- **TANQ** — total 1074; splits: test (1074); repo https://github.com/google-deepmind/tanq

## Metrics

- `F1` **(primary)** — range: [0, 100]
  - F1 = 2 * (Precision * Recall) / (Precision + Recall), computed at the cell level over the generated answer table compared to the gold table.

## Input / output format

**Input**: A natural language question q and a set of supporting documents D (oracle setting) or retrieved documents D' (open book/closed book settings).

**Output**: A structured answer table t with n rows and m columns, where each cell contains an extracted or derived entity/value.

## Scoring recipe

```python
def compute_f1(pred_table, gold_table):
    pred_cells = {cell for row in pred_table for cell in row}
    gold_cells = {cell for row in gold_table for cell in row}
    tp = len(pred_cells & gold_cells)
    fp = len(pred_cells - gold_cells)
    fn = len(gold_cells - pred_cells)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- Column ordering in the answer table does not affect the F1 score but can confuse models and human evaluators.
- Some questions partially reveal the answer within the prompt, making the task easier than intended.
- Ambiguous or unclear relations in questions can lead to inconsistent cell extraction.

## Evidence (verbatim from paper)

> TANQ evaluates the capability to answer open domain, multi-hop questions by aggregating data and generating answer tables. ... resulting in a test set of 1,074 TANQ samples for evaluation. ... Table 5: Baseline performance by question type. For all question types, we observe Gemini Flash (60.7 F1) and PaLM-2 (47.6 F1) to outperform other baselines in oracle and closed book setting respectively, lagging 12.3 and 25.4 points behind the human baseline of 73.0.

## Citation

```bibtex
@misc{akhtar2024tanq,
  title={TANQ: An open domain dataset of table answered questions},
  author={Akhtar et al. (2024)},
  year={2024},
  note={arXiv:2405.07765}
}
```

- arXiv: 2405.07765