Hybridqa Eval

---
name: hybridqa-eval
description: Multi-hop question answering that requires integrating information from both tabular and textual sources. It probes a model's ability to perform cross-modal reasoning and extract precise answers from heterogeneous data. Use when the user wants to benchmark on HybridQA, or asks about evaluating this task. Reports exact match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2004.07347
  bibtex_key: chen2020hybridqa
  confidence: high
---

# hybridqa-eval

> HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data — Wenhu Chen et al. (arXiv:2004.07347, 2020)

## What this evaluates

Multi-hop question answering that requires integrating information from both tabular and textual sources. It probes a model's ability to perform cross-modal reasoning and extract precise answers from heterogeneous data.

## Datasets

- **HybridQA** — total ?; splits: test (500), dev (-1); repo https://github.com/wenhuchen/HybridQA

## Metrics

- `exact match (EM)` **(primary)** — range: [0, 1]
  - 1 if the predicted answer exactly matches the ground-truth answer string, 0 otherwise.
- `F1` — range: [0, 1]
  - Token-level F1 score measuring the average overlap between the prediction and ground-truth answers.

## Input / output format

**Input**: A natural language question, a retrieved table (or specific cell), and a retrieved text passage.

**Output**: A text span extracted from the table cell or passage that answers the question.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    em_scores = [1.0 if pred == gold else 0.0 for pred, gold in zip(predictions, golds)]
    f1_scores = []
    for pred, gold in zip(predictions, golds):
        p_tokens = set(pred.lower().split())
        g_tokens = set(gold.lower().split())
        if not p_tokens or not g_tokens:
            f1_scores.append(0.0)
            continue
        intersection = len(p_tokens & g_tokens)
        precision = intersection / len(p_tokens)
        recall = intersection / len(g_tokens)
        f1_scores.append(2 * precision * recall / (precision + recall))
    return {'EM': sum(em_scores) / len(em_scores), 'F1': sum(f1_scores) / len(f1_scores)}
```

## Common pitfalls

- Models using only table-only or passage-only inputs achieve <20% accuracy, so cross-modal integration is mandatory.
- Errors cascade across retrieval, ranking, hop, and reading comprehension stages, making step-wise error analysis critical for debugging.
- The retrieval threshold τ trades off recall vs. precision, but the overall model performance remains relatively stable across different τ values.

## Evidence (verbatim from paper)

> Following previous work, we use exact match (EM) and F1 as two evaluation metrics. F1 metric measures the average overlap between the prediction and ground-truth answers.

## Citation

```bibtex
@misc{chen2020hybridqa,
  title={HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data},
  author={Wenhu Chen et al.},
  year={2020},
  note={arXiv:2004.07347}
}
```

- arXiv: 2004.07347