Multi Hop Qa Eval

---
name: multi-hop-qa-eval
description: Evaluates multi-hop question answering capabilities across diverse reasoning types, including implicit commonsense/arithmetic reasoning, explicit composition/comparison, and fact verification. It tests the model's ability to synthesize information from retrieved evidence and generate step-by-step explanations. Use when the user wants to benchmark on STRATEGYQA, FERMI, QUARTZ, HOTPOTQA, 2WIKIMQA, BAMBOOGLE, FEVEROUS, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.13007
  bibtex_key: yoran2023answering
  confidence: high
---

# multi-hop-qa-eval

> Answering Questions by Meta-Reasoning over Multiple Chains of Thought — Yoran et al. (2023) (arXiv:2304.13007, 2023)

## What this evaluates

Evaluates multi-hop question answering capabilities across diverse reasoning types, including implicit commonsense/arithmetic reasoning, explicit composition/comparison, and fact verification. It tests the model's ability to synthesize information from retrieved evidence and generate step-by-step explanations.

## Datasets

- **STRATEGYQA** — total 1000; splits: dev (1000), test (490)
- **FERMI** — total 286; splits: dev (286), test (558)
- **QUARTZ** — total 374; splits: eval (374)
- **HOTPOTQA** — total 500; splits: eval (500)
- **2WIKIMQA** — total 500; splits: eval (500)
- **BAMBOOGLE** — total 120; splits: eval (120)
- **FEVEROUS** — total 500; splits: eval (500)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Token-level F1 score between the predicted answer and the gold answer. Used for all explicit reasoning datasets.
- `exact-match` — range: [0, 1]
  - Exact string match between the predicted answer and the gold answer. Used for binary-choice datasets.
- `order-of-magnitude` — range: [0, 1]
  - Official FERMI evaluation metric that checks if the predicted answer falls within an order of magnitude of the gold answer.

## Input / output format

**Input**: Question and retrieved context (top-1 evidence sentence or list from Wikipedia via Google Search or ColBERTv2), formatted with in-context exemplars showing step-by-step reasoning.

**Output**: Step-by-step reasoning followed by a final answer. For binary-choice datasets, a Yes/No answer. 'Unknown' if the model is unsure.

## Scoring recipe

```python
def evaluate(predictions, golds, dataset_type):
    if dataset_type == 'explicit':
        return f1_score(predictions, golds)
    elif dataset_type == 'binary':
        return exact_match(predictions, golds)
    elif dataset_type == 'fermi':
        return order_of_magnitude_accuracy(predictions, golds)
    return None
```

## Common pitfalls

- Using the development set instead of the official test set for STRATEGYQA and FERMI when comparing to reported test results.
- Applying exact-match scoring to open-ended explicit reasoning datasets instead of token-level F1.
- Using standard exact-match or F1 for FERMI instead of the official order-of-magnitude evaluation.

## Evidence (verbatim from paper)

> For evaluation, we use F1 to compare predicted and gold answers for all explicit reasoning datasets and exact-match for the binary-choice datasets. In FERMI, we use the official order-of-magnitude evaluation by Kalyan et al. (2021).

## Citation

```bibtex
@misc{yoran2023answering,
  title={Answering Questions by Meta-Reasoning over Multiple Chains of Thought},
  author={Yoran et al. (2023)},
  year={2023},
  note={arXiv:2304.13007}
}
```

- arXiv: 2304.13007