Ms Marco Eval

---
name: ms-marco-eval
description: Evaluates machine reading comprehension models on real-world search queries across multiple answer types (numeric, yes/no, descriptive) and tasks (answer generation, span extraction, passage ranking). Probes a model's ability to extract or generate accurate answers from noisy, multi-document web contexts and handle unanswerable questions. Use when the user wants to benchmark on MS MARCO, or asks about evaluating this task. Reports ROUGE-L.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1611.09268
  bibtex_key: [REDACTED]
  confidence: high
---

# ms-marco-eval

> MS MARCO: A Human Generated MAchine Reading COmprehension Dataset — Bajaj et al. (2016) (arXiv:1611.09268, 2016)

## What this evaluates

Evaluates machine reading comprehension models on real-world search queries across multiple answer types (numeric, yes/no, descriptive) and tasks (answer generation, span extraction, passage ranking). Probes a model's ability to extract or generate accurate answers from noisy, multi-document web contexts and handle unanswerable questions.

## Datasets

- **MS MARCO** — total ?; splits: train (-1), test (-1)

## Metrics

- `ROUGE-L` **(primary)** — range: [0, 1]
  - Longest common subsequence overlap between predicted and reference answers, normalized by reference length. Used as the headline metric for descriptive and generative QA tasks.
- `BLEU` — range: [0, 1]
  - Geometric mean of modified n-gram precisions (typically up to 4-grams) with a brevity penalty. Applied to subsets with multiple human references.
- `pa-BLEU` — range: [0, 1]
  - Pairwise BLEU that computes similarity between a prediction and multiple human references, then aggregates scores to measure consensus across diverse phrasings.
- `Accuracy` — range: [0, 1]
  - Exact match rate for numeric and yes/no answer categories.

## Input / output format

**Input**: Question paired with one or more candidate web passages (context). For cloze-style tasks, a text sequence with a masked entity.

**Output**: Generated natural language answer, predicted answer span from the passage, or a confidence score for the selected span.

## Scoring recipe

```python
def compute_rouge_l(pred, refs):
    best_ref = max(refs, key=lambda r: lcs_length(pred, r))
    lcs_len = lcs_length(pred, best_ref)
    return lcs_len / len(best_ref.split())

def compute_accuracy(pred, gold):
    return 1.0 if normalize(pred) == normalize(gold) else 0.0

def compute_bleu(pred, refs):
    return standard_bleu_score(refs, pred)
```

## Common pitfalls

- Using a single reference answer for descriptive questions instead of the curated multi-reference set, which underestimates model performance and ignores natural phrasing diversity.
- Applying exact-match accuracy to open-ended descriptive answers where paraphrasing is expected and ROUGE-L should be used instead.
- Ignoring the unanswerable question subset in the novice task, leading to inflated scores on models that always predict an answer.

## Evidence (verbatim from paper)

> We use accuracy and precision-recall measures for numeric answers and apply metrics like ROUGE-L (Lin, 2004) and phrasing-aware evaluation framework (Mitra et al., 2016) for long textual answers. The evaluation requires several reference answers per question that are each curated by a different human editor, thus providing a natural way to estimate how diversely a group of individuals may phrase the answer to the same question.

## Citation

```bibtex
@misc{bajaj2016msmarco,
  title={MS MARCO: A Human Generated MAchine Reading COmprehension Dataset},
  author={Bajaj et al. (2016)},
  year={2016},
  note={arXiv:1611.09268}
}
```

- arXiv: 1611.09268