Mlqa Eval

---
name: mlqa-eval
description: Evaluates cross-lingual extractive question answering by measuring how well models can answer questions in one language using context in another, and how performance generalizes across different language pairs. Use when the user wants to benchmark on MLQA, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1910.07475
  bibtex_key: lewis2019mlqa
  confidence: high
---

# mlqa-eval

> MLQA: Evaluating Cross-lingual Extractive Question Answering — Lewis et al. (2019) (arXiv:1910.07475, 2019)

## What this evaluates

Evaluates cross-lingual extractive question answering by measuring how well models can answer questions in one language using context in another, and how performance generalizes across different language pairs.

## Datasets

- **MLQA** — total ?; splits: test (-1); repo https://github.com/facebookresearch/mlqa

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Token-level F1 score measuring the overlap between the predicted answer span and the gold answer span.
- `Exact Match` — range: [0, 1]
  - Binary metric that scores 1 if the predicted answer span exactly matches the gold answer span, and 0 otherwise.

## Input / output format

**Input**: A question in a target language and a context paragraph in either the same or a different language.

**Output**: A text span extracted verbatim from the provided context.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    em_scores = []
    f1_scores = []
    for pred, gold in zip(predictions, golds):
        em_scores.append(1.0 if pred == gold else 0.0)
        pred_tokens = set(pred.split())
        gold_tokens = set(gold.split())
        if not pred_tokens or not gold_tokens:
            f1_scores.append(0.0)
        else:
            common = pred_tokens & gold_tokens
            precision = len(common) / len(pred_tokens)
            recall = len(common) / len(gold_tokens)
            f1 = 2 * precision * recall / (precision + recall)
            f1_scores.append(f1)
    return sum(em_scores) / len(em_scores), sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- MLQA uses a single gold answer annotation per question, unlike SQuAD which uses multiple; comparing directly without adjustment inflates SQuAD scores.
- MLQA contexts are on average 28% longer than SQuAD contexts, which can negatively impact model performance.
- Cross-lingual transfer from English often results in significant performance drops compared to monolingual English baselines, which is a key finding of the benchmark.

## Evidence (verbatim from paper)

> Table 5: F1 score and Exact Match on the MLQA test set for the cross-lingual transfer task (XLT)

## Citation

```bibtex
@misc{lewis2019mlqa,
  title={MLQA: Evaluating Cross-lingual Extractive Question Answering},
  author={Lewis et al. (2019)},
  year={2019},
  note={arXiv:1910.07475}
}
```

- arXiv: 1910.07475