Mfaq Eval

---
name: mfaq-eval
description: Evaluates multilingual and monolingual bi-encoder models on a FAQ retrieval task across 21 languages. It probes cross-lingual knowledge transfer, semantic robustness to lexical changes, and the impact of training data distribution on retrieval performance. Use when the user wants to benchmark on MFAQ, or asks about evaluating this task. Reports MRR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2109.12870
  bibtex_key: debruyn2021mfaq
  confidence: high
---

# mfaq-eval

> MFAQ: a Multilingual FAQ Dataset — De Bruyn et al. (2021) (arXiv:2109.12870, 2021)

## What this evaluates

Evaluates multilingual and monolingual bi-encoder models on a FAQ retrieval task across 21 languages. It probes cross-lingual knowledge transfer, semantic robustness to lexical changes, and the impact of training data distribution on retrieval performance.

## Datasets

- **MFAQ** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/clips/mfaq

## Metrics

- `MRR` **(primary)** — range: percent
  - Mean Reciprocal Rank: the average of 1/rank for the ground-truth answer across all queries, where rank is the position of the correct answer in the retrieved list.
- `P@1` — range: percent
  - Precision at 1: 1 if the ground-truth answer is ranked first, 0 otherwise.
- `R@5` — range: percent
  - Recall at 5: 1 if the ground-truth answer appears in the top 5 retrieved results, 0 otherwise.

## Input / output format

**Input**: A question string (prepended with <question>) and a candidate answer string (prepended with <answer>). The model encodes them into a shared embedding space for similarity-based retrieval.

**Output**: A ranked list of candidate answers for each query question, based on cosine similarity between question and answer embeddings.

## Scoring recipe

```python
def compute_metrics(retrieved_ranks, gold_idx):
    p_at_1 = 0.0
    mrr = 0.0
    r_at_5 = 0.0
    for ranks in retrieved_ranks:
        if gold_idx in ranks:
            rank = ranks.index(gold_idx) + 1
            mrr += 1.0 / rank
            if rank == 1: p_at_1 += 1.0
            if rank <= 5: r_at_5 += 1.0
    n = len(retrieved_ranks)
    return {'P@1': p_at_1 / n, 'MRR': mrr / n, 'R@5': r_at_5 / n}
```

## Common pitfalls

- Batch size differences between models (e.g., DPR vs RoBERTa) can significantly impact reported performance, as noted in the text.
- Filtering training data to one page per domain flattens topic distribution and artificially inflates MRR compared to using the full unfiltered dataset.
- Cross-lingual evaluation translates queries to English while keeping answers in the original language, which tests translation robustness rather than pure multilingual retrieval.

## Evidence (verbatim from paper)

> In all our experiments, we use three metrics to evaluate the performance: precision-at-one $(\mathrm{P}@\mathrm{1})$ , mean reciprocal rank (MRR), and recall-at-5  $(\mathrm{R}@\mathrm{5})$ . For space reasons, we only report on MRR in the main text, the full results are available in the annex.

## Citation

```bibtex
@misc{debruyn2021mfaq,
  title={MFAQ: a Multilingual FAQ Dataset},
  author={De Bruyn et al. (2021)},
  year={2021},
  note={arXiv:2109.12870}
}
```

- arXiv: 2109.12870