Mt Geneval Eval

---
name: mt-geneval-eval
description: Evaluates machine translation models' ability to correctly translate gender-specific words and maintain gender agreement across multiple languages. It also measures representational bias by comparing translation quality between male and female counterfactual sentence pairs. Use when the user wants to benchmark on MT-GenEval, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2211.01355
  bibtex_key: [REDACTED]
  confidence: high
---

# mt-geneval-eval

> MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation — Currey et al. (2022) (arXiv:2211.01355, 2022)

## What this evaluates

Evaluates machine translation models' ability to correctly translate gender-specific words and maintain gender agreement across multiple languages. It also measures representational bias by comparing translation quality between male and female counterfactual sentence pairs.

## Datasets

- **MT-GenEval** — total ?; splits: test (-1), contextual (-1), counterfactual (-1); repo https://github.com/amazon-research/machine-translation-gender-eval

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Identify words in the contrastive reference that are not in the correct reference (unique_con = w_con \ w_ref). A segment is marked incorrect if the hypothesis contains any word from unique_con; otherwise it is correct. Accuracy is the fraction of correct segments.
- `Δ_qual` — range: other
  - Δ_qual = BLEU_male - BLEU_female, where BLEU_gender is the standard BLEU score computed on the subset of the counterfactual test set corresponding to that gender. Measures representational bias.

## Input / output format

**Input**: Source sentence in a source language (e.g., English) to be translated into a target language with gendered morphology.

**Output**: Translated hypothesis sentence in the target language.

## Scoring recipe

```python
# accuracy
unique_con = set(w_con) - set(w_ref)
is_correct = len(unique_con.intersection(set(w_hyp))) == 0
accuracy = sum(is_correct) / len(segments)

# Δ_qual
bleu_male = compute_bleu(hypotheses_male, references_male)
bleu_female = compute_bleu(hypotheses_female, references_female)
delta_qual = bleu_male - bleu_female
```

## Common pitfalls

- The accuracy metric relies on exact word overlap with contrastive references, so tokenization differences or morphological variations in the hypothesis can cause false negatives.
- On the counterfactual subset, a pair is only counted as correct if BOTH the original and its counterfactual counterpart are marked correct, which penalizes inconsistent models.
- The quality gap uses standard BLEU, which may not reflect gender-specific errors if the overall translation fluency is high.

## Evidence (verbatim from paper)

> We define accuracy of gender in translation on our test set as follows. Let $w_{hyp}$, $w_{ref}$, and $w_{con}$ denote the set of words in the hypothesis, reference, and contrastive reference, respectively. First, we obtain the set of words in the contrastive reference that are not in the correct reference: $unique_{con} = w_{con} \backslash w_{ref}$. We consider a segment incorrect if: $unique_{con} \cap w_{hyp} \neq \emptyset$. ... We define $\Delta_{\text{qual}}$ as: $\Delta_{\text{qual}} = \mathrm{BLEU}_{\mathrm{male}} - \mathrm{BLEU}_{\mathrm{female}}$ where $\mathrm{BLEU}_{\mathrm{gender}}$ is the BLEU score of the gender subset of the counterfactual test set.

## Citation

```bibtex
@misc{currey2022mtgeneval,
  title={MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation},
  author={Currey et al. (2022)},
  year={2022},
  note={arXiv:2211.01355}
}
```

- arXiv: 2211.01355