Cross Lingual Pronoun Prediction Eval

---
name: cross-lingual-pronoun-prediction-eval
description: Evaluates systems' ability to predict target-language pronoun class labels from source-language pronouns using lemmatized, POS-tagged translations and word alignments. It probes cross-lingual anaphora resolution and functional ambiguity handling in machine translation pipelines. Use when the user wants to benchmark on WMT 2016 Cross-lingual Pronoun Prediction Task, or asks about evaluating this task. Reports macro-averaged recall.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1911.12091
  bibtex_key: guillou2019wmt2016pronoun
  confidence: high
---

# cross-lingual-pronoun-prediction-eval

> Findings of the 2016 WMT Shared Task on Cross-lingual Pronoun Prediction — Guillou et al. (2019) (arXiv:1911.12091, 2019)

## What this evaluates

Evaluates systems' ability to predict target-language pronoun class labels from source-language pronouns using lemmatized, POS-tagged translations and word alignments. It probes cross-lingual anaphora resolution and functional ambiguity handling in machine translation pipelines.

## Datasets

- **WMT 2016 Cross-lingual Pronoun Prediction Task** — total ?; splits: test (-1)

## Metrics

- `macro-averaged recall` **(primary)** — range: [0, 1]
  - Recall averaged across all classes, computed as the mean of per-class recall values. Ranges from 0 to 1, where 1 is perfect and 1/C is the trivial baseline for C classes.
- `accuracy` — range: [0, 1]
  - Standard classification accuracy, calculated as the proportion of correctly predicted instances out of the total. Noted as sensitive to class imbalance.

## Input / output format

**Input**: Source-language pronouns, lemmatized and POS-tagged target translations, and word alignments.

**Output**: Predicted target-language pronoun class label for each instance.

## Scoring recipe

```python
def macro_recall(predictions, gold, num_classes):
    recalls = []
    for c in range(num_classes):
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        recalls.append(tp / (tp + fn) if (tp + fn) > 0 else 0.0)
    return sum(recalls) / num_classes
```

## Common pitfalls

- Using accuracy or standard F1 as the primary metric can be misleading due to class imbalance in the test set.
- Interpreting absolute accuracy values without comparing them to a dataset-specific majority-class baseline.
- Assuming a random classifier baseline is 0.5; it is actually 1/C where C is the number of classes.

## Evidence (verbatim from paper)

> While in 2015 we used macro-averaged $F_{1}$ as an official evaluation measure, this year we adopted macro-averaged recall, which was also recently adopted by some other competitions, e.g., by SemEval-2016 Task 4 (Nakov et al., 2016). Moreover, as in 2015, we also report accuracy as a secondary evaluation measure. Macro-averaged recall ranges in [0, 1], where a value of 1 is achieved by the perfect classifier, and a value of 0 is achieved by the classifier that mis-classifies all examples.

## Citation

```bibtex
@misc{guillou2019wmt2016pronoun,
  title={Findings of the 2016 WMT Shared Task on Cross-lingual Pronoun Prediction},
  author={Guillou et al. (2019)},
  year={2019},
  note={arXiv:1911.12091}
}
```

- arXiv: 1911.12091