WinoBias Eval

---
name: winoBias-eval
description: Evaluates gender bias in coreference resolution systems by measuring performance disparity between pro-stereotypical and anti-stereotypical sentences. It probes whether models rely on gender stereotypes when resolving coreferences in challenging, Winograd-style contexts. Use when the user wants to benchmark on WinoBias, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1804.06876
  bibtex_key: zhao2018genderbias
  confidence: high
---

# winoBias-eval

> Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods — Zhao et al. (2018) (arXiv:1804.06876, 2018)

## What this evaluates

Evaluates gender bias in coreference resolution systems by measuring performance disparity between pro-stereotypical and anti-stereotypical sentences. It probes whether models rely on gender stereotypes when resolving coreferences in challenging, Winograd-style contexts.

## Datasets

- **WinoBias** — total ?; splits: dev (-1), test (-1)

## Metrics

- `F1` **(primary)** — range: [0, 100]
  - Standard coreference F1 score computed using scorer v8.01 (Pradhan et al., 2014).
- `|Diff|` — range: [0, 100]
  - Absolute difference between F1 scores on pro-stereotyped and anti-stereotyped subsets.

## Input / output format

**Input**: Winograd-style sentences containing occupation-based entities with gendered pronouns, presented in pro-stereotypical and anti-stereotypical conditions.

**Output**: Set of coreference links (pairs of entity mentions) for each input sentence.

## Scoring recipe

```python
pred_links = model.predict(sentence)
gold_links = get_gold_coref(sentence)
f1 = scorer_v801.compute_f1(pred_links, gold_links)

f1_pro = compute_f1(pro_stereotypical_sentences)
f1_anti = compute_f1(anti_stereotypical_sentences)
diff = abs(f1_pro - f1_anti)
```

## Common pitfalls

- Using a coreference scorer other than v8.01 (Pradhan et al., 2014), which yields different F1 values.
- Ignoring statistical significance testing for |Diff|; the paper requires an approximate randomized test (p < .05) to flag significant bias.
- Evaluating on standard coreference benchmarks alone, which masks bias that only appears in WinoBias's adversarial pro/anti-stereotypical split.

## Evidence (verbatim from paper)

> Table 2: F1 on OntoNotes and WinoBias development set. WinoBias results are split between Type-1 and Type-2 and in pro/anti-stereotypical conditions.

## Citation

```bibtex
@misc{zhao2018genderbias,
  title={Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods},
  author={Zhao et al. (2018)},
  year={2018},
  note={arXiv:1804.06876}
}
```

- arXiv: 1804.06876