Conll2012 Coref Eval

---
name: conll2012-coref-eval
description: Evaluates a model's ability to jointly detect mentions and resolve coreferential chains in English text without relying on external syntactic parsers or hand-crafted features. It measures how well the model groups word spans into entity clusters based on contextual and structural cues. Use when the user wants to benchmark on CoNLL-2012 (English), or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1707.07045
  bibtex_key: lee2017endtoend
  confidence: high
---

# conll2012-coref-eval

> End-to-end Neural Coreference Resolution — Lee et al. (2017) (ACL 2017, 2017)

## What this evaluates

Evaluates a model's ability to jointly detect mentions and resolve coreferential chains in English text without relying on external syntactic parsers or hand-crafted features. It measures how well the model groups word spans into entity clusters based on contextual and structural cues.

## Datasets

- **CoNLL-2012 (English)** — total 3493; splits: train (2802), dev (343), test (348)

## Metrics

- `F1` **(primary)** — range: percent
  - Average F1 score of three cluster-level metrics: MUC, B³, and CEAFφ4. Computed over both mention detection and coreference clustering, following the official CoNLL-2012 evaluation protocol.

## Input / output format

**Input**: Raw English documents (paragraphs of text).

**Output**: A set of coreference clusters, where each cluster contains a list of mention spans (start, end indices) that refer to the same entity.

## Scoring recipe

```python
def compute_f1(pred_clusters, gold_clusters):
    muc_f1 = muc_score(pred_clusters, gold_clusters)
    b3_f1 = b3_score(pred_clusters, gold_clusters)
    ceaf_f1 = ceaf_phi4_score(pred_clusters, gold_clusters)
    return (muc_f1 + b3_f1 + ceaf_f1) / 3.0
```

## Common pitfalls

- Aggressive pruning (max span width 10, max antecedents 250) can hurt recall on long documents if not replicated exactly.
- Official CoNLL scorer requires exact mention boundary matching; slight boundary shifts penalize F1 heavily.
- Ensemble averaging of scores vs. majority voting changes results significantly; the paper averages mention and antecedent scores across 5 models.

## Evidence (verbatim from paper)

> We use the English coreference resolution data from the CoNLL-2012 shared task in our experiments. This dataset contains 2802 training documents, 343 development documents, and 348 test documents. Achieves +1.5 F1 on OntoNotes and +3.1 F1 with a 5-model ensemble by directly optimizing marginal likelihood of gold clusters.

## Citation

```bibtex
@misc{lee2017endtoend,
  title={End-to-end Neural Coreference Resolution},
  author={Lee et al. (2017)},
  year={2017},
  note={ACL 2017}
}
```

- arXiv: 1707.07045