SpaCy Ner Eval

---
name: spaCy-ner-eval
description: Evaluates named entity recognition performance across diverse domains and languages, specifically probing a model's ability to handle out-of-vocabulary words and morphological variation through hash-based embeddings versus traditional lookup embeddings. Use when the user wants to benchmark on CoNLL 2002, WNUT 2017, AnEM, Dutch Archaeology, OntoNotes 5.0, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2212.09255
  bibtex_key: miranda2022multihash
  confidence: high
---

# spaCy-ner-eval

> Multi hash embeddings in spaCy — Miranda et al. (2022) (arXiv:2212.09255, 2022)

## What this evaluates

Evaluates named entity recognition performance across diverse domains and languages, specifically probing a model's ability to handle out-of-vocabulary words and morphological variation through hash-based embeddings versus traditional lookup embeddings.

## Datasets

- **CoNLL 2002** — total ?; splits: train (-1), val (-1), test (-1)
- **WNUT 2017** — total ?; splits: train (-1), test (-1)
- **AnEM** — total ?; splits: train (-1), dev (-1), test (-1)
- **Dutch Archaeology** — total ?; splits: train (-1), dev (-1), test (-1)
- **OntoNotes 5.0** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Standard token-level F1 score computed over exact entity span matches: F1 = 2 * (precision * recall) / (precision + recall), where precision and recall are calculated over predicted vs. gold entity spans.

## Input / output format

**Input**: Tokenized text with sentence/document boundaries and gold entity annotations.

**Output**: BILUO sequence tags (Begin, In, Last, Unit, Out) for each token, mapping to entity types.

## Scoring recipe

```python
def compute_f1(gold_spans, pred_spans):
    tp = len(gold_spans & pred_spans)
    precision = tp / len(pred_spans) if pred_spans else 0.0
    recall = tp / len(gold_spans) if gold_spans else 0.0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
```

## Common pitfalls

- Unseen entity evaluation strictly ignores spans verbatim present in training, which differs from standard exact-match or substring matching.
- Custom random splits for AnEM and Dutch Archaeology deviate from canonical benchmarks, limiting direct comparability.
- Document-level segmentation for Dutch CoNLL 2002 vs sentence-level for others affects context availability.

## Evidence (verbatim from paper)

> Unseen entities are evaluated by ignoring all known entity spans during evaluation. Evaluated on the test set.

## Citation

```bibtex
@misc{miranda2022multihash,
  title={Multi hash embeddings in spaCy},
  author={Miranda et al. (2022)},
  year={2022},
  note={arXiv:2212.09255}
}
```

- arXiv: 2212.09255