Masakhaner20 Eval

---
name: masakhaner20-eval
description: Evaluates named entity recognition (NER) capabilities across 20 typologically and geographically diverse African languages. It probes zero-shot cross-lingual transfer performance and measures how well models generalize to unseen entities and languages when fine-tuned on limited African language data. Use when the user wants to benchmark on MasakhaNER 2.0, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.12391
  bibtex_key: adelani2022masakhaner20
  confidence: high
---

# masakhaner20-eval

> MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition — Adelani et al. (2022) (arXiv:2210.12391, 2022)

## What this evaluates

Evaluates named entity recognition (NER) capabilities across 20 typologically and geographically diverse African languages. It probes zero-shot cross-lingual transfer performance and measures how well models generalize to unseen entities and languages when fine-tuned on limited African language data.

## Datasets

- **MasakhaNER 2.0** — total ?; splits: train (-1), test (-1)

## Metrics

- `F1` **(primary)** — range: percent
  - Standard entity-level F1 score computed over exact match of predicted and gold entity spans and types. Calculated as the harmonic mean of precision and recall.

## Input / output format

**Input**: Tokenized text sequences with corresponding token-level entity annotations (e.g., BIO tags) for each of the 20 target African languages.

**Output**: Token-level entity labels predicting the type and span of each named entity in the input sequence.

## Scoring recipe

```python
def compute_f1(predictions, gold):
    tp = sum(1 for p in predictions if p in gold)
    fp = len(predictions) - tp
    fn = len(gold) - tp
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1 * 100
```

## Common pitfalls

- Models are evaluated using exact span and type matching; partial overlap or fuzzy matching is not used.
- Results are averaged over 5 independent training runs with different random seeds, not just a single run.
- Zero-shot transfer evaluation requires training on a subset of languages and testing on held-out languages without any fine-tuning on the target language.

## Evidence (verbatim from paper)

> We fine-tune the PLMs on each language's training data and evaluate performance on the test set using HuggingFace Transformers (Wolf et al., 2020). Table 4 shows the results of training NER models on each language using the eight multilingual and Africa-centric PLMs. ... Average is over 5 runs.

## Citation

```bibtex
@misc{adelani2022masakhaner20,
  title={MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition},
  author={Adelani et al. (2022)},
  year={2022},
  note={arXiv:2210.12391}
}
```

- arXiv: 2210.12391