Masakhaner Eval

---
name: masakhaner-eval
description: Evaluates named entity recognition (NER) capabilities across ten African languages, probing models' ability to identify PER, ORG, and LOC entities in low-resource, morphologically complex, and culturally specific news text. Use when the user wants to benchmark on MasakhaNER, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2103.11811
  bibtex_key: adelani2021masakhaner
  confidence: high
---

# masakhaner-eval

> MasakhaNER: Named Entity Recognition for African Languages — Adelani et al. (2021) (arXiv:2103.11811, 2021)

## What this evaluates

Evaluates named entity recognition (NER) capabilities across ten African languages, probing models' ability to identify PER, ORG, and LOC entities in low-resource, morphologically complex, and culturally specific news text.

## Datasets

- **MasakhaNER** — total ?; splits: train (-1), test (-1)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Standard NER evaluation metric computed over exact entity span matches. Precision = correct entities / predicted entities, Recall = correct entities / gold entities, F1 = 2 * (Precision * Recall) / (Precision + Recall).

## Input / output format

**Input**: Tokenized text sequences (sentences) from African language news articles.

**Output**: Sequence of BIO/IOB entity labels for each token, indicating PER, ORG, LOC, or O.

## Scoring recipe

```python
def compute_f1(predictions, gold):
    pred_spans = extract_spans(predictions)
    gold_spans = extract_spans(gold)
    tp = len(pred_spans & gold_spans)
    fp = len(pred_spans - gold_spans)
    fn = len(gold_spans - pred_spans)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1
```

## Common pitfalls

- Gazetteer augmentation uses English Wikipedia for languages without native Wikipedia (Luo, Nigerian-Pidgin), which may bias results toward English-named entities.
- Cross-lingual transfer experiments restrict evaluation to PER, ORG, and LOC categories because DATE entities are missing from some source domains/languages.
- Domain adaptation (WikiAnn) uses fixed small splits (100 sentences) for most languages but 1K for Swahili, creating uneven transfer baselines.

## Evidence (verbatim from paper)

> For each language, we train the models on the in-language training data and evaluate on its test data. ... For these experiments, we focus on the PER, ORG, and LOC categories, because the gazetteers from Wikipedia do not contain DATE entities and some source domains and languages that we transfer from do not have the DATE annotation.

## Citation

```bibtex
@misc{adelani2021masakhaner,
  title={MasakhaNER: Named Entity Recognition for African Languages},
  author={Adelani et al. (2021)},
  year={2021},
  note={arXiv:2103.11811}
}
```

- arXiv: 2103.11811