Universal Ner V2 Eval

---
name: universal-ner-v2-eval
description: Evaluates multilingual named entity recognition (NER) capabilities across 22 languages and 30 datasets, probing both in-language performance and cross-lingual transfer. It also benchmarks large language models as annotators against human inter-annotator agreement to assess guideline adherence and annotation quality. Use when the user wants to benchmark on UNER v2, or asks about evaluating this task. Reports micro F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.12744
  bibtex_key: blevins2026universalner
  confidence: high
---

# universal-ner-v2-eval

> Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark — Blevins et al. (2026) (arXiv:2604.12744, 2026)

## What this evaluates

Evaluates multilingual named entity recognition (NER) capabilities across 22 languages and 30 datasets, probing both in-language performance and cross-lingual transfer. It also benchmarks large language models as annotators against human inter-annotator agreement to assess guideline adherence and annotation quality.

## Datasets

- **UNER v2** — total 3000000; splits: train (-1), dev (-1), test (-1)

## Metrics

- `micro F1` **(primary)** — range: [0, 1]
  - Standard F1 score computed as 2 * (precision * recall) / (precision + recall), where precision and recall are calculated over predicted versus gold entity spans and their corresponding per/loc/org tags. Micro averaging aggregates true positives, false positives, and false negatives across all datasets and languages before computing the final score.

## Input / output format

**Input**: Sentences from the benchmark datasets, optionally accompanied by the official 3-tag (per/loc/org) annotation guidelines when prompting LLMs.

**Output**: Predicted entity spans with assigned labels (per, loc, or org) for each input sentence.

## Scoring recipe

```python
def compute_f1(predictions, gold):
    pred_spans = set((s, e, l) for s, e, l in predictions)
    gold_spans = set((s, e, l) for s, e, l in gold)
    tp = len(pred_spans & gold_spans)
    fp = len(pred_spans - gold_spans)
    fn = len(gold_spans - pred_spans)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
```

## Common pitfalls

- Cross-lingual transfer performance drops sharply for typologically distant languages (e.g., Japanese, Korean) compared to European languages.
- LLMs systematically over-annotate non-named entities in some languages (e.g., English) but under-annotate in others (e.g., Hebrew, Korean), failing to consistently follow nuanced guidelines like distinguishing geopolitical entities from locations.
- Entity-level F1 varies significantly by type: 'org' is consistently the hardest to predict, while 'per' is the easiest across most datasets.

## Evidence (verbatim from paper)

> Figure 3 reports the micro F1 scores on all test sets when XLM-R${}_{\text{Large}}$ is finetuned on different languages. ... Figure 4 presents the inter-annotator agreement between each model and the human annotator who annotated the most documents in a given dataset. ... F1 score comparison across 19 multilingual NER datasets (test sets) for three LLMs against human Inter-Annotator Agreement (IAA) baseline.

## Citation

```bibtex
@misc{blevins2026universalner,
  title={Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark},
  author={Blevins et al. (2026)},
  year={2026},
  note={arXiv:2604.12744}
}
```

- arXiv: 2604.12744