Turkish Ner Eval

---
name: turkish-ner-eval
description: Tests the ability to identify and classify named entities (Person, Location, Organization) in Turkish text. It probes fine-grained token-level classification and boundary detection. Use when the user wants to benchmark on Milliyet-Ner, WikiANN (Turkish subset), or asks about evaluating this task. Reports CoNLL F-1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2203.01215
  bibtex_key: safaya2022mukayese
  confidence: high
---

# turkish-ner-eval

> Mukayese: Turkish NLP Strikes Back — Safaya et al. (2022) (arXiv:2203.01215, 2022)

## What this evaluates

Tests the ability to identify and classify named entities (Person, Location, Organization) in Turkish text. It probes fine-grained token-level classification and boundary detection.

## Datasets

- **Milliyet-Ner** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/alisafaya/mukayese
- **WikiANN (Turkish subset)** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/alisafaya/mukayese

## Metrics

- `CoNLL F-1` **(primary)** — range: [0, 100]
  - F1 score based on exact match of entity spans and labels. Counts a named entity as correct only if it is an exact match of the corresponding entity in the ground truth.

## Input / output format

**Input**: Tokenized text with word-level annotations.

**Output**: Sequence of entity labels (e.g., BIO format) corresponding to each token.

## Scoring recipe

```python
def score_ner(predictions, gold):
    pred_entities = extract_entities(predictions)
    gold_entities = extract_entities(gold)
    correct = len(set(pred_entities) & set(gold_entities))
    precision = correct / len(pred_entities) if pred_entities else 0
    recall = correct / len(gold_entities) if gold_entities else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1 * 100
```

## Common pitfalls

- CoNLL F-1 requires exact span and label matching; partial overlaps are not counted as correct.
- Human performance is expected to be 98-99%, so models plateau quickly and small differences matter.
- Datasets use only three entity classes, limiting evaluation of fine-grained types.

## Evidence (verbatim from paper)

> Following previous work on Turkish NER, we report the CoNLL F-1 metric to assess our NER baselines. CoNLL F-1 counts a named entity as correct, only if it is an exact match of the corresponding entity in the ground truth.

## Citation

```bibtex
@misc{safaya2022mukayese,
  title={Mukayese: Turkish NLP Strikes Back},
  author={Safaya et al. (2022)},
  year={2022},
  note={arXiv:2203.01215}
}
```

- arXiv: 2203.01215