Persian Ner Eval

---
name: persian-ner-eval
description: Evaluates the quality and cross-lingual transferability of machine-translated Persian named entity recognition datasets by measuring model performance against original English benchmarks. It probes how well translation-based dataset generation preserves entity boundaries and labels across languages with different scripts and linguistic structures. Use when the user wants to benchmark on CoNLL 2003, OntoNotes 5.0, NCBI Disease, WNUT 2017, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2302.09611
  bibtex_key: sartipi2023mtner
  confidence: high
---

# persian-ner-eval

> Exploring the Potential of Machine Translation for Generating Named Entity Datasets: A Case Study between Persian and English — Amir Sartipi et al. (2023) (arXiv:2302.09611, 2023)

## What this evaluates

Evaluates the quality and cross-lingual transferability of machine-translated Persian named entity recognition datasets by measuring model performance against original English benchmarks. It probes how well translation-based dataset generation preserves entity boundaries and labels across languages with different scripts and linguistic structures.

## Datasets

- **CoNLL 2003** — total ?; splits: test (-1)
- **OntoNotes 5.0** — total ?; splits: test (-1)
- **NCBI Disease** — total ?; splits: test (-1)
- **WNUT 2017** — total ?; splits: test (-1)

## Metrics

- `F1` **(primary)** — range: percent
  - Harmonic mean of precision and recall: 2 * (P * R) / (P + R). Reported as a percentage in the results table.
- `Precision` — range: percent
  - Ratio of correctly predicted named entities to all predicted entities.
- `Recall` — range: percent
  - Ratio of correctly predicted named entities to all actual named entities in the gold standard.

## Input / output format

**Input**: Raw text sequences (sentences/documents) in English or Persian.

**Output**: Token-level or span-level named entity labels (e.g., B-PER, I-LOC, O).

## Scoring recipe

```python
def compute_ner_f1(pred_spans, gold_spans):
    pred_set = set(pred_spans)
    gold_set = set(gold_spans)
    tp = len(pred_set & gold_set)
    fp = len(pred_set - gold_set)
    fn = len(gold_set - pred_set)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return f1 * 100
```

## Common pitfalls

- Translation-based dataset generation often introduces surface form mismatches and translation errors that disproportionately hurt performance on noisy or novel entity types (e.g., WNUT 2017).
- Performance varies significantly between monolingual (Pars-Bert) and multilingual (xlm-roberta-base) models across datasets, making cross-model comparisons non-trivial without controlling for language coverage.
- F1 scores are reported as percentages in the results table, which can cause confusion if compared against decimal-scale benchmarks or if not explicitly converted.

## Evidence (verbatim from paper)

> Table 2 presents the results of our evaluation for both the English and translated versions of the main datasets. The rows where the Model column is Δ tar-src show the difference between the English and Persian datasets. Our results indicate that the largest discrepancy between the source and target datasets in terms of F1 score was observed for the WNUT 2017 dataset, while the smallest difference was noted for the NCBI Disease dataset.

## Citation

```bibtex
@misc{sartipi2023mtner,
  title={Exploring the Potential of Machine Translation for Generating Named Entity Datasets: A Case Study between Persian and English},
  author={Amir Sartipi et al. (2023)},
  year={2023},
  note={arXiv:2302.09611}
}
```

- arXiv: 2302.09611