Biouner Eval

---
name: biouner-eval
description: Evaluates the ability of models to perform clinical named entity recognition in Urdu, specifically identifying and classifying biomedical entities like diseases, genes, and proteins within clinical text sequences. It probes sequence labeling capabilities in a low-resource, domain-specific language setting. Use when the user wants to benchmark on BioUNER, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.02904
  bibtex_key: [REDACTED]
  confidence: medium
---

# biouner-eval

> BioUNER: A Benchmark Dataset for Clinical Urdu Named Entity Recognition — Wazir Ali et al. (2026) (arXiv:2604.02904, 2026)

## What this evaluates

Evaluates the ability of models to perform clinical named entity recognition in Urdu, specifically identifying and classifying biomedical entities like diseases, genes, and proteins within clinical text sequences. It probes sequence labeling capabilities in a low-resource, domain-specific language setting.

## Datasets

- **BioUNER** — total 153000; splits: test (-1)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Standard token-level F1 score for sequence labeling, calculated as the harmonic mean of precision and recall over all correctly predicted entity tokens.

## Input / output format

**Input**: A sequence of tokens X = {x_1, x_2, ..., x_N} representing a clinical Urdu sentence.

**Output**: A sequence of labels Y = {y_1, y_2, ..., y_N} where each y_i is a BIES-tagged entity class (e.g., Disease, Gene, Protein) or O.

## Scoring recipe

```python
def calculate_f1(predictions, gold):
    pred_spans = extract_entity_spans(predictions)
    gold_spans = extract_entity_spans(gold)
    tp = len(pred_spans & gold_spans)
    fp = len(pred_spans - gold_spans)
    fn = len(gold_spans - pred_spans)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1
```

## Common pitfalls

- Token-level vs. entity-level matching: NER evaluation requires exact span matching rather than simple token-level accuracy.
- BIES labeling scheme must be strictly followed; misinterpreting B (Begin), I (Inside), E (End), S (Single) tags leads to incorrect entity boundary detection.
- Low-resource language effects: Urdu's script and morphology may cause tokenization mismatches if subword tokenizers are not aligned with the annotation guidelines.

## Evidence (verbatim from paper)

> Formally, the task is to learn a mapping from the input sequence X to an output label sequence Y = {y_1,y_2,...,y_N}, where each label y_i ∈ V and V denotes a predefined set of NE labels such as Disease, Gene, and Protein. In order to comprehensively evaluate the newly proposed BioUNER dataset, we exploit Conditional Random Fields (CRF)... Support Vector Machines (SVM)... LSTM... mBERT... and XLM-RoBERTa... Models are evaluated using F1 score.

## Citation

```bibtex
@misc{wazirali2026biouner,
  title={BioUNER: A Benchmark Dataset for Clinical Urdu Named Entity Recognition},
  author={Wazir Ali et al. (2026)},
  year={2026},
  note={arXiv:2604.02904}
}
```

- arXiv: 2604.02904