Urdu Mner Eval

---
name: urdu-mner-eval
description: Evaluates the ability of models to recognize named entities (Person, Location, Organization, Miscellaneous) in Urdu social media posts by jointly processing textual and visual inputs. It probes cross-modal alignment, handling of low-resource language morphological complexity, and ambiguity resolution using visual context. Use when the user wants to benchmark on Twitter2015-Urdu, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.05148
  bibtex_key: ahmad2025urdu
  confidence: high
---

# urdu-mner-eval

> A Benchmark Dataset and a Framework for Urdu Multimodal Named Entity Recognition — Ahmad et al. (2025) (arXiv:2505.05148, 2025)

## What this evaluates

Evaluates the ability of models to recognize named entities (Person, Location, Organization, Miscellaneous) in Urdu social media posts by jointly processing textual and visual inputs. It probes cross-modal alignment, handling of low-resource language morphological complexity, and ambiguity resolution using visual context.

## Datasets

- **Twitter2015-Urdu** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `F1 score` **(primary)** — range: percent
  - Standard token-level or span-level precision, recall, and F1 for named entity recognition. F1 = 2 * (Precision * Recall) / (Precision + Recall). Evaluated per entity type and overall.

## Input / output format

**Input**: Paired Urdu tweet text and associated image. Text is tokenized to a maximum of 128 tokens; images are processed via a fixed-weight ResNet-152.

**Output**: Sequence of BIO/IOB entity labels for each token, corresponding to classes: PER, LOC, ORG, MISC, and O.

## Scoring recipe

```python
def compute_ner_f1(preds, golds):
    tp = fp = fn = 0
    for p, g in zip(preds, golds):
        if set(p) == set(g) and p:
            tp += 1
        elif p and g:
            fp += 1
        elif g:
            fn += 1
    p = tp / (tp + fp) if (tp + fp) > 0 else 0
    r = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
    return f1
```

## Common pitfalls

- Urdu lacks capitalization, making entity boundary detection significantly harder without visual context or character-level features.
- Visual noise (e.g., concert images for 'Google') can mislead models that lack proper cross-modal gating or attention mechanisms.
- Low-resource nature requires careful fine-tuning of pre-trained models (e.g., Urdu-BERT) rather than using standard English BERT out-of-the-box.

## Evidence (verbatim from paper)

> To assess the performance of the MNER models, we utilize the F1 score for each entity type, along with the overall precision (P), recall (R), and F1 score (F1). These metrics provide a comprehensive evaluation of model accuracy and effectiveness.

## Citation

```bibtex
@misc{ahmad2025urdu,
  title={A Benchmark Dataset and a Framework for Urdu Multimodal Named Entity Recognition},
  author={Ahmad et al. (2025)},
  year={2025},
  note={arXiv:2505.05148}
}
```

- arXiv: 2505.05148