Tlunified Ner Eval

---
name: tlunified-ner-eval
description: Evaluates Named Entity Recognition (NER) capabilities on Tagalog news text, specifically measuring performance across Person, Organization, and Location entities using supervised learning and zero-shot LLM prompting. Use when the user wants to benchmark on TLUNIFIED-NER, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.07161
  bibtex_key: miranda2023developing
  confidence: high
---

# tlunified-ner-eval

> Developing a Named Entity Recognition Dataset for Tagalog — Miranda (2023) (arXiv:2311.07161, 2023)

## What this evaluates

Evaluates Named Entity Recognition (NER) capabilities on Tagalog news text, specifically measuring performance across Person, Organization, and Location entities using supervised learning and zero-shot LLM prompting.

## Datasets

- **TLUNIFIED-NER** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/ljvmiranda921/calamanCy

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Standard token-level F1-score for exact span matching, computed per entity class (Person, Organization, Location) and overall. Calculated as the harmonic mean of precision and recall.

## Input / output format

**Input**: Raw Tagalog text sequences from news documents.

**Output**: BILUO sequence encoding scheme (e.g., B-PER, I-PER, B-ORG, I-ORG, B-LOC, I-LOC, O) for each token.

## Scoring recipe

```python
def compute_f1(gold_spans, pred_spans):
    tp = len(gold_spans & pred_spans)
    fp = len(pred_spans - gold_spans)
    fn = len(gold_spans - pred_spans)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1
```

## Common pitfalls

- Models frequently confuse lexical vs. semantic entity tags (e.g., 'Ombudsman' as Person vs Organization).
- Zero-shot LLM prompting exhibits high variance and consistently underperforms supervised baselines on this corpus.
- Evaluation relies on exact span matching; boundary mismatches or incorrect BIO/BILUO formatting will artificially deflate scores.

## Evidence (verbatim from paper)

> Table 4 reports the F1-score on the test set across three trials. We trained several NER models using spaCy's transition-based parser (Honnibal et al., 2020). The state transitions are based on the BILUO sequence encoding scheme and the actions are decided by a convolutional neural network with a maxout (Goodfellow et al., 2013) activation function.

## Citation

```bibtex
@misc{miranda2023developing,
  title={Developing a Named Entity Recognition Dataset for Tagalog},
  author={Miranda (2023)},
  year={2023},
  note={arXiv:2311.07161}
}
```

- arXiv: 2311.07161