Vietmed Ner Eval

---
name: vietmed-ner-eval
description: Evaluates the ability of NER models to identify and classify medically defined entity spans in Vietnamese spoken text. It specifically probes robustness to ASR-generated noise and compares monolingual vs. multilingual, encoder vs. seq2seq architectures. Use when the user wants to benchmark on VietMed-NER, or asks about evaluating this task. Reports micro F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.13337
  bibtex_key: leduc2024medicalspokenner
  confidence: high
---

# vietmed-ner-eval

> Medical Spoken Named Entity Recognition — Khai Le-Duc et al. (2024) (arXiv:2406.13337, 2024)

## What this evaluates

Evaluates the ability of NER models to identify and classify medically defined entity spans in Vietnamese spoken text. It specifically probes robustness to ASR-generated noise and compares monolingual vs. multilingual, encoder vs. seq2seq architectures.

## Datasets

- **VietMed-NER** — total ?; splits: test (-1)

## Metrics

- `micro F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall, micro-averaged across all entity instances and types. Calculated as 2 * (Precision * Recall) / (Precision + Recall).

## Input / output format

**Input**: Raw text sequences, either ground-truth reference transcripts or ASR-generated transcripts.

**Output**: Sequence of token-level entity labels indicating the start and end of named entity spans with their corresponding medical entity types.

## Scoring recipe

```python
def compute_ner_metrics(predictions, gold):
    pred_spans = extract_spans(predictions)
    gold_spans = extract_spans(gold)
    tp = len(pred_spans & gold_spans)
    fp = len(pred_spans - gold_spans)
    fn = len(gold_spans - pred_spans)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1
```

## Common pitfalls

- ASR output introduces significant noise, causing a consistent performance drop across all models compared to reference text.
- Multilingual models may suffer from capacity dilution, where per-language performance decreases as the number of pre-trained languages increases.
- Seq2seq generative models underperform encoder-only models for NER, likely because generation is less suited for span classification.

## Evidence (verbatim from paper)

> Table 4: NER results on ASR output of test set for different NER and ASR models. Metrics shown are Precision, Recall, and overall micro F1 score.

## Citation

```bibtex
@misc{leduc2024medicalspokenner,
  title={Medical Spoken Named Entity Recognition},
  author={Khai Le-Duc et al. (2024)},
  year={2024},
  note={arXiv:2406.13337}
}
```

- arXiv: 2406.13337