Ner Trigger Efficiency Eval

---
name: ner-trigger-efficiency-eval
description: Evaluates the data-efficiency and labor-cost effectiveness of a trigger-enhanced Named Entity Recognition model compared to a standard baseline. It probes how well the model generalizes when trained on varying fractions of labeled sentences and trigger-annotated data. Use when the user wants to benchmark on CoNLL2003, BC5CDR, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2004.07493
  bibtex_key: lin2020triggerner
  confidence: high
---

# ner-trigger-efficiency-eval

> TriggerNER: Learning with Entity Triggers as Explanations for Named Entity Recognition — Lin et al. (2020) (arXiv:2004.07493, 2020)

## What this evaluates

Evaluates the data-efficiency and labor-cost effectiveness of a trigger-enhanced Named Entity Recognition model compared to a standard baseline. It probes how well the model generalizes when trained on varying fractions of labeled sentences and trigger-annotated data.

## Datasets

- **CoNLL2003** — total ?; splits: train (-1)
- **BC5CDR** — total ?; splits: train (-1)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of Precision and Recall over all correctly identified entity mentions. F1 = 2 * (Precision * Recall) / (Precision + Recall).

## Input / output format

**Input**: Tokenized sentence sequences.

**Output**: Sequence of entity tags (e.g., BIO format) for each token in the input sentence.

## Scoring recipe

```python
def compute_f1(pred_tags, gold_tags):
    pred_ents = extract_mentions(pred_tags)
    gold_ents = extract_mentions(gold_tags)
    tp = len(pred_ents & gold_ents)
    fp = len(pred_ents - gold_ents)
    fn = len(gold_ents - pred_ents)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
```

## Common pitfalls

- Results are reported as curves across varying training data fractions (5% to 70%), not as a single fixed score.
- Trigger annotation requires approximately 1.5x human effort compared to standard entity tagging, which must be factored into labor-efficiency comparisons.
- Self-training uses a fixed top-20% confidence threshold per epoch, which may not generalize without dataset-specific tuning.

## Evidence (verbatim from paper)

> Even if we consider the extreme case that tagging triggers requires twice the human effort ("BLSTM-CRF (x2)"), the TMN is still significantly more labor-efficient in terms of F1 scores.

## Citation

```bibtex
@misc{lin2020triggerner,
  title={TriggerNER: Learning with Entity Triggers as Explanations for Named Entity Recognition},
  author={Lin et al. (2020)},
  year={2020},
  note={arXiv:2004.07493}
}
```

- arXiv: 2004.07493