Ner Eval

---
name: ner-eval
description: Evaluates named entity recognition (NER) models under data-scarce conditions, specifically low-resource settings with no human-annotated training labels and few-shot settings with minimal labeled examples. It probes the model's ability to identify and classify entity types in text using automatically generated pseudo-dictionaries and weak supervision. Use when the user wants to benchmark on CoNLL-2003, Wikigold, WNUT-16, NCBI-disease, BC5CDR, CHEMDNER, or asks about evaluating this task. Reports F1score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2112.08808
  bibtex_key: kim2021simple
  confidence: high
---

# ner-eval

> Simple Questions Generate Named Entity Recognition Datasets — Kim et al. (2021) (arXiv:2112.08808, 2021)

## What this evaluates

Evaluates named entity recognition (NER) models under data-scarce conditions, specifically low-resource settings with no human-annotated training labels and few-shot settings with minimal labeled examples. It probes the model's ability to identify and classify entity types in text using automatically generated pseudo-dictionaries and weak supervision.

## Datasets

- **CoNLL-2003** — total ?; splits: test (-1)
- **Wikigold** — total ?; splits: test (-1)
- **WNUT-16** — total ?; splits: test (-1)
- **NCBI-disease** — total ?; splits: test (-1)
- **BC5CDR** — total ?; splits: test (-1)
- **CHEMDNER** — total ?; splits: test (-1)

## Metrics

- `F1score` **(primary)** — range: [0, 1]
  - Entity-level F1 score computed from precision and recall over all correctly identified entity spans. Macro-averaged across datasets in low-resource experiments.

## Input / output format

**Input**: Raw text sequences (sentences, abstracts, or tweets) for token-level entity prediction. In few-shot settings, a small set of labeled training examples (10-20 sentences) is also provided for fine-tuning.

**Output**: Token-level entity type labels indicating the class and boundaries of named entities in the input text.

## Scoring recipe

```python
def compute_ner_f1(pred_spans, gold_spans):
    pred_set = set(pred_spans)
    gold_set = set(gold_spans)
    tp = len(pred_set & gold_set)
    fp = len(pred_set - gold_set)
    fn = len(gold_set - pred_set)
    p = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    r = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * p * r / (p + r) if (p + r) > 0 else 0.0
```

## Common pitfalls

- The paper excludes 'miscellaneous' entity types in low-resource experiments but includes them in few-shot experiments for fair baseline comparison, which can cause score inconsistencies if not handled carefully.
- Hyperparameters and model checkpoints are selected using validation sets due to the absence of human-annotated training labels, deviating from standard train/val/test splits.
- Entity subtype definitions vary significantly across datasets (e.g., 'organization' in CoNLL-2003 vs Wikigold), requiring dataset-specific query formulation rather than a one-size-fits-all dictionary.

## Evidence (verbatim from paper)

> We used entity-level precision (P), recall (R), and F1score (F1) as the evaluation metrics.

## Citation

```bibtex
@misc{kim2021simple,
  title={Simple Questions Generate Named Entity Recognition Datasets},
  author={Kim et al. (2021)},
  year={2021},
  note={arXiv:2112.08808}
}
```

- arXiv: 2112.08808