Tweebank Ner Eval

---
name: tweebank-ner-eval
description: Evaluates named entity recognition (NER) and syntactic NLP capabilities on noisy, informal social media text. It probes a model's ability to handle domain-specific challenges like abbreviations, irregular capitalization, and complex entity structures common in tweets, while establishing baselines for tokenization, lemmatization, POS tagging, and dependency parsing. Use when the user wants to benchmark on Tweebank-NER (TB2), or asks about evaluating this task. Reports entity-level F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2201.07281
  bibtex_key: jiang2022tweebank
  confidence: high
---

# tweebank-ner-eval

> Annotating the Tweebank Corpus on Named Entity Recognition and Building NLP Models for Social Media Analysis — Jiang et al. (2022) (arXiv:2201.07281, 2022)

## What this evaluates

Evaluates named entity recognition (NER) and syntactic NLP capabilities on noisy, informal social media text. It probes a model's ability to handle domain-specific challenges like abbreviations, irregular capitalization, and complex entity structures common in tweets, while establishing baselines for tokenization, lemmatization, POS tagging, and dependency parsing.

## Datasets

- **Tweebank-NER (TB2)** — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/social-machines/TweebankNLP

## Metrics

- `entity-level F1` **(primary)** — range: [0, 1]
  - Standard F1 score calculated over extracted entity spans rather than individual tokens. Precision and recall are computed by comparing predicted entity sets against gold entity sets.
- `F1` — range: [0, 1]
  - Token-level F1 score used for tokenization and lemmatization tasks. Computed as the harmonic mean of token-level precision and recall.
- `accuracy` — range: [0, 1]
  - Percentage of correctly predicted POS tags out of the total number of tokens in the test set.
- `UAS` — range: [0, 1]
  - Unlabeled Attachment Score: percentage of tokens where the predicted head token matches the gold head, regardless of the dependency relation label.
- `LAS` — range: [0, 1]
  - Labeled Attachment Score: percentage of tokens where both the predicted head token and the dependency relation label match the gold standard.

## Input / output format

**Input**: Raw Twitter text or pre-tokenized token sequences.

**Output**: For NER: entity labels (PER, LOC, ORG, MISC, O) per token. For syntactic tasks: token boundaries, lemmas, UPOS tags, and dependency head/label pairs.

## Scoring recipe

```python
def entity_level_f1(preds, gold):
    pred_entities = set(extract_spans(preds))
    gold_entities = set(extract_spans(gold))
    tp = len(pred_entities & gold_entities)
    fp = len(pred_entities - gold_entities)
    fn = len(gold_entities - pred_entities)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1
```

## Common pitfalls

- Using token-level F1 instead of entity-level F1 for NER evaluation, which inflates scores and misrepresents span-level performance.
- Evaluating on combined training data (TB2+WNUT17) instead of the held-out TB2 test set, violating the reported protocol.
- Ignoring domain shift: blending Twitter data with formal corpora (UD_English-EWT) often degrades performance on the TB2 test set, contrary to expectations from formal-domain baselines.

## Evidence (verbatim from paper)

> We pick the best models based on the corresponding dev sets and report their performance on their TB2 test sets. For each task, we compare Stanza models with existing studies and alternative NLP frameworks. Table 3: NER comparison on the TB2 test set in entity-level F1.

## Citation

```bibtex
@misc{jiang2022tweebank,
  title={Annotating the Tweebank Corpus on Named Entity Recognition and Building NLP Models for Social Media Analysis},
  author={Jiang et al. (2022)},
  year={2022},
  note={arXiv:2201.07281}
}
```

- arXiv: 2201.07281