Tab Eval

---
name: tab-eval
description: Evaluates NLP models' ability to identify and mask personally identifiable information (direct and quasi-identifiers) in legal texts while preserving non-sensitive content. It probes both privacy protection (full coverage of masking spans) and information utility (minimizing unnecessary masking of non-identifying entities). Use when the user wants to benchmark on TAB corpus, or asks about evaluating this task. Reports ER_{di}.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2202.00443
  bibtex_key: pilan2022tab
  confidence: high
---

# tab-eval

> The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization — Pilán et al. (2022) (arXiv:2202.00443, 2022)

## What this evaluates

Evaluates NLP models' ability to identify and mask personally identifiable information (direct and quasi-identifiers) in legal texts while preserving non-sensitive content. It probes both privacy protection (full coverage of masking spans) and information utility (minimizing unnecessary masking of non-identifying entities).

## Datasets

- **TAB corpus** — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/NorskRegnesentral/text-anonymization-benchmark

## Metrics

- `R_{di+qi}` — range: [0, 1]
  - Standard mention-level recall on all identifiers (direct + quasi), micro-averaged over annotators. Computed as true positive mentions divided by total gold mentions.
- `P_{di+qi}` — range: [0, 1]
  - Standard mention-level precision on all identifiers, micro-averaged over annotators. Computed as true positive mentions divided by total predicted mentions.
- `ER_{di}` **(primary)** — range: [0, 1]
  - Entity-level recall for direct identifiers. Measures the proportion of direct identifiers that are fully covered by predicted masking spans.
- `ER_{qi}` — range: [0, 1]
  - Entity-level recall for quasi identifiers. Measures the proportion of quasi-identifiers that are fully covered by predicted masking spans.
- `WP_{di+qi}` — range: [0, 1]
  - Weighted utility metric. Computes precision but weights each unnecessarily masked term by its information content, rather than treating all terms uniformly like standard precision.

## Input / output format

**Input**: Raw text documents (e.g., ECHR court cases or biographies) containing potential personal identifiers.

**Output**: IOB sequence labels per token indicating whether each token should be masked (MASK) or not (NO_MASK), corresponding to DIRECT, QUASI, or NO_MASK entity types.

## Scoring recipe

```python
def score_tab(pred_spans, gold_spans):
    # pred/gold_spans: list of (start, end, type) where type in {DIRECT, QUASI, NO_MASK}
    pred_mentions = {(s,e) for s,e,t in pred_spans if t in {DIRECT, QUASI}}
    gold_mentions = {(s,e) for s,e,t in gold_spans if t in {DIRECT, QUASI}}
    R_di_qi = len(pred_mentions & gold_mentions) / len(gold_mentions) if gold_mentions else 0.0
    P_di_qi = len(pred_mentions & gold_mentions) / len(pred_mentions) if pred_mentions else 0.0
    ER_di = entity_recall(pred_spans, gold_spans, type_filter=DIRECT)
    ER_qi = entity_recall(pred_spans, gold_spans, type_filter=QUASI)
    WP_di_qi = weighted_precision(pred_spans, gold_spans)
    return {'R_di+qi': R_di_qi, 'P_di+qi': P_di_qi, 'ER_di': ER_di, 'ER_qi': ER_qi, 'WP_di+qi': WP_di_qi}
```

## Common pitfalls

- Conflating direct and quasi identifiers in a single recall measure hides poor performance on direct identifiers, which are most harmful for privacy.
- Using standard precision/recall without weighting for information content overestimates utility preservation because it treats all masked terms equally regardless of their actual privacy risk.
- NER-based systems often mask non-identifying entities (false positives) because they rely on predefined semantic categories rather than context-specific masking decisions.

## Evidence (verbatim from paper)

> Presidio's results illustrate the importance of computing separate recall measures for the direct and the quasi identifiers: although the standard, mention-level recall seems relatively good at first sight (around 0.7), a closer look at the entity-level recall over direct identifiers $ER_{di}$ shows a much poorer performance (around 0.45).

## Citation

```bibtex
@misc{pilan2022tab,
  title={The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization},
  author={Pilán et al. (2022)},
  year={2022},
  note={arXiv:2202.00443}
}
```

- arXiv: 2202.00443