Piibench Eval

---
name: piibench-eval
description: This benchmark probes the ability of NER and PII detection systems to accurately identify and classify personally identifiable information spans across highly heterogeneous, cross-domain text sources. It specifically evaluates cross-domain generalization and robustness to diverse, fine-grained PII entity types that are rarely seen together in standard training corpora. Use when the user wants to benchmark on PIIBench, or asks about evaluating this task. Reports span-level F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.15776
  bibtex_key: jha2026piibench
  confidence: high
---

# piibench-eval

> PIIBench: A Unified Multi-Source Benchmark Corpus for Personally Identifiable Information Detection — Jha (2026) (arXiv:2604.15776, 2026)

## What this evaluates

This benchmark probes the ability of NER and PII detection systems to accurately identify and classify personally identifiable information spans across highly heterogeneous, cross-domain text sources. It specifically evaluates cross-domain generalization and robustness to diverse, fine-grained PII entity types that are rarely seen together in standard training corpora.

## Datasets

- **PIIBench** — total 2370000; splits: test (1398); repo https://github.com/pritesh-2711/pii-bench

## Metrics

- `span-level F1` **(primary)** — range: [0, 1]
  - Computed using seqeval with exact match criteria: both the character span boundaries and the entity type label must exactly match the ground truth. Precision and Recall are reported alongside F1.
- `Precision` — range: [0, 1]
  - Fraction of predicted spans that exactly match ground truth span boundaries and entity type labels.
- `Recall` — range: [0, 1]
  - Fraction of ground truth spans that are exactly matched by predicted span boundaries and entity type labels.

## Input / output format

**Input**: Raw text sequences. Models output predicted character-offset spans, which are then mapped to the 48 canonical PIIBench entity types.

**Output**: BIO token labels derived from character-to-token alignment of predicted spans, or directly character-offset spans mapped to canonical PII types.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    # 1. Map predicted character offsets to BIO token labels
    aligned_preds = char_offsets_to_bio(predictions, gold_tokens)
    # 2. Discard any predicted entity types not in PIIBench's 48-type taxonomy
    aligned_preds = filter_canonical_types(aligned_preds, canonical_types)
    # 3. Compute exact-match span-level metrics using seqeval
    precision, recall, f1 = seqeval.evaluate(aligned_preds, gold_bio_labels)
    return {'precision': precision, 'recall': recall, 'f1': f1}
```

## Common pitfalls

- Predicted entity types must be mapped to PIIBench's 48 canonical labels using a hand-constructed alignment table; types not in the taxonomy are discarded rather than treated as errors.
- Models output character-offset spans, which must be aligned to BIO token labels before evaluation; raw offsets cannot be directly compared to token-level gold standards.
- Specialized models (e.g., financial NER) often exhibit high precision but near-zero recall on mixed-domain test sets due to domain-silo effects, skewing overall F1.

## Evidence (verbatim from paper)

> Evaluation uses span-level seqeval metrics (Precision, Recall, F1), which require exact match of both the entity span boundaries and the entity type label. We reconstruct each system's predicted character-offset spans back to BIO token labels using a character-to-token alignment procedure, then apply seqeval to the aligned predictions.

## Citation

```bibtex
@misc{jha2026piibench,
  title={PIIBench: A Unified Multi-Source Benchmark Corpus for Personally Identifiable Information Detection},
  author={Jha (2026)},
  year={2026},
  note={arXiv:2604.15776}
}
```

- arXiv: 2604.15776