Pii Tagging Eval

---
name: pii-tagging-eval
description: Evaluates a model's ability to identify and extract private or sensitive information spans from text across legal, healthcare, and finance domains. It probes the model's capacity for fine-grained named entity recognition under limited labeled data and domain-specific privacy definitions. Use when the user wants to benchmark on ECHR, MACCROBAT, PUPA (Finance Subset), or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.15968
  bibtex_key: thareja2026mac
  confidence: high
---

# pii-tagging-eval

> MAC: Multi-Agent Constitution Learning — Thareja et al. (2026) (arXiv:2603.15968, 2026)

## What this evaluates

Evaluates a model's ability to identify and extract private or sensitive information spans from text across legal, healthcare, and finance domains. It probes the model's capacity for fine-grained named entity recognition under limited labeled data and domain-specific privacy definitions.

## Datasets

- **ECHR** — total 192; splits: train (160), val (16), test (16)
- **MACCROBAT** — total 192; splits: train (160), val (16), test (16)
- **PUPA (Finance Subset)** — total 192; splits: train (160), val (16), test (16)

## Metrics

- `F1` **(primary)** — range: percent
  - Computed via exact span-level matching between predicted and ground-truth private spans. Standard for PII/NER tasks.

## Input / output format

**Input**: Raw text documents (case documents, clinical notes, or user-assistant interactions) from legal, healthcare, or finance domains.

**Output**: Predicted private information spans (start/end offsets or text spans) corresponding to fine-grained privacy types.

## Scoring recipe

```python
def compute_f1(predictions, gold):
    pred_set = set(predictions)
    gold_set = set(gold)
    tp = len(pred_set & gold_set)
    fp = len(pred_set - gold_set)
    fn = len(gold_set - pred_set)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1 * 100
```

## Common pitfalls

- Exact span matching requires precise boundary alignment; off-by-one errors or differing tokenization can drastically reduce F1.
- The evaluation uses a fixed 192-document sample per domain to ensure consistency, which may not reflect full dataset performance.
- LLM-as-a-judge is used for agent-level analysis but not for the main benchmark metric, which relies strictly on exact span matching.

## Evidence (verbatim from paper)

> We report F1 computed via exact span-level matching between predicted and ground-truth private spans, which is the standard evaluation metric for this task (Sang and De Meulder, 2003).

## Citation

```bibtex
@misc{thareja2026mac,
  title={MAC: Multi-Agent Constitution Learning},
  author={Thareja et al. (2026)},
  year={2026},
  note={arXiv:2603.15968}
}
```

- arXiv: 2603.15968