Subject Level Inference Eval

---
name: subject-level-inference-eval
description: This benchmark evaluates text anonymization methods by measuring both span-level masking accuracy and subject-level privacy leakage. It probes whether anonymized text successfully prevents adversarial LLMs from inferring personal identifiable information (PII) and sensitive attributes, while maintaining text utility. Use when the user wants to benchmark on PANORAMA, TAB, or asks about evaluating this task. Reports CPR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.21211
  bibtex_key: oh2026subject
  confidence: high
---

# subject-level-inference-eval

> Subject-level Inference for Realistic Text Anonymization Evaluation — Oh et al. (2026) (arXiv:2604.21211, 2026)

## What this evaluates

This benchmark evaluates text anonymization methods by measuring both span-level masking accuracy and subject-level privacy leakage. It probes whether anonymized text successfully prevents adversarial LLMs from inferring personal identifiable information (PII) and sensitive attributes, while maintaining text utility.

## Datasets

- **PANORAMA** — total 151; splits: test (151); repo https://github.com/maisonOP/spia.git
- **TAB** — total 144; splits: test (144); repo https://github.com/maisonOP/spia.git

## Metrics

- `Token Recall ($R_{di+qi}$)` — range: [0, 1]
  - Fraction of ground-truth PII tokens correctly masked or replaced in the anonymized output at the individual mention level.
- `Entity Recall ($ER_{di}$, $ER_{qi}$)` — range: [0, 1]
  - Whether all spans belonging to the same direct identifier ($ER_{di}$) or quasi-identifier ($ER_{qi}$) entity are fully masked.
- `1-AAC` — range: [0, 1]
  - Protection rate for the target subject only (e.g., applicant or author), calculated as 1 minus the attribute inference accuracy for that specific subject.
- `CPR` **(primary)** — range: [0, 1]
  - Coverage Protection Rate; measures protection across all subjects in the text after matching ground-truth subjects with those identified by the adversarial LLM.
- `IPR` — range: [0, 1]
  - Inference Protection Rate; similar to CPR but focuses on attribute inference protection across all subjects.
- `Mean Utility` — range: [0, 1]
  - Average of LLM-based Readability, Meaning, and ROUGE-L scores computed on the anonymized text.

## Input / output format

**Input**: Original text containing personal identifiable information (PII) and multiple subjects.

**Output**: Anonymized text where PII spans are replaced with masking tokens (e.g., [PERSON]) or redacted, generated by an anonymization method and backbone.

## Scoring recipe

```python
def evaluate_span_based(gold_text, anon_text):
    gold_spans = extract_pii_spans(gold_text)
    masked_count = count_tokens_in_output(replaced_with_mask)
    return masked_count / len(gold_spans)

def evaluate_inference_based(gold_text, anon_text, adversary_llm):
    inferred_subjects = adversary_llm.identify_and_infer(anon_text)
    matched = match_subjects(extract_subjects(gold_text), inferred_subjects)
    cpr = 1 - (correct_inferences(matched) / total_subjects)
    ipr = 1 - (attribute_inference_errors(matched) / total_attributes)
    target_subject = get_target(gold_text) # applicant/author
    1_aac = 1 - attribute_inference_accuracy(matched[target_subject])
    return cpr, ipr, 1_aac
```

## Common pitfalls

- High span-based masking scores (e.g., ER_di ≈ 1.0) do not guarantee privacy; inference attacks can still recover ~33% of attributes via context.
- Evaluating only target-subject protection (1-AAC) masks leakage to non-target individuals, as CPR/IPR often differ significantly.
- Utility metrics are averaged across readability, meaning, and ROUGE-L, which may obscure specific degradation in factual or stylistic fidelity.

## Evidence (verbatim from paper)

> After matching Ground Truth subjects from original text with subjects identified from anonymized text, PII-level scoring is performed to calculate CPR and IPR as defined in Section 4. Additionally, 1-AAC is reported to express AAC as a protection rate, which measures protection for the target subject only (applicant for TAB, author for PANORAMA).

## Citation

```bibtex
@misc{oh2026subject,
  title={Subject-level Inference for Realistic Text Anonymization Evaluation},
  author={Oh et al. (2026)},
  year={2026},
  note={arXiv:2604.21211}
}
```

- arXiv: 2604.21211