Presidio Pii Detection

---
name: presidio-pii-detection
description: "Author and run Microsoft Presidio PII detection - wraps presidio-analyzer (PII detector) + presidio-anonymizer (replace/redact/mask/hash/encrypt operators) for scanning datasets, log streams, and free-text fields. Covers AnalyzerEngine + AnonymizerEngine setup, built-in recognizers (PERSON, EMAIL_ADDRESS, CREDIT_CARD, US_SSN, IBAN_CODE, country-specific IDs across US/UK/Spain/Italy/Poland/Singapore/Australia/India and more), custom PatternRecognizer authoring, score thresholds, and CI gating. Use when scanning *existing* data for PII (vs synthesising fresh fixtures with synthetic-pii-generator)."
---

# presidio-pii-detection

## Overview

Microsoft Presidio is an open-source SDK for PII detection and
anonymisation. Two engines compose:

- **`presidio-analyzer`** - detects PII entities in free text using
  per-entity recognisers (regex + NER + checksum validation).
- **`presidio-anonymizer`** - applies operators to the detected
  spans (replace, redact, mask, hash, encrypt, custom).

This skill wraps both. For the **categories** of PII that Presidio
detects across regulatory regimes see the pii-categories catalog in
`pii-masking-pipeline-builder` references/;
for the **operator** chosen per field see the masking-techniques
catalog there.

## When to use

- Scanning a database dump, log stream, or document corpus before
  promoting to a non-production environment.
- Building a masking pipeline
  (`pii-masking-pipeline-builder`)
  that needs entity detection upstream of operator application.
- CI gating that fails the build when test fixtures contain
  unexpected PII patterns (e.g., real emails leaked into a JSON
  fixture).

For **generating** fake PII fixtures, use
`synthetic-pii-generator` (in the qa-test-data plugin).
Presidio detects; this is the orthogonal axis.

## Authoring

### Install

Per [presidio.dataprivacystack.org/analyzer](https://presidio.dataprivacystack.org/analyzer/):

```bash
pip install presidio-analyzer presidio-anonymizer
python -m spacy download en_core_web_lg
```

The spaCy model powers the PERSON / LOCATION NER recognisers. For
non-English text use `en_core_web_md` (smaller) or
language-specific spaCy models.

### Basic detection

```python
from presidio_analyzer import AnalyzerEngine

analyzer = AnalyzerEngine()
results = analyzer.analyze(
    text="Contact John Doe at john@example.com or +1 555-123-4567",
    language="en",
)
for r in results:
    print(r.entity_type, r.start, r.end, r.score)
# PERSON 8 16 0.85
# EMAIL_ADDRESS 20 36 1.0
# PHONE_NUMBER 40 54 0.75
```

`AnalyzerEngine()` loads the default NLP model and all built-in
recognisers. Per
[presidio.dataprivacystack.org/analyzer](https://presidio.dataprivacystack.org/analyzer/),
`analyze()` returns a list of `RecognizerResult` with fields
`start`, `end`, `score` (0 - 1 confidence), and `entity_type`.

### Restricting entity types

```python
results = analyzer.analyze(
    text=text,
    language="en",
    entities=["US_SSN", "CREDIT_CARD", "EMAIL_ADDRESS"],
    score_threshold=0.5,
)
```

`entities` whitelists which recognisers run. `score_threshold` (0 - 1)
drops low-confidence hits. Default threshold is 0; raise to 0.4 - 0.6
for noisy text (logs) where partial matches inflate false positives.

### Built-in entity catalog

Presidio ships global recognisers (`CREDIT_CARD`, `EMAIL_ADDRESS`, `IP_ADDRESS`,
`PERSON`, `PHONE_NUMBER`, `IBAN_CODE`, `URL`, and more) plus country-specific IDs
across the US, UK, Spain, Italy, Poland, Singapore, Australia, India, Finland,
Korea, Nigeria, and Thailand (`US_SSN`, `UK_NHS`, `ES_NIF`, `PL_PESEL`,
`SG_NRIC_FIN`, `IN_AADHAAR`, and so on). Pass the ones you need to
`entities=[...]`. The full global and country-specific tables are in
[references/entity-catalog.md](references/entity-catalog.md); for the
authoritative list and descriptions see
[presidio.dataprivacystack.org/supported_entities](https://presidio.dataprivacystack.org/supported_entities/).

### Custom PatternRecognizer

For entities Presidio doesn't ship - internal employee IDs, custom
account-number formats, vendor-specific IDs - extend the analyzer:

```python
from presidio_analyzer import PatternRecognizer, Pattern

employee_id_pattern = Pattern(
    name="employee_id_pattern",
    regex=r"\bEMP-\d{6}\b",
    score=0.9,
)

employee_id_recognizer = PatternRecognizer(
    supported_entity="EMPLOYEE_ID",
    patterns=[employee_id_pattern],
    context=["employee", "staff", "personnel"],  # boosts score when nearby
)

analyzer.registry.add_recognizer(employee_id_recognizer)
```

Per Presidio docs the `Pattern` (regex + score) and `PatternRecognizer`
(supported_entity + patterns + context words + optional validation
function) compose the standard custom-recogniser pattern.

For non-regex detection (ML-based custom NER) extend
`EntityRecognizer` directly.

## Running

### Anonymise after detect

```python
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import OperatorConfig

anonymizer = AnonymizerEngine()
anonymized = anonymizer.anonymize(
    text=text,
    analyzer_results=results,
    operators={
        "PERSON": OperatorConfig("replace", {"new_value": "<PERSON>"}),
        "EMAIL_ADDRESS": OperatorConfig("mask",
            {"chars_to_mask": 8, "masking_char": "*", "from_end": False}),
        "CREDIT_CARD": OperatorConfig("hash",
            {"hash_type": "sha256", "salt": "secret-per-tenant"}),
        "US_SSN": OperatorConfig("redact"),
    },
)
print(anonymized.text)
```

Per [presidio.dataprivacystack.org/anonymizer](https://presidio.dataprivacystack.org/anonymizer/),
`OperatorConfig(operator_name, params={})` is the constructor; the
default operator is `replace` with `<entity_type>` placeholder when
no operator is configured.

See the masking-techniques catalog in
`pii-masking-pipeline-builder` references/
for which operator suits which field.

### Batch processing

For datasets too large to hold in memory, iterate row-by-row:

```python
import csv

with open("input.csv") as src, open("masked.csv", "w") as dst:
    reader = csv.DictReader(src)
    writer = csv.DictWriter(dst, fieldnames=reader.fieldnames)
    writer.writeheader()
    for row in reader:
        for col, val in row.items():
            if val:
                hits = analyzer.analyze(text=val, language="en")
                if hits:
                    row[col] = anonymizer.anonymize(
                        text=val, analyzer_results=hits, operators=ops
                    ).text
        residual = [
            r for v in row.values() if v
            for r in analyzer.analyze(text=v, language="en", score_threshold=0.5)
        ]
        if residual:
            raise ValueError(f"residual PII after masking: {residual}")
        writer.writerow(row)
```

Verify: the re-scan asserts zero residual detections at your threshold before
the row is written. If any remain, the operator map missed an entity type -
add an operator for it (or lower the threshold) and re-run rather than shipping
the row. For Spark / pandas batches see Presidio's structured-data tutorial.

## Parsing results

`RecognizerResult.to_dict()` serialises to JSON; collect across a
scan to feed downstream tools (CI report, quarantine queue):

```python
import json

findings = [r.to_dict() for r in results]
print(json.dumps(findings, indent=2))
# [{"entity_type": "PERSON", "start": 8, "end": 16, "score": 0.85, ...}]
```

To classify findings by regulatory regime, map `entity_type` →
regime via the pii-categories catalog in
`pii-masking-pipeline-builder` references/
(e.g., `US_SSN` → GDPR Art. 4(1) identifier + CPRA SPI + NIST
direct identifier + HIPAA Safe Harbor #7).

## CI integration

Block PRs that introduce real PII into test fixtures:

```yaml
# .github/workflows/pii-fixture-scan.yml
name: pii-fixture-scan
on: pull_request

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-python@v6
        with: { python-version: '3.12' }
      - run: |
          pip install presidio-analyzer
          python -m spacy download en_core_web_lg
      - run: python scripts/pii-fixture-scan.py tests/fixtures/
```

`pii-fixture-scan.py`:

```python
import sys
from pathlib import Path
from presidio_analyzer import AnalyzerEngine

analyzer = AnalyzerEngine()
BLOCKING = {"US_SSN", "CREDIT_CARD", "IBAN_CODE", "EMAIL_ADDRESS"}

violations = []
for path in Path(sys.argv[1]).rglob("*.json"):
    text = path.read_text()
    for r in analyzer.analyze(text=text, language="en", score_threshold=0.5):
        if r.entity_type in BLOCKING:
            violations.append((path, r.entity_type, r.start, r.score))

if violations:
    for v in violations:
        print(f"BLOCK {v[0]}:{v[2]}  {v[1]}  (score {v[3]:.2f})")
    sys.exit(1)
print("No blocking PII found.")
```

Tune `score_threshold` per project - 0.5 balances false positives
(synthetic-looking fixtures) against false negatives (real emails
in random strings).

Each `BLOCK` line is the feedback loop: it names the fixture, byte offset, and
entity type so the author fixes that exact span - swap the real value for a
reserved test value or add it to a known-safe-value allowlist - and re-runs
until the job prints `No blocking PII found.` rather than only seeing a red
build.

## Example - scanning a log line

```python
text = (
    "2026-05-20T10:32:18Z user=alice@acme.com "
    "req_id=r-9182 ip=192.0.2.55 ssn=123-45-6789 "
    "card=4111-1111-1111-1111"
)

results = analyzer.analyze(text=text, language="en")
print([(r.entity_type, text[r.start:r.end]) for r in results])
# [('EMAIL_ADDRESS', 'alice@acme.com'),
#  ('IP_ADDRESS', '192.0.2.55'),
#  ('US_SSN', '123-45-6789'),
#  ('CREDIT_CARD', '4111-1111-1111-1111')]
```

Note: the SSN `123-45-6789` is the
[example-only SSN per SSA guidance](https://www.ssa.gov/employer/randomization.html);
`4111-1111-1111-1111` is a Visa test card from
[Stripe test-cards docs](https://stripe.com/docs/testing). Presidio
does not distinguish "real-looking but reserved-for-testing"
values - it flags the format regardless. Pair the detector with a
known-safe-value allowlist if your test fixtures intentionally use
these reserved values.

## Anti-patterns

| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Skipping spaCy model download | `PERSON` and `LOCATION` recognisers return zero hits silently | Always run `python -m spacy download en_core_web_lg` before first use |
| Default `score_threshold = 0` on log files | Flood of low-confidence PHONE_NUMBER hits on numeric IDs | Raise threshold to 0.4 - 0.6 for log scanning |
| Single regex for SSN | Misses unformatted `123456789` and `123 45 6789` variants | Use the built-in `US_SSN` recogniser; it covers Luhn-like variants |
| No custom recogniser for in-house IDs | Internal employee IDs slip through | Define `PatternRecognizer` per Authoring section |
| `replace` operator with default placeholder for analytics | Loses distribution / cardinality | Use deterministic hash or substitution for analytics-bound output |
| Running analyzer + anonymizer on every request in prod | High latency (NER model is heavy) | Run as batch / offline; or use a lighter recogniser set |
| Trusting Presidio to be regime-complete | Built-in recognisers cover GDPR/CCPA broadly but miss specialised IDs (e.g., medical record numbers); HIPAA #8 not detected by default | Add custom `PatternRecognizer` per regime - see the pii-categories catalog in `pii-masking-pipeline-builder` references/ |

## Limitations

- **NER models drift.** `en_core_web_lg` from spaCy is updated
  periodically; recogniser hits may shift between versions.
- **Score is a heuristic.** A score of 0.85 on PERSON doesn't mean
  85 % chance of correctness - it's a calibration relative to
  Presidio's training data.
- **No formal de-identification guarantee.** Presidio is a
  detect-and-mask toolkit, not a k-anonymity / differential
  privacy engine. For statistical privacy models see the
  masking-techniques catalog in `pii-masking-pipeline-builder`
  references/ on NIST SP 800-188 models.
- **English-default NLP.** Non-English text needs a different
  spaCy model and may have weaker built-in PERSON detection.
- **Free-text only.** Structured-column detection (PII in a column
  with no surrounding text) needs schema-aware logic on top - see
  `pii-masking-pipeline-builder`.

## References

- Presidio Analyzer docs - 
  [presidio.dataprivacystack.org/analyzer](https://presidio.dataprivacystack.org/analyzer/).
- Presidio Anonymizer docs - 
  [presidio.dataprivacystack.org/anonymizer](https://presidio.dataprivacystack.org/anonymizer/).
- Supported entities - 
  [presidio.dataprivacystack.org/supported_entities](https://presidio.dataprivacystack.org/supported_entities/).
- Companion catalogs: pii-categories.md and masking-techniques.md in
  `pii-masking-pipeline-builder` references/.
- Downstream consumers:
  `pii-masking-pipeline-builder`.
- Orthogonal sibling:
  `synthetic-pii-generator` - fixture generation (distinct from detection of existing data).