Papillon Eval

---
name: papillon-eval
description: Evaluates a privacy-preserving LLM delegation pipeline that sanitizes user queries before sending them to a remote API model. It measures the trade-off between maintaining response quality and minimizing personally identifiable information (PII) leakage in the sanitized prompts. Use when the user wants to benchmark on PUPA-TNB, or asks about evaluating this task. Reports QUAL.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.17127
  bibtex_key: li2024papillon
  confidence: high
---

# papillon-eval

> PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensembles — Li Siyan et al. (2024) (arXiv:2410.17127, 2024)

## What this evaluates

Evaluates a privacy-preserving LLM delegation pipeline that sanitizes user queries before sending them to a remote API model. It measures the trade-off between maintaining response quality and minimizing personally identifiable information (PII) leakage in the sanitized prompts.

## Datasets

- **PUPA-TNB** — total ?; splits: test (-1)

## Metrics

- `QUAL` **(primary)** — range: other
  - Pairwise LLM judge score: 1 if response A is at least as good as response B, else 0. Averaged across instances and scaled to 0-100. Ties are counted as alignment with A.
- `LEAK` — range: other
  - Raw count of PII units in the sanitized prompt generated by the local model. Scaled to 0-100 (lower leakage is better).

## Input / output format

**Input**: User query from the PUPA-TNB dataset.

**Output**: Sanitized prompt (for leakage evaluation) and final response from the remote model (for quality evaluation).

## Scoring recipe

```python
def compute_qual(pipeline_resp, original_resp):
    # LLM judge returns 1 if pipeline_resp >= original_resp, else 0
    return 1 if llm_judge_at_least_as_good(pipeline_resp, original_resp) else 0

def compute_leak(sanitized_prompt):
    # Count PII units in the prompt
    return count_pii_units(sanitized_prompt)

# Aggregate over dataset of size N
qual_scores = [compute_qual(r, g) for r, g in zip(responses, originals)]
leak_scores = [compute_leak(p) for p in prompts]

qual_avg = (sum(qual_scores) / N) * 100
leak_avg = (sum(leak_scores) / N) * 100  # scaled to 0-100
```

## Common pitfalls

- The QUAL metric is a pairwise comparison (A vs B), not an absolute score, and explicitly counts ties as alignment with response A.
- The LEAK metric measures PII leakage in the *sanitized prompt* sent to the remote model, not in the final generated response.
- PUPA-TNB is a held-out test set explicitly not used for prompt optimization, so results should not be conflated with training/optimization performance.

## Evidence (verbatim from paper)

> When QUAL(A,B) = 0, the alignment rate is 70.8%, i.e. participants prefer B 70.8% of the time. When QUAL(A,B) = 1, we obtain an alignment of 65.4%, considering both majority voting for A and for "It is a tie" as alignment. This establishes the general validity of our quality metric.

## Citation

```bibtex
@misc{li2024papillon,
  title={PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensembles},
  author={Li Siyan et al. (2024)},
  year={2024},
  note={arXiv:2410.17127}
}
```

- arXiv: 2410.17127