Privacylens Eval

---
name: privacylens-eval
description: Assesses an LLM agent's ability to understand and follow privacy norms while performing real-world tasks. It measures both helpfulness and the rate at which sensitive information is incorrectly exposed. Use when the user wants to benchmark on PrivacyLens, or asks about evaluating this task. Reports privacy leakage rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.03991
  bibtex_key: bao2025galaxy
  confidence: high
---

# privacylens-eval

> Galaxy: A Cognition-Centered Framework for Proactive, Privacy-Preserving, and Self-Evolving LLM Agents — Bao et al. (2025) (arXiv:2508.03991, 2025)

## What this evaluates

Assesses an LLM agent's ability to understand and follow privacy norms while performing real-world tasks. It measures both helpfulness and the rate at which sensitive information is incorrectly exposed.

## Datasets

- **PrivacyLens** — total ?; splits: test (-1)

## Metrics

- `privacy leakage rate` **(primary)** — range: percent
  - Percentage of tasks where the agent incorrectly exposes sensitive information or violates privacy norms during execution.

## Input / output format

**Input**: Real-world task prompts requiring the agent to handle sensitive user data or privacy-sensitive contexts.

**Output**: Agent actions and responses that must balance task completion with privacy preservation.

## Scoring recipe

```python
leaked = 0
total = len(tasks)
for task in tasks:
  if task.privacy_violated:
    leaked += 1
return (leaked / total) * 100
```

## Common pitfalls

- Confusing helpfulness metrics with privacy leakage rates.
- Not masking sensitive content before cloud transmission when evaluating privacy protection.

## Evidence (verbatim from paper)

> PrivacyLens measures the ability of LLM agents to understand and follow privacy norms when performing real-world tasks. It uses helpfulness, privacy leakage rate and accuracy to comprehensively evaluate the privacy protection capabilities.

## Citation

```bibtex
@misc{bao2025galaxy,
  title={Galaxy: A Cognition-Centered Framework for Proactive, Privacy-Preserving, and Self-Evolving LLM Agents},
  author={Bao et al. (2025)},
  year={2025},
  note={arXiv:2508.03991}
}
```

- arXiv: 2508.03991