Pebench Eval

---
name: pebench-eval
description: Evaluates multimodal large language models on their ability to selectively forget specific person or event concepts while preserving general knowledge. It probes cross-concept interference, unlearning efficacy, and the trade-off between forgetting targeted data and maintaining model utility. Use when the user wants to benchmark on PEBench, or asks about evaluating this task. Reports Efficacy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.12545
  bibtex_key: xu2025pebench
  confidence: high
---

# pebench-eval

> PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Models — Zhaopan Xu et al. (arXiv:2503.12545, 2025)

## What this evaluates

Evaluates multimodal large language models on their ability to selectively forget specific person or event concepts while preserving general knowledge. It probes cross-concept interference, unlearning efficacy, and the trade-off between forgetting targeted data and maintaining model utility.

## Datasets

- **PEBench** — total 8000; splits: forget (-1), retain (-1)

## Metrics

- `Efficacy` **(primary)** — range: [0, 1]
  - Precision measuring the proportion of correctly unlearned target samples (person or event). Higher values indicate successful forgetting.
- `Generality` — range: [0, 1]
  - Precision measuring unlearning success on related or unseen samples of the target concept, testing generalization of the forgetting process.
- `Retain` — range: [0, 1]
  - Precision measuring the preservation of non-target knowledge on the retain set after unlearning.
- `Scope` — range: [0, 1]
  - ROUGE-L score measuring the preservation of related descriptive text (e.g., event descriptions when unlearning a person, or vice versa).
- `Real` — range: [0, 1]
  - POPE metric evaluating the preservation of factual image content and visual grounding after unlearning.
- `World Fact` — range: [0, 1]
  - POPE metric evaluating the preservation of general world knowledge and reasoning capabilities post-unlearning.

## Input / output format

**Input**: Multimodal instances containing an image and a text prompt/question targeting either a specific person or event depicted in the image.

**Output**: Text response generated by the MLLM, evaluated against ground-truth answers or using automated metrics (Precision, ROUGE-L, G-Eval, POPE).

## Scoring recipe

```python
def compute_metric(predictions, gold, metric_name):
    if metric_name in ['Efficacy', 'Generality', 'Retain', 'Scope', 'Real']:
        return precision_score(predictions, gold)
    elif metric_name in ['Event Efficacy', 'Event Generality']:
        return g_eval_score(predictions, gold)
    elif metric_name in ['Real', 'World Fact']:
        return poep_score(predictions, gold)
    return 0.0
```

## Common pitfalls

- Cross-concept interference: unlearning a person often degrades event recognition and vice versa due to semantic coupling in the same image.
- Trade-off sensitivity: increasing unlearning steps or forget set size (5% to 15%) consistently improves efficacy but sharply degrades utility metrics like Retain and Scope.
- Early stopping is guided by training loss rather than validation metrics, which may lead to suboptimal utility preservation.

## Evidence (verbatim from paper)

> The performance metrics include Efficacy, Generality, Retain, Real, and World Fact. A higher score represents better performance.

## Citation

```bibtex
@misc{xu2025pebench,
  title={PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Models},
  author={Zhaopan Xu et al.},
  year={2025},
  note={arXiv:2503.12545}
}
```

- arXiv: 2503.12545