Dynamic Unlearning Eval

---
name: dynamic-unlearning-eval
description: Evaluates the effectiveness and robustness of LLM unlearning methods by measuring residual knowledge retrieval across dynamically generated single-hop, multi-hop, and alias-based queries, alongside the retention of adjacent and general knowledge. Use when the user wants to benchmark on RWKU, TOFU, or asks about evaluating this task. Reports Multi-hop Forgetting Criterion.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.11266
  bibtex_key: shah2026unlearningmirage
  confidence: high
---

# dynamic-unlearning-eval

> The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning — Raj Sanjay Shah et al. (arXiv:2603.11266, 2026)

## What this evaluates

Evaluates the effectiveness and robustness of LLM unlearning methods by measuring residual knowledge retrieval across dynamically generated single-hop, multi-hop, and alias-based queries, alongside the retention of adjacent and general knowledge.

## Datasets

- **RWKU** — total ?; splits: test (-1)
- **TOFU** — total ?; splits: test (-1)

## Metrics

- `Multi-hop Forgetting Criterion` **(primary)** — range: percent
  - Accuracy of residual knowledge retrieval on dynamically generated 1-hop, 2-hop, and 3-hop queries. Lower values indicate better unlearning performance.
- `Retention Criterion` — range: percent
  - Accuracy of retaining facts 1-fact away, 2-facts away, and relative retention of general knowledge. Higher values indicate better knowledge preservation.
- `Overall Score` — range: percent
  - Aggregated metric balancing forgetting and retention performance across all query types and proximity levels.

## Input / output format

**Input**: Structured natural language queries generated dynamically from knowledge graphs, including single-hop, multi-hop (1- to 3-hop), and alias-perturbed variants, prompted to the LLM.

**Output**: Text completion answering the query. Evaluated for factual accuracy regarding the unlearned entity or retained adjacent facts.

## Scoring recipe

```python
for query in dynamic_benchmark:
  pred = model.generate(query)
  if is_forgetting_query(query):
    resolved = check_factual_accuracy(pred, gold_entity)
    forgetting_accs.append(resolved)
  else:
    retained = check_factual_accuracy(pred, gold_retained_fact)
    retention_accs.append(retained)
forget_score = mean(forgetting_accs) * 100
retain_score = mean(retention_accs) * 100
overall_score = aggregate(forget_score, retain_score)
```

## Common pitfalls

- Assuming static benchmarks fully capture unlearning efficacy; this framework shows multi-hop and alias queries reveal significant residual knowledge missed by static probes.
- Overestimating the effectiveness of Chain-of-Thought decomposition; the paper finds it yields negligible improvement in preventing residual knowledge recovery.
- Ignoring proximity effects; retention accuracy drops for facts directly adjacent (1-hop) to the unlearned target but improves for more distant facts (2-hop).

## Evidence (verbatim from paper)

> Our metric shows the same relative efficacy of methods as previous unlearning evaluation methods. Results for RWKU are summarized in Table 2, showing each unlearning method’s performance across the multi-hop forgetting criterion and the retention criterion.

## Citation

```bibtex
@misc{shah2026unlearningmirage,
  title={The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning},
  author={Raj Sanjay Shah et al.},
  year={2026},
  note={arXiv:2603.11266}
}
```

- arXiv: 2603.11266