Counterfact Edit Eval

---
name: counterfact-edit-eval
description: Evaluates the effectiveness and stability of sequential LLM knowledge editing methods. It measures how well a model updates a specific fact while preserving related paraphrases, neighboring facts, generation fluency, and overall general capabilities over thousands of edits. Use when the user wants to benchmark on CounterFact, GLUE_MMLU_GSM8K_HumanEval_MBPP, or asks about evaluating this task. Reports Efficacy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.15686
  bibtex_key: wang2026rlsedit
  confidence: high
---

# counterfact-edit-eval

> Beyond Hard Writes and Rigid Preservation: Soft Recursive Least-Squares for Lifelong LLM Editing — Xinyu Wang et al. (2026) (arXiv:2601.15686, 2026)

## What this evaluates

Evaluates the effectiveness and stability of sequential LLM knowledge editing methods. It measures how well a model updates a specific fact while preserving related paraphrases, neighboring facts, generation fluency, and overall general capabilities over thousands of edits.

## Datasets

- **CounterFact** — total ?; splits: test (-1)
- **GLUE_MMLU_GSM8K_HumanEval_MBPP** — total ?; splits: test (-1)

## Metrics

- `Efficacy` **(primary)** — range: percent
  - Percentage of generated outputs that successfully rewrite the target fact as instructed.
- `Generalization` — range: percent
  - Percentage of paraphrased prompts that successfully generate the target fact.
- `Specificity` — range: percent
  - Percentage of neighborhood prompts that correctly preserve the original fact (i.e., do not leak the new fact).
- `Fluency` — range: other
  - Generation entropy of the model's output distribution; lower values indicate more fluent or coherent text.
- `Consistency` — range: percent
  - Reference score measuring similarity between generated text and a ground-truth reference completion.

## Input / output format

**Input**: Prompt containing a factual statement to be edited, typically with a masked target entity or a direct replacement instruction.

**Output**: Generated text completion following the prompt.

## Scoring recipe

```python
def score_edit_eval(predictions, golds, paraphrases, neighbors, references):
    efficacy = sum(1 for p in predictions if matches_target(p, golds)) / len(predictions)
    generalization = sum(1 for p in predictions if matches_target(p, golds)) / len(paraphrases)
    specificity = sum(1 for p in predictions if matches_original(p, neighbors)) / len(neighbors)
    fluency = compute_generation_entropy(predictions)
    consistency = compute_reference_similarity(predictions, references)
    return {'Efficacy': efficacy, 'Generalization': generalization, 'Specificity': specificity, 'Fluency': fluency, 'Consistency': consistency}
```

## Common pitfalls

- Sequential editing requires evaluating at multiple checkpoints (e.g., every 1K edits) to track degradation, not just at the final step.
- Fluency is measured via generation entropy, which may not correlate perfectly with human-perceived naturalness.
- CounterFact's neighborhood prompts must be strictly filtered to ensure they do not inadvertently contain the target fact, which would inflate Specificity scores.

## Evidence (verbatim from paper)

> Following prior work, we use the CounterFact dataset*(Meng et al., [2022])*. We report Efficacy (rewrite success), Generalization (paraphrase success), Specificity (neighborhood success), Fluency (generation entropy), and Consistency (reference score).

## Citation

```bibtex
@misc{wang2026rlsedit,
  title={Beyond Hard Writes and Rigid Preservation: Soft Recursive Least-Squares for Lifelong LLM Editing},
  author={Xinyu Wang et al. (2026)},
  year={2026},
  note={arXiv:2601.15686}
}
```

- arXiv: 2601.15686