Counterfactual Text Gen Eval

---
name: counterfactual-text-gen-eval
description: This benchmark evaluates the effectiveness and linguistic quality of counterfactual text generation methods. It probes a model's ability to modify input text to flip a target classifier's predicted label while preserving grammatical correctness, fluency, and coherence, highlighting the trade-off between label-flipping success and text quality. Use when the user wants to benchmark on IMDB, SNLI, or asks about evaluating this task. Reports flip rate (FR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.17475
  bibtex_key: nguyen2024ceval
  confidence: high
---

# counterfactual-text-gen-eval

> CEval: A Benchmark for Evaluating Counterfactual Text Generation — Van Bach Nguyen et al. (2024) (arXiv:2404.17475, 2024)

## What this evaluates

This benchmark evaluates the effectiveness and linguistic quality of counterfactual text generation methods. It probes a model's ability to modify input text to flip a target classifier's predicted label while preserving grammatical correctness, fluency, and coherence, highlighting the trade-off between label-flipping success and text quality.

## Datasets

- **IMDB** — total ?; splits: test (-1); repo https://github.com/aix-group/CEval-Counterfactual-Generation-Benchmark
- **SNLI** — total ?; splits: test (-1); repo https://github.com/aix-group/CEval-Counterfactual-Generation-Benchmark

## Metrics

- `flip rate (FR)` **(primary)** — range: [0, 1]
  - The proportion of generated counterfactuals that successfully change the target classifier's predicted label compared to the original text. Calculated as (number of successful flips) / (total generated instances).
- `text quality (Grammar, Cohesiveness, Fluency)` — range: [1, 5]
  - Scores assigned by LLM judges (GPT-3.5 Turbo and Mistral) evaluating grammatical correctness, logical flow, and naturalness of the generated text. Typically rated on a Likert-like scale.
- `perplexity` — range: positive real
  - A measure of how well a probability model predicts a sample, computed here using GPT-2. Lower values indicate more fluent and predictable text.
- `edit distance (Levenshtein)` — range: non-negative integer
  - The minimum number of single-character edits (insertions, deletions, or substitutions) required to change the original text into the generated counterfactual.
- `diversity` — range: non-negative
  - A measure of variation across generated counterfactuals, often calculated via unique token/n-gram counts or type-token ratios.
- `probability change ($\Delta$P)` — range: [0, 1]
  - The absolute difference in the target classifier's predicted probability for the original label versus the counterfactual label. Indicates how far the generation pushes the instance away from the decision boundary.

## Input / output format

**Input**: Original text instance and a prompt instructing the model to generate a counterfactual version that flips the target classifier's label.

**Output**: A single generated counterfactual text string.

## Scoring recipe

```python
def score_counterfactuals(predictions, golds, target_classifier):
    flip_count = 0
    ed_scores = []
    for pred, gold in zip(predictions, golds):
        if target_classifier.predict(pred) != target_classifier.predict(gold):
            flip_count += 1
        ed_scores.append(levenshtein_distance(gold, pred))
    fr = flip_count / len(predictions)
    avg_ed = sum(ed_scores) / len(ed_scores)
    # LLM quality scores computed via separate API calls per instance
    return {'flip_rate': fr, 'avg_edit_distance': avg_ed}
```

## Common pitfalls

- Optimizing solely for label flipping often degrades text quality, grammar, and fluency; the benchmark explicitly highlights this trade-off rather than treating flip rate as the sole success metric.
- LLM-based evaluators (GPT, Mistral) may exhibit bias toward LLM-generated outputs over human-written text, skewing quality scores upward for models like LLAMA-2.
- High diversity scores strongly correlate with high edit distance (r=0.93), so they should not be interpreted as independent measures of generation variation.

## Evidence (verbatim from paper)

> Interestingly, MICE has the highest flip rate (FR), but not the largest change in target label probability change ($\Delta$P) on the IMDB dataset.

## Citation

```bibtex
@misc{nguyen2024ceval,
  title={CEval: A Benchmark for Evaluating Counterfactual Text Generation},
  author={Van Bach Nguyen et al. (2024)},
  year={2024},
  note={arXiv:2404.17475}
}
```

- arXiv: 2404.17475