Counterfactual Rep Eval

---
name: counterfactual-rep-eval
description: Evaluates how well counterfactual representations (CFRs) in high-dimensional embedding space mimic true text counterfactuals, measuring prediction consistency, probability alignment, and downstream fairness improvements across synthetic and real-world biased datasets. Use when the user wants to benchmark on EEEC+, BiasInBios, or asks about evaluating this task. Reports PIP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.00711
  bibtex_key: lemberger2024counterfactual
  confidence: high
---

# counterfactual-rep-eval

> Explaining Text Classifiers with Counterfactual Representations — Lemberger et al. (2024) (arXiv:2402.00711, 2024)

## What this evaluates

Evaluates how well counterfactual representations (CFRs) in high-dimensional embedding space mimic true text counterfactuals, measuring prediction consistency, probability alignment, and downstream fairness improvements across synthetic and real-world biased datasets.

## Datasets

- **EEEC+** — total ?; splits: test (-1); repo http://github.com/toinesayan/counterfactual-representations-for-explanation
- **BiasInBios** — total ?; splits: test (-1)

## Metrics

- `PIP` **(primary)** — range: [0, 1]
  - Proportion of Identical Predictions. Measures the fraction of instances where the classifier's prediction on the original representation matches its prediction on the counterfactual representation. Range [0, 1], higher is better.
- `ATV` — range: [0, 1]
  - Average Total Variation distance. Measures the average L1 distance (divided by 2) between the predicted probability distributions of the original and counterfactual representations. Range [0, 1], lower is better.
- `TPR-Gap` — range: other
  - True Positive Rate Gap. Defined as P[Ŷ=y|Z=z, Y=y] - P[Ŷ=y|Z=¬z, Y=y]. Measures the disparity in true positive rates across sensitive attribute values for a given class. Lower values indicate better fairness.

## Input / output format

**Input**: Text document s, sensitive attribute value z (e.g., gender/race), ground-truth label y, and classifier predictions/probabilities on original embedding X(s) and intervened counterfactual embedding X(s)_{Z←z}.

**Output**: Classifier predictions Ŷ, probability distributions p_Ŷ, and derived metric values (PIP, ATV, ATE, TPR-Gap, Π̂).

## Scoring recipe

```python
def compute_PIP(pred_orig, pred_cf):
    return np.mean(pred_orig == pred_cf)

def compute_ATV(probs_orig, probs_cf):
    return np.mean(0.5 * np.sum(np.abs(probs_orig - probs_cf), axis=1))

def compute_TPR_Gap(y_true, y_pred, z_attr, class_label):
    mask_z = (z_attr == 'z')
    mask_not_z = (z_attr == 'not_z')
    tp_z = np.mean((y_true[mask_z] == class_label) & (y_pred[mask_z] == class_label))
    tp_not_z = np.mean((y_true[mask_not_z] == class_label) & (y_pred[mask_not_z] == class_label))
    return tp_z - tp_not_z
```

## Common pitfalls

- Swapping only the gender attribute in BiasInBios ignores other correlated factors, making ground-truth counterfactuals imperfect.
- CFRs may poorly substitute true counterfactuals for a small fraction of observations, skewing ATE estimates if not filtered by TV distance.
- Stochastic CFRs accounting for variance in the parallel component do not significantly improve evaluation results over deterministic versions.

## Evidence (verbatim from paper)

> One possible metric for this evaluation is the proportion of observations for which predictions coincide. For a finer analysis, we can also evaluate the average distance in total variation between the probability distributions predicted by the classifiers Y_hat and Z_hat. Let S:={(s_i,z_i)} be a set of couples of text documents s_i and of CF values z_i≠Z(s_i). Define the proportion of identical predictions (PIP) by PIP_Y_hat[S] := (1/|S|) Σ 1[Y_hat(X(s_{Z←z})) = Y_hat(X(s)_{Z←z})]. The range of the PIP metric is [0,1], closer to 1 being better.

## Citation

```bibtex
@misc{lemberger2024counterfactual,
  title={Explaining Text Classifiers with Counterfactual Representations},
  author={Lemberger et al. (2024)},
  year={2024},
  note={arXiv:2402.00711}
}
```

- arXiv: 2402.00711