Counterfactual Situation Testing Eval

---
name: counterfactual-situation-testing-eval
description: Evaluates a fairness auditing framework's ability to detect individual discrimination in decision-making systems by comparing factual outcomes against counterfactual or similar-group outcomes. It probes whether protected attributes causally influence decisions beyond legitimate factors. Use when the user wants to benchmark on Synthetic Loan Application, Law School Admissions, or asks about evaluating this task. Reports individual discrimination cases.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2302.11944
  bibtex_key: alvarez2023counterfactual
  confidence: high
---

# counterfactual-situation-testing-eval

> Counterfactual Situation Testing: Uncovering Discrimination under Fairness given the Difference — Alvarez et al. (2023) (arXiv:2302.11944, 2023)

## What this evaluates

Evaluates a fairness auditing framework's ability to detect individual discrimination in decision-making systems by comparing factual outcomes against counterfactual or similar-group outcomes. It probes whether protected attributes causally influence decisions beyond legitimate factors.

## Datasets

- **Synthetic Loan Application** — total 5000; splits: test (5000); repo https://github.com/cc-jalvarez/counterfactual-situation-testing
- **Law School Admissions** — total 21790; splits: test (21790); repo https://github.com/cc-jalvarez/counterfactual-situation-testing

## Metrics

- `individual discrimination cases` **(primary)** — range: percent
  - Percentage of complainants where the difference in prediction probability between the complainant and the test group ($\Delta p$) exceeds a minimum deviation threshold ($\tau=0.0$) and the threshold falls outside the 95% confidence interval of $\Delta p$.

## Input / output format

**Input**: Factual feature vector $(X, A)$ for each applicant, along with the Structural Causal Model (SCM) equations and decision function $b()$ used to generate predictions.

**Output**: Binary discrimination label per instance (discriminated/not), aggregated as a count and percentage of total applicants.

## Scoring recipe

```python
discrimination_count = 0
for c in dataset:
    p_c = predict(c)
    p_t = predict(test_group_neighbors(c))
    delta_p = p_c - p_t
    ci = compute_confidence_interval(delta_p, alpha=0.05)
    if delta_p > 0.0 and 0.0 not in ci:
        discrimination_count += 1
return discrimination_count / len(dataset) * 100
```

## Common pitfalls

- Confusing CST with standard Counterfactual Fairness (CF), which only compares an individual's factual vs. counterfactual outcome without group similarity constraints.
- Ignoring the confidence interval filter: cases with $\Delta p > 0$ but where the 95% CI includes 0 are statistically insignificant and should be excluded.
- Treating the synthetic and law school datasets as real-world benchmarks; they are procedurally generated using specific structural equations for illustration.

## Evidence (verbatim from paper)

> We define individual discrimination as $\Delta p>\tau$ (Def.[3.4]) for a single protected attribute. We still, though, demonstrate the use of confidence intervals (Def.[3.5]) and how it would affect the final results. Under $\alpha\=5\%$, we would reject these cases as individual discrimination claims with confidence level of 95% since the minimum deviation is covered by the CIs. Table 1 reports the Number (and %) of detected individual discrimination cases for the illustrative example based on gender.

## Citation

```bibtex
@misc{alvarez2023counterfactual,
  title={Counterfactual Situation Testing: Uncovering Discrimination under Fairness given the Difference},
  author={Alvarez et al. (2023)},
  year={2023},
  note={arXiv:2302.11944}
}
```

- arXiv: 2302.11944