Fairness Recourse Subgroup Eval

---
name: fairness-recourse-subgroup-eval
description: Evaluates the fairness of algorithmic recourse across demographic subgroups by ranking them according to various counterfactual-based fairness definitions. It probes whether different recourse fairness metrics capture distinct aspects of bias, actionability constraints, and subgroup granularity in real-world decision systems. Use when the user wants to benchmark on Adult, or asks about evaluating this task. Reports unfairness_score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.14978
  bibtex_key: kavouras2023fairness
  confidence: high
---

# fairness-recourse-subgroup-eval

> Fairness Aware Counterfactuals for Subgroups — Kavouras et al. (2023) (arXiv:2306.14978, 2023)

## What this evaluates

Evaluates the fairness of algorithmic recourse across demographic subgroups by ranking them according to various counterfactual-based fairness definitions. It probes whether different recourse fairness metrics capture distinct aspects of bias, actionability constraints, and subgroup granularity in real-world decision systems.

## Datasets

- **Adult** — total ?; splits: train (-1), test (-1)

## Metrics

- `unfairness_score` **(primary)** — range: other
  - A scalar value quantifying the degree of recourse unfairness for a subgroup under a specific fairness definition (e.g., Equal Cost of Effectiveness, Equal Choice for Recourse, Equal Effectiveness within Budget). Computed by comparing the effectiveness, cost, or number of sufficiently effective counterfactual actions across protected subgroups. Higher scores indicate greater unfairness.

## Input / output format

**Input**: Feature vector for an individual instance from the Adult dataset (e.g., age, marital-status, race, relationship, workclass, hours-per-week, occupation), the trained logistic regression model, and a specified fairness definition with its threshold parameters (e.g., phi or c).

**Output**: A ranked list of subgroups with their corresponding unfairness_score, bias direction (e.g., Male/Female), and coverage percentage. For each subgroup, a set of counterfactual actions (e.g., 'Make marital-status=Married-civ-spouse') with associated effectiveness percentages is generated.

## Scoring recipe

```python
def compute_unfairness_score(predictions, gold_def, phi, c):
    # predictions: dict[subgroup] -> list of (action, effectiveness, cost)
    if gold_def == 'Equal Cost of Effectiveness':
        min_eff = min(max([a.eff for a in cfs] or [0]) for cfs in predictions.values())
        return float('inf') if min_eff < phi else 0.0
    elif gold_def == 'Equal Choice for Recourse':
        choices = [sum(1 for a in cfs if a.eff >= phi) for cfs in predictions.values()]
        return max(choices) - min(choices)
    elif gold_def == 'Equal Effectiveness within Budget':
        effs = [max([a.eff for a in cfs if a.cost <= c] or [0]) for cfs in predictions.values()]
        return max(effs) - min(effs)
    return 0.0
```

## Common pitfalls

- Treating different fairness definitions (e.g., Equal Cost of Effectiveness vs. Equal Choice for Recourse) as interchangeable, when they measure fundamentally different aspects of recourse difficulty and actionability.
- Ignoring the 1% support threshold used in fp-growth for subgroup generation, which directly impacts subgroup granularity and can mask or reveal gerrymandering effects.
- Assuming the unfairness_score is a single universal value; it is strictly definition-specific and threshold-dependent (phi, c).
- Generating subgroups on the training set instead of the test set, which violates the described experimental protocol and risks data leakage.

## Evidence (verbatim from paper)

> Table 1 presents three subgroups which were ranked at position 1 according to three different definitions: Equal Cost of Effectiveness ($\phi$ = 0.7), Equal Choice for Recourse ($\phi$ = 0.7) and Equal Cost of Effectiveness ($\phi$ = 0.3), meaning that these subgroups were detected to have the highest unfairness according to the respective definitions. For each subgroup, its rank, bias against, and unfairness score are provided for all definitions presented in the left-most column.

## Citation

```bibtex
@misc{kavouras2023fairness,
  title={Fairness Aware Counterfactuals for Subgroups},
  author={Kavouras et al. (2023)},
  year={2023},
  note={arXiv:2306.14978}
}
```

- arXiv: 2306.14978