Fairness Repair Eval

---
name: fairness-repair-eval
description: Evaluates the ability of an AutoML-based fairness repair framework to mitigate bias in machine learning models while preserving predictive accuracy. It measures the trade-off between accuracy retention and bias reduction across multiple binary classification datasets and model architectures. Use when the user wants to benchmark on Adult Census (race), Bank Marketing (age), German Credit (sex), Titanic (sex), or asks about evaluating this task. Reports Accuracy difference.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.09297
  bibtex_key: [REDACTED]
  confidence: high
---

# fairness-repair-eval

> Fix Fairness, Don't Ruin Accuracy: Performance Aware Fairness Repair using AutoML — Giang Nguyen et al. (2023) (arXiv:2306.09297, 2023)

## What this evaluates

Evaluates the ability of an AutoML-based fairness repair framework to mitigate bias in machine learning models while preserving predictive accuracy. It measures the trade-off between accuracy retention and bias reduction across multiple binary classification datasets and model architectures.

## Datasets

- **Adult Census (race)** — total 32561; splits: test (-1)
- **Bank Marketing (age)** — total 41188; splits: test (-1)
- **German Credit (sex)** — total 1000; splits: test (-1)
- **Titanic (sex)** — total 891; splits: test (-1)

## Metrics

- `Accuracy difference` **(primary)** — range: other
  - new accuracy - old accuracy. Positive values indicate improved predictive performance after repair.
- `DI difference` — range: other
  - old DI - new DI. Measures reduction in Disparate Impact bias.
- `SPD difference` — range: other
  - old SPD - new SPD. Measures reduction in Statistical Parity Difference bias.
- `EOD difference` — range: other
  - old EOD - new EOD. Measures reduction in Equalized Odds Difference bias.
- `AOD difference` — range: other
  - old AOD - new AOD. Measures reduction in Average Odds Difference bias.

## Input / output format

**Input**: Binary classification datasets containing feature columns, target labels, and a designated sensitive attribute column (race, age, or sex). Models are trained on these datasets to produce baseline predictions before repair.

**Output**: Repaired model predictions or optimized hyperparameter configurations generated by the Fair-AutoML framework, evaluated against the original baseline predictions.

## Scoring recipe

```python
def compute_evaluation(original_preds, repaired_preds, labels, sensitive_attr):
    acc_diff = accuracy_score(labels, repaired_preds) - accuracy_score(labels, original_preds)
    bias_diff = compute_fairness_metric(original_preds, sensitive_attr) - compute_fairness_metric(repaired_preds, sensitive_attr)
    return acc_diff, bias_diff
# Positive acc_diff and bias_diff indicate improvement in accuracy and fairness respectively.
```

## Common pitfalls

- Misinterpreting the sign of the difference metric: in this protocol, positive values indicate improvement for both accuracy and fairness, which is the opposite of standard loss-based metrics.
- Failing to use the correct sensitive attribute per dataset (e.g., using 'sex' for Adult Census instead of 'race'), leading to incorrect fairness calculations.
- Not distinguishing between the four fairness variants (DI, SPD, EOD, AOD), as they capture different bias dimensions and may yield conflicting results.

## Evidence (verbatim from paper)

> For accuracy, accuracy difference = new accuracy - old accuracy. For bias (DI, SPD, EOD, AOD), bias difference = old bias - new bias. Thus, a positive value indicates an improvement in bias/accuracy in the repaired model compared to the original and vice versa.

## Citation

```bibtex
@misc{nguyen2023fixfairness,
  title={Fix Fairness, Don't Ruin Accuracy: Performance Aware Fairness Repair using AutoML},
  author={Giang Nguyen et al. (2023)},
  year={2023},
  note={arXiv:2306.09297}
}
```

- arXiv: 2306.09297