Hmda Fairness Eval

---
name: hmda-fairness-eval
description: Evaluates the trade-off between predictive performance and group fairness when applying causal pre-processing to approximate an unbiased data distribution. It probes whether debiasing techniques can simultaneously satisfy multiple fairness constraints without degrading model accuracy. Use when the user wants to benchmark on HMDA (Wisconsin, 2022), or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.14710
  bibtex_key: [REDACTED]
  confidence: high
---

# hmda-fairness-eval

> Overcoming Fairness Trade-offs via Pre-processing: A Causal Perspective — Leininger et al. (2025) (arXiv:2501.14710, 2025)

## What this evaluates

Evaluates the trade-off between predictive performance and group fairness when applying causal pre-processing to approximate an unbiased data distribution. It probes whether debiasing techniques can simultaneously satisfy multiple fairness constraints without degrading model accuracy.

## Datasets

- **HMDA (Wisconsin, 2022)** — total 83808; splits: train (-1), test (-1)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve. Measures the model's ability to rank positive (loan approved) instances higher than negative ones across all classification thresholds.
- `Fairness metrics (Demographic Parity)` — range: percent
  - Difference in approval rates between the protected group (Black) and unprotected group (White). Evaluated as a percentage constraint, with the paper targeting 100% parity (0% difference).

## Input / output format

**Input**: Features: loan amount, loan purpose, debt ratio, age, gender. Protected attribute: race (Black vs. non-Hispanic White).

**Output**: Binary loan approval decision (1 = granted, 0 = not granted) or predicted probability.

## Scoring recipe

```python
def score(predictions, gold, protected_mask):
    auc = roc_auc_score(gold, predictions)
    approval_rate_protected = mean(predictions[protected_mask])
    approval_rate_unprotected = mean(predictions[~protected_mask])
    fairness_diff = abs(approval_rate_protected - approval_rate_unprotected)
    return {'AUC': auc, 'Fairness_Diff': fairness_diff}
```

## Common pitfalls

- Evaluating performance on real-world data instead of pre-processed data, which masks the true impact of bias mitigation and overestimates accuracy.
- Ignoring the severe class imbalance (protected group is only ~5.2% of the dataset) when interpreting fairness-performance trade-offs.
- Assuming higher AUC on real data indicates better model quality, whereas the paper argues it merely reflects entrenched historical bias.

## Evidence (verbatim from paper)

> For the real world HMDA data in Figure 4a, we again observe the common trade-off between fairness and performance, where an increase in fairness from 80% (fairness of the unconstraint model) to 100% is accompanied by a decrease in performance. Regarding performance, we observe that models trained and evaluated on real, adapted, and warped data all achieve similarly high AUCs.

## Citation

```bibtex
@misc{leininger2025overcoming,
  title={Overcoming Fairness Trade-offs via Pre-processing: A Causal Perspective},
  author={Leininger et al. (2025)},
  year={2025},
  note={arXiv:2501.14710}
}
```

- arXiv: 2501.14710