Fair Credit Scoring Eval

---
name: fair-credit-scoring-eval
description: This benchmark evaluates the trade-off between algorithmic fairness and financial profitability in credit scoring models. It probes how well various fairness-aware preprocessing, in-processing, and post-processing techniques maintain predictive accuracy and demographic parity while minimizing economic loss for lenders. Use when the user wants to benchmark on german, bene, taiwan, uk, pakdd, gmsc, homecredit, or asks about evaluating this task. Reports Profitability (Profit per EUR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2103.01907
  bibtex_key: kozodoi2021fairness
  confidence: high
---

# fair-credit-scoring-eval

> Fairness in Credit Scoring: Assessment, Implementation and Profit Implications — Kozodoi et al. (2021) (arXiv:2103.01907, 2021)

## What this evaluates

This benchmark evaluates the trade-off between algorithmic fairness and financial profitability in credit scoring models. It probes how well various fairness-aware preprocessing, in-processing, and post-processing techniques maintain predictive accuracy and demographic parity while minimizing economic loss for lenders.

## Datasets

- **german** — total 1000; splits: train (600), test (400)
- **bene** — total 3123; splits: train (1873), test (1250)
- **taiwan** — total 23531; splits: train (14118), test (9413)
- **uk** — total 30000; splits: train (18000), test (12000)
- **pakdd** — total 50000; splits: train (30000), test (20000)
- **gmsc** — total 150000; splits: train (90000), test (60000)
- **homecredit** — total 307511; splits: train (184506), test (123005)

## Metrics

- `Profitability (Profit per EUR)` **(primary)** — range: percent
  - Normalized expected profit per EUR issued, computed via the EMP criterion. It integrates over the cost of default (B) and opportunity cost/benefit (C=ROI=0.2664) using predicted cumulative density functions for good and bad risks at a cutoff τ. Base scenario is rejecting all applications.
- `AUC` — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the model's discriminatory ability across all classification thresholds.
- `Fairness (Independence, Separation, Sufficiency)` — range: [0, 1]
  - Independence measures demographic parity (prediction independent of sensitive attribute). Separation measures equalized odds (true/false positive rates equal across groups). Sufficiency measures calibration within groups (positive predictive value equal across groups).

## Input / output format

**Input**: Tabular loan applicant features and loan characteristics. Target is binary (1=repaid, 0=default). Sensitive attribute is age group split at 25 years (<25 vs ≥25).

**Output**: Continuous risk score or binary approval decision (approve/reject) derived via a threshold τ.

## Scoring recipe

```python
def compute_metrics(y_true, y_scores, sensitive, priors, B_dist, C=0.2664):
    # Profit: Apply Eq 16 using predicted CDFs F0(tau), F1(tau)
    # Integrate over B distribution: [C*(pi1*(1-F1) - pi1*F1) - B*pi0*(1-F0)] * f(B) dB
    profit = compute_emp_profit(y_scores, y_true, priors, B_dist, C)
    # AUC: Standard ROC-AUC
    auc = roc_auc_score(y_true, y_scores)
    # Fairness: Compute Independence, Separation, Sufficiency across sensitive groups
    fairness = compute_fairness_metrics(y_true, y_scores, sensitive)
    return {'profit': profit, 'auc': auc, 'fairness': fairness}
```

## Common pitfalls

- The profit baseline is normalized to 'rejecting all applications', not the standard 'accepting all' or random baseline, which drastically changes the magnitude and sign of profit differences.
- The sensitive attribute is age (<25 vs ≥25), not race or gender, and the threshold of 25 is empirically derived from disparate impact analysis, not a standard demographic split.
- Aggregation differs by processor type: pre/post-processors average over 140 runs (7 datasets × 5 folds × 4 classifiers), while in-processors average over only 35 runs (7 datasets × 5 folds).

## Evidence (verbatim from paper)

> Fairness processors and benchmarks are evaluated on the test set using multiple performance metrics. First, we measure the profitability of a scorecard by computing profit per EUR issued by a financial institution. ... Apart from estimating the profitability of each fairness processor, we also compute the area under the ROC curve (AUC), which is a widely used indicator of the discriminatory ability of a scoring model. In addition, we evaluate fairness by measuring independence, separation and sufficiency.

## Citation

```bibtex
@misc{kozodoi2021fairness,
  title={Fairness in Credit Scoring: Assessment, Implementation and Profit Implications},
  author={Kozodoi et al. (2021)},
  year={2021},
  note={arXiv:2103.01907}
}
```

- arXiv: 2103.01907