Fairness Label Noise Eval

---
name: fairness-label-noise-eval
description: This evaluation probes the ability of label noise correction methods to mitigate group-dependent label noise while preserving predictive performance and improving algorithmic fairness. It measures how well pre-processing techniques remove underlying discrimination from training data before classifier training. Use when the user wants to benchmark on OpenML (9 datasets), or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.15994
  bibtex_key: oliveira2023fairlnc
  confidence: high
---

# fairness-label-noise-eval

> Systematic analysis of the impact of label noise correction on ML Fairness — Inês Oliveira e Silva et al. (2023) (arXiv:2306.15994, 2023)

## What this evaluates

This evaluation probes the ability of label noise correction methods to mitigate group-dependent label noise while preserving predictive performance and improving algorithmic fairness. It measures how well pre-processing techniques remove underlying discrimination from training data before classifier training.

## Datasets

- **OpenML (9 datasets)** — total ?; splits: train (-1), test (-1); repo https://github.com/reluzita/fair-lnc-evaluation

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the ROC Curve. Measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
- `DP_diff` — range: [0, 1]
  - Demographic Parity difference: |P(y_hat=1|g=0) - P(y_hat=1|g=1)|. Measures the absolute difference in positive prediction rates between unprotected and protected groups.
- `EOD_diff` — range: [0, 1]
  - Equalized Odds difference: max(TPR_diff, FPR_diff). Takes the maximum absolute difference in True Positive Rates and False Positive Rates between groups.
- `PE_diff` — range: [0, 1]
  - Predictive Equality difference: |P(y_hat=1|y=0,g=0) - P(y_hat=1|y=0,g=1)|. Measures the absolute difference in False Positive Rates between groups.
- `EOP_diff` — range: [0, 1]
  - Equal Opportunity difference: |P(y_hat=0|y=1,g=0) - P(y_hat=0|y=1,g=1)|. Measures the absolute difference in False Negative Rates between groups.

## Input / output format

**Input**: Feature vectors and noisy binary labels for each instance, along with a binary protected group attribute (g=0 or g=1).

**Output**: Binary predicted labels (y_hat) for each instance.

## Scoring recipe

```python
def score(y_true, y_pred, group):
    auc = roc_auc_score(y_true, y_pred)
    dp = abs(mean(y_pred[group==0]) - mean(y_pred[group==1]))
    tpr0 = mean(y_pred[(y_true==1)&(group==0)])
    tpr1 = mean(y_pred[(y_true==1)&(group==1)])
    fpr0 = mean(y_pred[(y_true==0)&(group==0)])
    fpr1 = mean(y_pred[(y_true==0)&(group==1)])
    eod = max(abs(tpr0-tpr1), abs(fpr0-fpr1))
    pe = abs(fpr0-fpr1)
    eop = abs((1-tpr0)-(1-tpr1))
    return {'AUC': auc, 'DP_diff': dp, 'EOD_diff': eod, 'PE_diff': pe, 'EOP_diff': eop}
```

## Common pitfalls

- Computing fairness metrics on raw noisy labels instead of predictions from models trained on corrected data.
- Using accuracy as the primary performance metric despite significant class imbalance across datasets (e.g., 33% vs 58% positive rates).
- Treating group-dependent label noise as feature bias, which requires different correction strategies.

## Evidence (verbatim from paper)

> To evaluate the obtained models, we tested the predictive performance of the predictions by calculating the Area Under the ROC Curve (AUC) metric. In terms of fairness, the following metrics were analyzed: Demographic Parity (also known as statistical parity) is a statistical group fairness notion that is achieved when individuals from both protected and unprotected groups are equally likely to be predicted as positive by the model. We analyze the Demographic Parity difference between the two groups: $DP_{dif}=|P(\hat{y}=1|g=0)-P(\hat{y}=1|g=1)|$

## Citation

```bibtex
@misc{oliveira2023fairlnc,
  title={Systematic analysis of the impact of label noise correction on ML Fairness},
  author={Inês Oliveira e Silva et al. (2023)},
  year={2023},
  note={arXiv:2306.15994}
}
```

- arXiv: 2306.15994