Fair Weak Supervision Eval

---
name: fair-weak-supervision-eval
description: Evaluates a weak supervision pipeline's ability to mitigate labeling function bias and improve fairness across demographic groups. It measures how well a source bias mitigation method recovers accurate pseudolabels while reducing disparities in prediction rates between privileged and underrepresented groups. Use when the user wants to benchmark on Adult, Bank Marketing, CivilComments, HateXplain, CelebA, UTKFace, WRENCH, or asks about evaluating this task. Reports demographic parity gap ($\Delta_{DP}$).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2303.17713
  bibtex_key: shin2023mitigating
  confidence: high
---

# fair-weak-supervision-eval

> Mitigating Source Bias for Fairer Weak Supervision — Changho Shin et al. (arXiv:2303.17713, 2023)

## What this evaluates

Evaluates a weak supervision pipeline's ability to mitigate labeling function bias and improve fairness across demographic groups. It measures how well a source bias mitigation method recovers accurate pseudolabels while reducing disparities in prediction rates between privileged and underrepresented groups.

## Datasets

- **Adult** — total ?; splits: train (-1), test (-1)
- **Bank Marketing** — total ?; splits: train (-1), test (-1)
- **CivilComments** — total ?; splits: train (-1), test (-1)
- **HateXplain** — total ?; splits: train (-1), test (-1)
- **CelebA** — total ?; splits: train (-1), test (-1)
- **UTKFace** — total ?; splits: train (-1), test (-1)
- **WRENCH** — total ?; splits: train (-1), test (-1)

## Metrics

- `accuracy` — range: [0, 1]
  - Fraction of correctly predicted labels out of total test instances.
- `F1 score` — range: [0, 1]
  - Harmonic mean of precision and recall for the positive class.
- `demographic parity gap ($\Delta_{DP}$)` **(primary)** — range: [0, 1]
  - Absolute difference in positive prediction rates between the privileged and underrepresented groups: $|P(\hat{Y}=1|G=0) - P(\hat{Y}=1|G=1)|$.
- `equal opportunity gap ($\Delta_{EO}$)` — range: [0, 1]
  - Absolute difference in true positive rates between groups: $|P(\hat{Y}=1|Y=1, G=0) - P(\hat{Y}=1|Y=1, G=1)|$.

## Input / output format

**Input**: Training data with true labels (for LF evaluation) or unlabeled data, along with labeling functions (heuristics or pretrained models) that generate weak labels. Group membership labels are provided for fairness evaluation.

**Output**: Predicted labels (pseudolabels) from the label model, followed by final predictions from the end model (logistic regression) on the test set.

## Scoring recipe

```python
predictions = end_model.predict(test_data)
true_labels = test_labels
groups = test_groups
acc = mean(predictions == true_labels)
f1 = f1_score(true_labels, predictions)
dp_gap = abs(mean(predictions[groups==0]) - mean(predictions[groups==1]))
eo_gap = abs(mean(predictions[true_labels==1 & groups==0]) - mean(predictions[true_labels==1 & groups==1]))
return acc, f1, dp_gap, eo_gap
```

## Common pitfalls

- Fully supervised (FS) results are provided only as an upper bound, not as a baseline for comparison.
- One-hot encoded features can distort nearest-neighbor distance calculations, causing method failure unless mitigated (e.g., via LIFT embedding).
- Highly imbalanced class distributions (e.g., P(Y=1) ≈ 0.1) can cause F1 score drops when fairness metrics are optimized.

## Evidence (verbatim from paper)

> To see if our method can improve both fairness and performance, we measured the demographic parity gap  $(\Delta_{DP})$  and the equal opportunity gap  $(\Delta_{EO})$  as fairness metrics, and computed accuracy and F1 score as performance metrics as well.

## Citation

```bibtex
@misc{shin2023mitigating,
  title={Mitigating Source Bias for Fairer Weak Supervision},
  author={Changho Shin et al.},
  year={2023},
  note={arXiv:2303.17713}
}
```

- arXiv: 2303.17713