Fairness Comparison Eval

---
name: fairness-comparison-eval
description: Evaluates the comparative performance of fairness-enhancing machine learning interventions across multiple datasets. It probes how different algorithmic strategies trade off predictive accuracy against a comprehensive set of fairness metrics under standardized preprocessing and fixed train-test splits. Use when the user wants to benchmark on Standardized benchmark datasets (unspecified in excerpt), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1802.04422
  bibtex_key: friedler2018fairnesscomparison
  confidence: medium
---

# fairness-comparison-eval

> A comparative study of fairness-enhancing interventions in machine learning — Friedler et al. (2018) (arXiv:1802.04422, 2018)

## What this evaluates

Evaluates the comparative performance of fairness-enhancing machine learning interventions across multiple datasets. It probes how different algorithmic strategies trade off predictive accuracy against a comprehensive set of fairness metrics under standardized preprocessing and fixed train-test splits.

## Datasets

- **Standardized benchmark datasets (unspecified in excerpt)** — total ?; splits: train (-1), test (-1); repo https://github.com/algofairness/fairness-comparison

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correct predictions on the test set.
- `fairness measures` — range: [0, 1]
  - Aggregate evaluation across multiple fairness metrics (e.g., demographic parity, equal opportunity) computed on test predictions.

## Input / output format

**Input**: Preprocessed dataset instances with features, protected attributes, and ground-truth labels, partitioned into fixed training and test splits.

**Output**: Predicted labels for test instances, followed by computed accuracy and fairness metric scores for each evaluated algorithm.

## Scoring recipe

```python
def compute_metrics(predictions, labels, protected_attr):
    accuracy = (predictions == labels).mean()
    # Compute fairness metrics using standard definitions on the test split
    fairness_scores = compute_fairness_metrics(predictions, labels, protected_attr)
    return {'accuracy': accuracy, 'fairness': fairness_scores}
```

## Common pitfalls

- Combining dataset-specific preprocessing directly into algorithm code, which prevents fair cross-algorithm comparison.
- Analyzing algorithms under only one or two fairness measures instead of a comprehensive suite.
- Failing to account for sensitivity to training-test split variability and preprocessing dependencies.

## Evidence (verbatim from paper)

> ensure that each algorithm is compared using the same dataset (including the same preprocessing), the same set of training / test splits, and all desired fairness and accuracy measures.

## Citation

```bibtex
@misc{friedler2018fairnesscomparison,
  title={A comparative study of fairness-enhancing interventions in machine learning},
  author={Friedler et al. (2018)},
  year={2018},
  note={arXiv:1802.04422}
}
```

- arXiv: 1802.04422