Ffb Eval

---
name: ffb-eval
description: Evaluates in-processing group fairness methods by measuring the trade-off between model utility (acc) and various fairness metrics across multiple datasets and hyperparameter settings. Use when the user wants to benchmark on Adult, or asks about evaluating this task. Reports acc.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.09468
  bibtex_key: han2023ffb
  confidence: medium
---

# ffb-eval

> FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods — Han et al. (2023) (arXiv:2306.09468, 2023)

## What this evaluates

Evaluates in-processing group fairness methods by measuring the trade-off between model utility (acc) and various fairness metrics across multiple datasets and hyperparameter settings.

## Datasets

- **Adult** — total ?; splits: train (-1), test (-1); repo https://github.com/ahxt/fair_fairness_benchmark

## Metrics

- `acc` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted labels out of total instances. Computed as mean(predictions == labels).
- `fairness metric` — range: [0, 1]
  - Measures statistical parity or equalized odds between groups defined by a sensitive attribute. Specific formula varies by method.

## Input / output format

**Input**: Tabular feature vectors, ground-truth labels, and a designated sensitive attribute column.

**Output**: Predicted labels or probabilities, along with computed utility and fairness scores.

## Scoring recipe

```python
def compute_metrics(predictions, labels, sensitive_attr):
    acc = (predictions == labels).mean()
    # Fairness metric depends on the specific group fairness definition used
    fairness = compute_fairness_score(predictions, sensitive_attr)
    return acc, fairness
```

## Common pitfalls

- Utility-fairness trade-offs are highly sensitive to the choice of fairness control hyperparameters.
- Fairness performance on one dataset (e.g., Adult) does not guarantee similar results on other datasets due to varying inherent biases.

## Evidence (verbatim from paper)

> We plot the utility-fairness trade-offs for the Adult dataset with gender as the sensitive attribute and present the results in Figures 7 and 8. Figure 7: The Utility-Fairness Trade-offs with acc as utility metric.

## Citation

```bibtex
@misc{han2023ffb,
  title={FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods},
  author={Han et al. (2023)},
  year={2023},
  note={arXiv:2306.09468}
}
```

- arXiv: 2306.09468