Folktables Eval

---
name: folktables-eval
description: Evaluates how fairness interventions affect predictive accuracy and fairness violations across different geographic regions and time periods. It probes the stability of fairness metrics under distribution shift and the efficacy of pre-processing, in-processing, and post-processing interventions on tabular demographic data. Use when the user wants to benchmark on Folktables (ACS PUMS), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2108.04884
  bibtex_key: ding2021retiringadult
  confidence: high
---

# folktables-eval

> Retiring Adult: New Datasets for Fair Machine Learning — Ding et al. (2021) (arXiv:2108.04884, 2021)

## What this evaluates

Evaluates how fairness interventions affect predictive accuracy and fairness violations across different geographic regions and time periods. It probes the stability of fairness metrics under distribution shift and the efficacy of pre-processing, in-processing, and post-processing interventions on tabular demographic data.

## Datasets

- **Folktables (ACS PUMS)** — total ?; splits: train (-1), test (-1); repo https://github.com/zykls/folktables

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy: the proportion of correct predictions out of total predictions.
- `equality of opportunity violation` — range: [0, 1]
  - The absolute difference between the maximum and minimum true positive rates across protected groups. Measures disparity in correct positive predictions.
- `demographic parity violation` — range: [0, 1]
  - The absolute difference between the maximum and minimum positive prediction rates across protected groups. Measures disparity in overall acceptance rates.

## Input / output format

**Input**: Tabular demographic and socioeconomic features from US Census ACS PUMS data, including a protected attribute (race) and a binary target variable (e.g., income > $50k, public coverage, employment status).

**Output**: Binary prediction (0 or 1) or probability scores for the target variable.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred, protected_attr):
    accuracy = np.mean(y_true == y_pred)
    tpr_groups = {}
    pos_rates = {}
    for group in np.unique(protected_attr):
        mask = protected_attr == group
        tpr_groups[group] = np.mean((y_pred[mask] == 1) & (y_true[mask] == 1)) / max(np.sum(y_true[mask] == 1), 1)
        pos_rates[group] = np.mean(y_pred[mask] == 1)
    eo_violation = max(tpr_groups.values()) - min(tpr_groups.values())
    dp_violation = max(pos_rates.values()) - min(pos_rates.values())
    return accuracy, eo_violation, dp_violation
```

## Common pitfalls

- Assuming fairness metrics are stable across different US states or over time, when the paper shows they vary significantly.
- Using the original UCI Adult dataset's fixed $50k income threshold without accounting for geographic and temporal variation in living costs and wage distributions.
- Confusing in-distribution evaluation (same state/year) with out-of-distribution evaluation (other states/years) when assessing model transferability.

## Evidence (verbatim from paper)

> Throughout we keep the experimental details (models, hyperparameters, etc) identical to those detailed in Appendix C. ... we compare the accuracy and fairness criterion violation of this unconstrained model with the same model after applying one of three common fairness intervention... Figure 6 shows the result of this experiment for the ACSIncome prediction task for interventions to achieve equality of opportunity.

## Citation

```bibtex
@misc{ding2021retiringadult,
  title={Retiring Adult: New Datasets for Fair Machine Learning},
  author={Ding et al. (2021)},
  year={2021},
  note={arXiv:2108.04884}
}
```

- arXiv: 2108.04884