Fairness Sequential Eval

---
name: fairness-sequential-eval
description: Evaluates sequential decision policies under simulated historical and measurement bias to measure how accounting for unrealized outcomes affects fairness disparities and cumulative utility. It probes whether uncertainty-aware exploration mitigates selection rate differences and false positive rate parity violations without sacrificing profit. Use when the user wants to benchmark on Synthetic Sequential Simulation, or asks about evaluating this task. Reports selection rate difference.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.21711
  bibtex_key: lee2026fairness
  confidence: medium
---

# fairness-sequential-eval

> Fairness under uncertainty in sequential decisions — Lee et al. (2026) (arXiv:2604.21711, 2026)

## What this evaluates

Evaluates sequential decision policies under simulated historical and measurement bias to measure how accounting for unrealized outcomes affects fairness disparities and cumulative utility. It probes whether uncertainty-aware exploration mitigates selection rate differences and false positive rate parity violations without sacrificing profit.

## Datasets

- **Synthetic Sequential Simulation** — total ?; splits: simulation (-1)

## Metrics

- `selection rate difference` **(primary)** — range: percent
  - Difference in the proportion of candidates from the minority group (A=0) versus the majority group who are given a loan.
- `FPR parity` — range: percent
  - Difference in false positive rates between sensitive groups, computed over observed outcomes.
- `cumulative profit` — range: other
  - Total utility/profit accumulated over the sequential time steps (Q1-Q10).

## Input / output format

**Input**: Sequential decision instances over 10 quarters, including binary sensitive attribute A, observed outcomes Y (subject to measurement/historical bias), and policy actions.

**Output**: Policy decisions/actions per quarter, with final evaluation computed as aggregate fairness metrics and cumulative profit over the horizon.

## Scoring recipe

```python
def score(actions, sensitive_attrs, outcomes, horizon=10):
    sel_minority = sum(actions[sensitive_attrs == 0]) / len(sensitive_attrs[sensitive_attrs == 0])
    sel_majority = sum(actions[sensitive_attrs == 1]) / len(sensitive_attrs[sensitive_attrs == 1])
    selection_rate_diff = abs(sel_minority - sel_majority)
    
    fpr_minority = fpr(actions[sensitive_attrs == 0], outcomes[sensitive_attrs == 0])
    fpr_majority = fpr(actions[sensitive_attrs == 1], outcomes[sensitive_attrs == 1])
    fpr_parity = abs(fpr_minority - fpr_majority)
    
    cumulative_profit = sum(outcome_values for t in range(horizon))
    return {'selection_rate_diff': selection_rate_diff, 'fpr_parity': fpr_parity, 'cumulative_profit': cumulative_profit}
```

## Common pitfalls

- Simulations are synthetic and explicitly designed to illustrate mechanisms rather than benchmark deployable fairness algorithms.
- FPR parity results are noted as noisy due to the small scale of the simulation.
- Profit scale is narrow (spanning only 6 to 7), making initial disparities appear larger than they are.

## Evidence (verbatim from paper)

> Selection rate refers to the proportion of candidates from a specific group who are given a loan, with selection rate difference then being the difference in rate selection between the two sensitive groups. Other fairness metrics are more standard, and defined in [appendix B] Evaluation metrics ‣ Fairness under uncertainty in sequential decisions. Fig. 3 compares only the final values of the metrics at the end of our given time period (Q1-Q10).

## Citation

```bibtex
@misc{lee2026fairness,
  title={Fairness under uncertainty in sequential decisions},
  author={Lee et al. (2026)},
  year={2026},
  note={arXiv:2604.21711}
}
```

- arXiv: 2604.21711