Relaxed Equal Odds Fraud Eval

---
name: relaxed-equal-odds-fraud-eval
description: Evaluates a model-agnostic fairness calibration heuristic that adjusts prediction thresholds per protected attribute value to independently control false positive and false negative rates. It probes the ability to balance business-critical error costs across high-arity and multiple sensitive groups while maintaining predictive performance. Use when the user wants to benchmark on COMPAS (Criminal Recidivism), Income-Prediction, Health Prediction, Proprietary Online Fraud Dataset, or asks about evaluating this task. Reports FPR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2207.14355
  bibtex_key: [REDACTED]
  confidence: high
---

# relaxed-equal-odds-fraud-eval

> Multiple Attribute Fairness: Application to Fraud Detection — Meghanath M Y et al. (2022) (arXiv:2207.14355, 2022)

## What this evaluates

Evaluates a model-agnostic fairness calibration heuristic that adjusts prediction thresholds per protected attribute value to independently control false positive and false negative rates. It probes the ability to balance business-critical error costs across high-arity and multiple sensitive groups while maintaining predictive performance.

## Datasets

- **COMPAS (Criminal Recidivism)** — total ?; splits: test (-1); repo https://github.com/propublica/compas-analysis
- **Income-Prediction** — total ?; splits: test (-1)
- **Health Prediction** — total ?; splits: test (-1); repo https://github.com/gpleiss/equalized_odds_and_calibration/
- **Proprietary Online Fraud Dataset** — total ?; splits: test (-1)

## Metrics

- `FPR` **(primary)** — range: [0, 1]
  - False Positive Rate = FP / (FP + TN). Computed independently per protected attribute group to measure the rate of non-fraudulent transactions incorrectly flagged as fraudulent (or vice versa depending on class labeling).
- `FNR` — range: [0, 1]
  - False Negative Rate = FN / (FN + TP). Computed independently per protected attribute group to measure the rate of fraudulent transactions incorrectly flagged as non-fraudulent.

## Input / output format

**Input**: Model prediction scores/probabilities and ground truth labels, along with protected attribute values (e.g., country, currency, gender, race) for post-hoc threshold adjustment.

**Output**: Adjusted binary predictions or custom thresholds per protected attribute value that satisfy relaxed equalized odds constraints.

## Scoring recipe

```python
def compute_fpr_fnr(preds, labels, groups):
    results = {}
    for g in groups.unique():
        mask = groups == g
        fp = ((preds[mask] == 1) & (labels[mask] == 0)).sum()
        tn = ((preds[mask] == 0) & (labels[mask] == 0)).sum()
        fn = ((preds[mask] == 0) & (labels[mask] == 1)).sum()
        tp = ((preds[mask] == 1) & (labels[mask] == 1)).sum()
        results[g] = {
            'FPR': fp / (fp + tn) if (fp + tn) > 0 else 0.0,
            'FNR': fn / (fn + tp) if (fn + tp) > 0 else 0.0
        }
    return results
```

## Common pitfalls

- Confusing the proposed post-hoc threshold adjustment with pre-processing or in-processing fairness methods.
- Assuming standard equalized odds requires equal FPR and FNR simultaneously; the paper explicitly decouples them into 'relaxed equalized odds' to allow independent control.
- Reporting aggregate FPR/FNR instead of per-group values, which masks the fairness violations the method aims to fix.

## Evidence (verbatim from paper)

> From Figures [3](#S4.F3 "Figure 3 ‣ 4. Experiments ‣ Multiple Attribute Fairness: Application to Fraud Detection"), [4](#S4.F4 "Figure 4 ‣ 4. Experiments ‣ Multiple Attribute Fairness: Application to Fraud Detection") (green color bars), we observe that the final $F_{select}$ with custom thresholds per country considerably reduces the bias towards certain countries. Further, we also note that the FPRs and FNRs across countries conform to the relaxed equalized odds fairness measure – mean across the countries lies withing two standard deviations.

## Citation

```bibtex
@misc{meghanath2022multiple,
  title={Multiple Attribute Fairness: Application to Fraud Detection},
  author={Meghanath M Y et al. (2022)},
  year={2022},
  note={arXiv:2207.14355}
}
```

- arXiv: 2207.14355