Bias Metrics Eval

---
name: bias-metrics-eval
description: Evaluates whether text classification models exhibit unintended demographic bias by analyzing how model confidence scores are distributed across different identity groups. It probes the model's ability to rank toxic vs. non-toxic content fairly and detect systematic score shifts that threshold-dependent metrics might miss. Use when the user wants to benchmark on Synthetic Bias Test Set, Human-Labeled Online Comments, or asks about evaluating this task. Reports Subgroup AUC, BPSN AUC, BNSP AUC, AEG.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1903.04561
  bibtex_key: borkan2019nuanced
  confidence: high
---

# bias-metrics-eval

> Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification — Borkan et al. (2019) (arXiv:1903.04561, 2019)

## What this evaluates

Evaluates whether text classification models exhibit unintended demographic bias by analyzing how model confidence scores are distributed across different identity groups. It probes the model's ability to rank toxic vs. non-toxic content fairly and detect systematic score shifts that threshold-dependent metrics might miss.

## Datasets

- **Synthetic Bias Test Set** — total 77000; splits: test (77000)
- **Human-Labeled Online Comments** — total 1800000; splits: test (1800000); repo https://git.io/fhpcC

## Metrics

- `Subgroup AUC, BPSN AUC, BNSP AUC, AEG` **(primary)** — range: [0, 1] for AUCs, [-1, 1] for AEGs
  - Threshold-agnostic metrics analyzing score distributions across demographic groups. Subgroup AUC measures ranking performance within a group. BPSN/BNSP AUCs measure bias in ordering positive/negative scores. AEG measures the average equality gap (score distribution shift) between a subgroup and the background.

## Input / output format

**Input**: Online forum comments (text strings), optionally filtered by length (e.g., <100 characters).

**Output**: Continuous toxicity score (model confidence) for each comment.

## Scoring recipe

```python
def compute_bias_metrics(predictions, gold_labels, identities):
    results = {}
    for group in set(identities):
        mask = [i for i, id in enumerate(identities) if id == group]
        y_true = [gold_labels[i] for i in mask]
        y_score = [predictions[i] for i in mask]
        results[f'{group}_subgroup_auc'] = auc(y_true, y_score)
        # AEG computed as mean(score_subgroup) - mean(score_background) for non-toxic items
    return results
```

## Common pitfalls

- Relying on threshold-dependent metrics (e.g., accuracy or F1 at a fixed cutoff) masks subgroup score shifts and ranking biases.
- High Subgroup AUC does not guarantee fairness; models can have perfect ranking within groups but still exhibit systematic score inflation/deflation (detected by AEG).
- Synthetic datasets lack real-world noise and class imbalance, potentially overestimating model fairness compared to human-labeled real data.

## Evidence (verbatim from paper)

> Subgroup AUC and BNSP AUC show relatively high values across all groups, in both TOXICITY@1 and TOXICITY@6. This emphasizes that the model is generally effective at distinguishing toxic from non-toxic examples within every group (subgroup AUC), even for the groups that show an incorrect tendency towards toxicity in the BPSN AUC discussed above.

## Citation

```bibtex
@misc{borkan2019nuanced,
  title={Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification},
  author={Borkan et al. (2019)},
  year={2019},
  note={arXiv:1903.04561}
}
```

- arXiv: 1903.04561