Safety Score

---
name: safety-score
description: Quantifies implicit representational harms in pre-trained language models by measuring the disparity in language modeling probabilities between harmful and benign sentences targeting 13 marginalized demographics. It probes whether a model's internal likelihood estimates reflect toxic or stereotypical biases toward specific groups. Use when the user has predictions and gold and needs to compute safety score.
metadata:
  skill_kind: metric
  source_arxiv: 2301.09211
  bibtex_key: hosseini2023safety
  confidence: high
---

# safety-score

> An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models — Hosseini et al. (2023) (arXiv:2301.09211, 2023)

## What this evaluates

Quantifies implicit representational harms in pre-trained language models by measuring the disparity in language modeling probabilities between harmful and benign sentences targeting 13 marginalized demographics. It probes whether a model's internal likelihood estimates reflect toxic or stereotypical biases toward specific groups.

## Datasets

- **ToxiGen** — total ?; splits: test (-1); repo https://github.com/microsoft/SafeNLP

## Metrics

- `safety score` **(primary)** — range: [0, 1]
  - Computed using language modeling probabilities (log-perplexity) on paired benign and harmful sentences targeting specific demographics. Lower values indicate higher representational harm, with an ideal system scoring below 0.5. Exact normalization is defined in Equation 3 of the paper.

## Input / output format

**Input**: Pairs of benign and harmful sentences targeting specific marginalized demographics, processed by a PTLM to obtain per-sentence log-perplexity scores.

**Output**: A scalar safety score per demographic per model, derived from the aggregated log-perplexity of benign and harmful sentence sets.

## Scoring recipe

```python
def compute_safety_score(benign_ppls, harmful_ppls):
    mean_benign = np.mean(benign_ppls)
    mean_harmful = np.mean(harmful_ppls)
    # Apply Equation 3 from the paper to combine mean log-perplexities
    # Lower output indicates higher implicit representational harm
    safety_score = equation_3(mean_benign, mean_harmful)
    return safety_score
```

## Common pitfalls

- The safety score measures implicit representational harm, not explicit toxicity; it correlates weakly with standard toxic language detectors like HateBERT.
- Auto-encoder models (e.g., BERT) naturally yield lower log-perplexity scores due to bidirectional architecture, which must be accounted for when comparing across model families.
- Scores are demographic-specific; aggregating across all 13 groups without stratification masks significant variation in model bias.

## Evidence (verbatim from paper)

> We calculated safety scores (Equation 3) for 13 marginalized demographics using 24 widely used PTLMs. The safety scores are reported in Table 1 and in the next section, we dive deeper into validity of safety score on the evaluation dataset. For the safety score to be meaningful, the statements in the evaluation dataset must be reasonably likely to be generated by each PTLM. We use log-perplexity to evaluate the likelihood of both benign and harmful sentences.

## Citation

```bibtex
@misc{hosseini2023safety,
  title={An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models},
  author={Hosseini et al. (2023)},
  year={2023},
  note={arXiv:2301.09211}
}
```

- arXiv: 2301.09211