Wildguard Moderation Eval

---
name: wildguard-moderation-eval
description: This evaluation protocol assesses the safety moderation capabilities of LLMs and dedicated moderation models. It probes their ability to detect harmful content in user prompts, classify harmful or safe model responses, and identify whether a model appropriately refuses unsafe requests across multiple risk categories. Use when the user wants to benchmark on ToxicChat, OpenAI Mod, AegisSafetyTest, SimpleSafetyTests, Harmbench Prompt, Harmbench Resp, BeaverTails, SafeRLHF, XSTest-Resp, WildGuardTest, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.18495
  bibtex_key: han2024wildguard
  confidence: medium
---

# wildguard-moderation-eval

> WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs — Han et al. (2024) (arXiv:2406.18495, 2024)

## What this evaluates

This evaluation protocol assesses the safety moderation capabilities of LLMs and dedicated moderation models. It probes their ability to detect harmful content in user prompts, classify harmful or safe model responses, and identify whether a model appropriately refuses unsafe requests across multiple risk categories.

## Datasets

- **ToxicChat** — total 2853; splits: test (2853)
- **OpenAI Mod** — total 1680; splits: test (1680)
- **AegisSafetyTest** — total 359; splits: test (359)
- **SimpleSafetyTests** — total 100; splits: test (100)
- **Harmbench Prompt** — total 239; splits: test (239)
- **Harmbench Resp** — total 602; splits: test (602)
- **BeaverTails** — total 3021; splits: test (3021)
- **SafeRLHF** — total 1000; splits: test (1000)
- **XSTest-Resp** — total 746; splits: test (746)
- **WildGuardTest** — total 4648; splits: test (4648)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall, calculated per class or macro-averaged across all risk categories. Used as the headline metric for refusal detection and harm classification.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly classified instances (harmful/benign or refusal/compliance) out of the total evaluated instances.

## Input / output format

**Input**: Text prompts, and in response-level benchmarks, paired model-generated responses.

**Output**: Binary or multi-class classification labels indicating harm/refusal status, or probability scores for each risk category.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels):
    # predictions and gold_labels are lists of class labels or probabilities
    f1 = f1_score(gold_labels, predictions, average='macro')
    acc = accuracy_score(gold_labels, predictions)
    return {'f1': f1, 'accuracy': acc}
```

## Common pitfalls

- Datasets are frequently subsampled (e.g., BeaverTails, SafeRLHF) to reduce evaluation cost, which may alter performance statistics compared to the full test set.
- Tasks are strictly divided into prompt-level harm, response-level harm, and refusal detection; evaluating the wrong task boundary leads to invalid results.
- Adversarial prompts are treated separately from standard prompts in several benchmarks, requiring distinct evaluation pipelines and reporting.

## Evidence (verbatim from paper)

> up to 26.4% F1 gain on refusal detection and outperforming GPT-4 by 3.9% on adversarial prompt harm.

## Citation

```bibtex
@misc{han2024wildguard,
  title={WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs},
  author={Han et al. (2024)},
  year={2024},
  note={arXiv:2406.18495}
}
```

- arXiv: 2406.18495