Wmdp Eval

---
name: wmdp-eval
description: Evaluates large language models' knowledge of hazardous topics in biosecurity, cybersecurity, and chemical security, as well as their general knowledge and fluency. It serves as a proxy for measuring dual-use risk and benchmarking unlearning methods. Use when the user wants to benchmark on WMDP, or asks about evaluating this task. Reports WMDP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.03218
  bibtex_key: li2024wmdp
  confidence: high
---

# wmdp-eval

> The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning — Li et al. (2024) (arXiv:2403.03218, 2024)

## What this evaluates

Evaluates large language models' knowledge of hazardous topics in biosecurity, cybersecurity, and chemical security, as well as their general knowledge and fluency. It serves as a proxy for measuring dual-use risk and benchmarking unlearning methods.

## Datasets

- **WMDP** — total 3668; splits: test (3668)

## Metrics

- `WMDP` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions using top-1 logit selection. Lower scores indicate less hazardous knowledge.
- `MMLU` — range: percent
  - Percentage of correctly answered multiple-choice questions across college and high school subjects.
- `MT-Bench` — range: other
  - Single-evaluation mode scored by GPT-4-0613 as an automated judge.

## Input / output format

**Input**: Zero-shot multiple-choice question prompt with four options (A, B, C, D) and a trailing 'Answer:' token.

**Output**: Model generates the top logit choice (A, B, C, or D).

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100
```

## Common pitfalls

- WMDP scores are lower-is-better (indicated by ↓ in tables), unlike standard benchmarks where higher is better.
- MT-Bench evaluation relies on GPT-4 as an automated judge rather than exact-match or human grading.
- The benchmark uses a strict zero-shot format with a fixed prompt template; deviating from the template may alter model behavior.

## Evidence (verbatim from paper)

> We use the default lm-evaluation-harness zero-shot format for MMLU and WMDP, taking the top logit between A, B, C, and D as the answer choice. ... Table 2: Comparing base models and unlearning methods on question-answer evaluation (WMDP, MMLU) and fluency (MT-Bench). All WMDP and MMLU scores are percentage points.

## Citation

```bibtex
@misc{li2024wmdp,
  title={The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2403.03218}
}
```

- arXiv: 2403.03218