Mentalbench Eval

---
name: mentalbench-eval
description: Evaluates large language models' ability to perform psychiatric diagnostic decision-making using DSM-5 criteria. It probes their capacity to handle information incompleteness, perform differential diagnosis among overlapping disorders, and calibrate diagnostic commitment under varying prompt constraints. Use when the user wants to benchmark on MentalBench, or asks about evaluating this task. Reports accuracy (exact match).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.12871
  bibtex_key: song2026mentalbench
  confidence: high
---

# mentalbench-eval

> MentalBench: A Benchmark for Evaluating Psychiatric Diagnostic Capability of Large Language Models — Hoyun Song et al. (2026) (arXiv:2602.12871, 2026)

## What this evaluates

Evaluates large language models' ability to perform psychiatric diagnostic decision-making using DSM-5 criteria. It probes their capacity to handle information incompleteness, perform differential diagnosis among overlapping disorders, and calibrate diagnostic commitment under varying prompt constraints.

## Datasets

- **MentalBench** — total 24750; splits: test (24750); repo https://github.com/HoyunS/MentalBench

## Metrics

- `accuracy (exact match)` **(primary)** — range: [0, 1]
  - Exact match accuracy. A prediction is scored as correct only if the model's selected diagnosis set exactly matches the complete set of ground-truth diagnoses; otherwise it is scored as incorrect.

## Input / output format

**Input**: Synthetic clinical case descriptions (patient narratives or structured charts) varying in information completeness and diagnostic complexity, accompanied by instructions specifying whether to select a single diagnosis or one or more diagnoses.

**Output**: A list of psychiatric disorder names/codes (DSM-5) representing the predicted diagnoses for the given case.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if set(pred) == set(gold):
        correct += 1
accuracy = correct / len(predictions)
```

## Common pitfalls

- Models often fail to infer the correct answer cardinality (single vs. multiple diagnoses) without explicit prompt constraints, leading to over- or under-diagnosis.
- Performance degrades sharply for disorders underrepresented in pre-training data (e.g., Schizoaffective Disorder), revealing knowledge retention gaps rather than reasoning flaws.
- Proprietary models tend to underperform on ambiguous cases due to rigid decisiveness, while open-source models struggle with unambiguous cases by failing to enforce exclusionary rules.

## Evidence (verbatim from paper)

> We measure accuracy using an exact match metric, recognizing a prediction as correct only if it identifies the complete set of ground-truth answers.

## Citation

```bibtex
@misc{song2026mentalbench,
  title={MentalBench: A Benchmark for Evaluating Psychiatric Diagnostic Capability of Large Language Models},
  author={Hoyun Song et al. (2026)},
  year={2026},
  note={arXiv:2602.12871}
}
```

- arXiv: 2602.12871