Cbt Bench Eval

---
name: cbt-bench-eval
description: Evaluates LLMs on cognitive behavioral therapy (CBT) assistance across basic knowledge recall and cognitive model understanding. The benchmark probes multiple-choice knowledge acquisition and multi-label classification of cognitive distortions and core beliefs to measure therapeutic applicability and fine-grained clinical reasoning. Use when the user wants to benchmark on CBT-QA, CBT-CD, CBT-PC, CBT-FC, or asks about evaluating this task. Reports Accuracy, F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.13218
  bibtex_key: [REDACTED]
  confidence: high
---

# cbt-bench-eval

> CBT-Bench: Evaluating Large Language Models on Assisting Cognitive Behavior Therapy — Zhang et al. (2024) (arXiv:2410.13218, 2024)

## What this evaluates

Evaluates LLMs on cognitive behavioral therapy (CBT) assistance across basic knowledge recall and cognitive model understanding. The benchmark probes multiple-choice knowledge acquisition and multi-label classification of cognitive distortions and core beliefs to measure therapeutic applicability and fine-grained clinical reasoning.

## Datasets

- **CBT-QA** — total ?; splits: test (-1); repo https://github.com/mianzhang/CBT-Bench
- **CBT-CD** — total ?; splits: test (-1); repo https://github.com/mianzhang/CBT-Bench
- **CBT-PC** — total ?; splits: test (-1); repo https://github.com/mianzhang/CBT-Bench
- **CBT-FC** — total ?; splits: test (-1); repo https://github.com/mianzhang/CBT-Bench

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly answered multiple-choice questions. Calculated as correct predictions divided by total instances.
- `Precision` — range: [0, 1]
  - Weighted average of precision across all classes, where weights correspond to class portion (support).
- `Recall` — range: [0, 1]
  - Weighted average of recall across all classes, where weights correspond to class portion (support).
- `F1` **(primary)** — range: [0, 1]
  - Weighted harmonic mean of precision and recall across classes, weighted by class portion. F1 = 2 * (Precision * Recall) / (Precision + Recall).

## Input / output format

**Input**: Multiple-choice questions (Level I: CBT-QA; Level II: CBT-CD, CBT-PC, CBT-FC). Prompts are provided in Appendix C. Inference temperature is set to 0.0 and bfloat16 precision is used.

**Output**: Model selects one or more correct options from the provided choices.

## Scoring recipe

```python
def score(predictions, golds, task):
    if task == 'CBT-QA':
        correct = sum(1 for p, g in zip(predictions, golds) if p == g)
        return correct / len(golds)
    else:
        # Multi-label classification; weighted by class portion
        precision = compute_weighted_precision(predictions, golds)
        recall = compute_weighted_recall(predictions, golds)
        f1 = 2 * (precision * recall) / (precision + recall)
        return f1
```

## Common pitfalls

- Tasks are cast as multiple-choice questions but allow multiple correct options, requiring multi-label evaluation rather than single-choice.
- F1 scores are weighted/averaged by class portion, which can mask performance on minority cognitive distortion or core belief categories.
- Models are evaluated at temperature 0.0; higher temperatures may yield different results but break the reproducibility claims.

## Evidence (verbatim from paper)

> We report accuracy for CBT-QA and weighted precision, recall, and F1 score for the other three datasets.

## Citation

```bibtex
@misc{zhang2024cbtbench,
  title={CBT-Bench: Evaluating Large Language Models on Assisting Cognitive Behavior Therapy},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2410.13218}
}
```

- arXiv: 2410.13218