Reddit Cssrs Screening Eval

---
name: reddit-cssrs-screening-eval
description: This benchmark evaluates zero-shot large language models on their ability to classify suicide risk severity from Reddit posts using the clinically validated Columbia-Suicide Severity Rating Scale (C-SSRS). It probes the models' ordinal classification capabilities, intent detection, and alignment with human clinical annotations across seven severity levels. Use when the user wants to benchmark on Reddit r/SuicideWatch posts (C-SSRS labeled), or asks about evaluating this task. Reports F1-Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.13480
  bibtex_key: [REDACTED]
  confidence: high
---

# reddit-cssrs-screening-eval

> Evaluating Reasoning LLMs for Suicide Screening with the Columbia-Suicide Severity Rating Scale — Patil et al. (2025) (arXiv:2505.13480, 2025)

## What this evaluates

This benchmark evaluates zero-shot large language models on their ability to classify suicide risk severity from Reddit posts using the clinically validated Columbia-Suicide Severity Rating Scale (C-SSRS). It probes the models' ordinal classification capabilities, intent detection, and alignment with human clinical annotations across seven severity levels.

## Datasets

- **Reddit r/SuicideWatch posts (C-SSRS labeled)** — total ?; splits: test (-1)

## Metrics

- `F1-Score` **(primary)** — range: [0, 1]
  - Weighted harmonic mean of precision and recall across all C-SSRS severity classes. Computed as the macro/weighted average to account for class imbalance.
- `Quadratic Weighted Kappa (QWK)` — range: [0, 1]
  - Ordinal agreement metric that weights disagreements by the squared distance between predicted and actual severity levels, penalizing errors between distant levels more heavily.
- `Spearman’s Rank Correlation (SRC)` — range: [-1, 1]
  - Non-parametric measure of rank-order correlation between predicted and ground-truth severity levels, assessing how well models preserve the ordinal gradient.
- `MAE` — range: other
  - Mean Absolute Error; average of the absolute differences between predicted and actual C-SSRS severity integers.
- `MSE` — range: other
  - Mean Squared Error; average of the squared differences between predicted and actual severity integers, emphasizing larger ordinal deviations.
- `Accuracy` — range: [0, 1]
  - Proportion of posts where the predicted severity level exactly matches the ground-truth C-SSRS label.

## Input / output format

**Input**: Raw text of a single Reddit post from r/SuicideWatch.

**Output**: A single integer from 0 to 6 representing the C-SSRS severity level (0: No ideation, 1: Wish to be dead, 2: Non-active suicidal thoughts, 3: Active ideation without intent, 4: Active ideation with intent but no plan, 5: Plan with intent, 6: Actual attempt).

## Scoring recipe

```python
def compute_metrics(preds, gold):
    n = len(gold)
    acc = sum(p == g for p, g in zip(preds, gold)) / n
    f1 = f1_score(gold, preds, average='weighted')
    mae = sum(abs(p - g) for p, g in zip(preds, gold)) / n
    mse = sum((p - g)**2 for p, g in zip(preds, gold)) / n
    qwk = quadratic_weighted_kappa(gold, preds)
    src = spearmanr(gold, preds).correlation
    return {'accuracy': acc, 'f1': f1, 'mae': mae, 'mse': mse, 'qwk': qwk, 'src': src}
```

## Common pitfalls

- Misclassifications predominantly occur between adjacent severity levels (e.g., 1-2, 3-4, 5-6) rather than random errors, making standard accuracy misleading without ordinal metrics.
- Models frequently confuse nuanced intent (e.g., hopelessness vs. explicit plan) and produce false positives on emotionally charged but non-suicidal posts, requiring careful thresholding or human review.
- Short or context-poor posts significantly increase misclassification rates, so dataset composition and post length distribution heavily influence reported performance.

## Evidence (verbatim from paper)

> The evaluation includes standard classification metrics (accuracy, precision, recall, F1-score) as well as regression-based error measures (MAE, MSE) and ordinal agreement metrics (Quadratic Weighted Kappa [QWK] and Spearman’s Rank Correlation [SRC]).

## Citation

```bibtex
@misc{patil2025evaluating,
  title={Evaluating Reasoning LLMs for Suicide Screening with the Columbia-Suicide Severity Rating Scale},
  author={Patil et al. (2025)},
  year={2025},
  note={arXiv:2505.13480}
}
```

- arXiv: 2505.13480