Clinicrealm Eval

---
name: clinicrealm-eval
description: Evaluates clinical prediction capabilities on unstructured notes and structured EHR data. It benchmarks zero-shot LLMs, finetuned BERTs, and conventional ML/DL models on mortality, readmission, and length-of-stay prediction tasks. The setup tests out-of-the-box prompting versus task-specific finetuning across diverse model families. Use when the user wants to benchmark on MIMIC-IV, TJH, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.18525
  bibtex_key: [REDACTED]
  confidence: high
---

# clinicrealm-eval

> ClinicRealm: Re-evaluating Large Language Models with Conventional Machine Learning for Non-Generative Clinical Prediction Tasks — Yinghao Zhu et al. (2024) (arXiv:2407.18525, 2024)

## What this evaluates

Evaluates clinical prediction capabilities on unstructured notes and structured EHR data. It benchmarks zero-shot LLMs, finetuned BERTs, and conventional ML/DL models on mortality, readmission, and length-of-stay prediction tasks. The setup tests out-of-the-box prompting versus task-specific finetuning across diverse model families.

## Datasets

- **MIMIC-IV** — total ?; splits: test (-1)
- **TJH** — total ?; splits: test (-1)

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
- `AUPRC` — range: [0, 1]
  - Area under the Precision-Recall Curve, summarizing precision-recall trade-offs, particularly useful for imbalanced datasets.
- `MAE` — range: other
  - Mean Absolute Error, the average of absolute differences between predicted and actual values for regression tasks like Length of Stay.
- `MSE` — range: other
  - Mean Squared Error, the average of squared differences between predicted and actual values.
- `RMSE` — range: other
  - Root Mean Squared Error, the square root of MSE, providing error magnitude in the same units as the target variable.

## Input / output format

**Input**: Unstructured clinical notes (text) or structured Electronic Health Record (EHR) features (tabular/codified patient data).

**Output**: Binary classification probability or label (mortality/readmission) or continuous regression value (Length of Stay).

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    auc = roc_auc_score(gold, predictions)
    aupr = average_precision_score(gold, predictions)
    mae = mean_absolute_error(gold, predictions)
    mse = mean_squared_error(gold, predictions)
    rmse = np.sqrt(mse)
    return {'AUROC': auc, 'AUPRC': aupr, 'MAE': mae, 'MSE': mse, 'RMSE': rmse}
```

## Common pitfalls

- All metric values in the tables are multiplied by 100 for readability; divide by 100 to get standard [0,1] or raw scale values.
- Standard deviations are reported via bootstrapping over all test set samples 100 times, not standard cross-validation splits.
- Zero-shot prompting settings for LLMs are evaluated differently than finetuned settings; some smaller LLMs failed to parse prompts and were excluded.

## Evidence (verbatim from paper)

> We use a bootstrapping strategy on all test set samples 100 times to report the mean±std results. All metrics are multiplied by 100 for readability purposes. For mortality prediction, DeepSeek-R1 achieved the highest AUROC (97.64%), closely followed by o3-mini-high (97.58%) and DeepSeek-V3 (97.13%).

## Citation

```bibtex
@misc{zhu2024clinicrealm,
  title={ClinicRealm: Re-evaluating Large Language Models with Conventional Machine Learning for Non-Generative Clinical Prediction Tasks},
  author={Yinghao Zhu et al. (2024)},
  year={2024},
  note={arXiv:2407.18525}
}
```

- arXiv: 2407.18525