Mimic Cdm Eval

---
name: mimic-cdm-eval
description: Evaluates large language models on clinical decision-making tasks by predicting diagnoses from structured patient evidence. It probes whether models possess latent domain-specific reasoning capabilities that are masked by unfamiliarity with benchmark input formats and task definitions. Use when the user wants to benchmark on MIMIC-CDM, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.12875
  bibtex_key: wang2026testtime
  confidence: high
---

# mimic-cdm-eval

> Test-time RL alignment exposes task familiarity artifacts in LLM benchmarks — Wang et al. (2026) (arXiv:2603.12875, 2026)

## What this evaluates

Evaluates large language models on clinical decision-making tasks by predicting diagnoses from structured patient evidence. It probes whether models possess latent domain-specific reasoning capabilities that are masked by unfamiliarity with benchmark input formats and task definitions.

## Datasets

- **MIMIC-CDM** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted diagnoses out of the total number of clinical cases evaluated.

## Input / output format

**Input**: A single prompt containing all available clinical information for a case (patient history, physical examination, lab results, and imaging reports). Sections exceeding 1,000 tokens are pre-summarized using GPT5-nano.

**Output**: A direct prediction of the underlying pathology or diagnosis.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Direct evaluation without task alignment systematically underestimates model capabilities due to unfamiliarity with domain-specific formats.
- Long clinical sections (e.g., imaging reports) exceed context limits and require external summarization (GPT5-nano) before model input.
- Task alignment via TTRA can drastically alter relative model rankings, making base models appear comparable to fine-tuned or human baselines.

## Evidence (verbatim from paper)

> We consider the MIMIC-CDM full-information(MIMIC-CDM-FI) setup of the benchmark, where we evaluate the models by providing all available clinical information in a single prompt. Figure 8 shows the clinical decision-making accuracy of several base models before and after alignment with TTRA, alongside a human doctor baseline reported in[[13]]. Without alignment, the Gemma base models[[32]] perform poorly. For instance, Gemma-3-1B reaches only 21.9% accuracy, which is worse than random guessing. After applying TTRA, all Gemma models improve substantially.

## Citation

```bibtex
@misc{wang2026testtime,
  title={Test-time RL alignment exposes task familiarity artifacts in LLM benchmarks},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2603.12875}
}
```

- arXiv: 2603.12875