Medcalc Bench Eval

---
name: medcalc-bench-eval
description: Probes LLMs' ability to perform evidence-based medical calculations by decomposing the task into formula selection, entity extraction, arithmetic computation, and final answer formatting. It evaluates both final numerical accuracy and granular step-wise reasoning to diagnose specific clinical and computational failure modes. Use when the user wants to benchmark on MedCalc-Bench, or asks about evaluating this task. Reports Step-wise LLM Evaluation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.16584
  bibtex_key: [REDACTED]
  confidence: high
---

# medcalc-bench-eval

> From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations — Wang et al. (2025) (arXiv:2509.16584, 2025)

## What this evaluates

Probes LLMs' ability to perform evidence-based medical calculations by decomposing the task into formula selection, entity extraction, arithmetic computation, and final answer formatting. It evaluates both final numerical accuracy and granular step-wise reasoning to diagnose specific clinical and computational failure modes.

## Datasets

- **MedCalc-Bench** — total 940; splits: test (940); repo https://github.com/Super-Billy/EMNLP-2025-MedRaC

## Metrics

- `Step-wise LLM Evaluation` **(primary)** — range: [0, 1]
  - Separately grades four stages: formula selection, entity extraction, arithmetic computation, and final answer formatting. Each stage is scored as binary correct or incorrect, and accuracy is reported per stage and across the pipeline.

## Input / output format

**Input**: Clinical calculation cases containing patient notes, calculator type (rule-based or equation-based), and required demographic/clinical variables to be computed.

**Output**: Final numerical answer (for direct setting), or a detailed reasoning trace followed by the final answer (for CoT, One-shot, MedPrompt, Self-Refine, and MedRaC variants).

## Scoring recipe

```python
def score_stepwise(prediction, gold):
    steps = ['formula', 'entities', 'calculation', 'final_answer']
    scores = {}
    for step in steps:
        pred_val = prediction[step]
        gold_val = gold[step]
        scores[step] = 1.0 if pred_val == gold_val else 0.0
    return scores
```

## Common pitfalls

- Relying solely on final-answer accuracy masks intermediate hallucinations or logical errors in clinical reasoning.
- Using wide tolerance margins for numerical answers can obscure clinically significant calculation mistakes.
- Assuming stronger LLMs benefit equally from external retrieval; larger models may already possess sufficient internal medical knowledge for rule-based tasks.

## Evidence (verbatim from paper)

> Our primary metric is the Step-wise LLM Evaluation proposed in Section[3.1], which separately grades formula selection, entity extraction, and arithmetic computation. For the direct setting, we score only the final answer, whereas all reasoning-based variants are assessed with the automatic step-wise rubric described above.

## Citation

```bibtex
@misc{wang2025fromscorestosteps,
  title={From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2509.16584}
}
```

- arXiv: 2509.16584