Medcalceval Eval

---
name: medcalceval-eval
description: Evaluates large language models' quantitative reasoning and clinical calculation capabilities across multiple medical specialties. It probes the model's ability to correctly select medical formulas or scoring rules, extract relevant patient attributes from clinical text, and perform accurate multi-step numerical computations. Use when the user wants to benchmark on MedCalc-Eval, MedCalc-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.27267
  bibtex_key: [REDACTED]
  confidence: high
---

# medcalceval-eval

> MedCalc-Eval and MedCalc-Env: Advancing Medical Calculation Capabilities of Large Language Models — Mao et al. (2025) (arXiv:2510.27267, 2025)

## What this evaluates

Evaluates large language models' quantitative reasoning and clinical calculation capabilities across multiple medical specialties. It probes the model's ability to correctly select medical formulas or scoring rules, extract relevant patient attributes from clinical text, and perform accurate multi-step numerical computations.

## Datasets

- **MedCalc-Eval** — total ?; splits: test (-1); repo https://github.com/maokangkun/MedCalc-Eval
- **MedCalc-Bench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly calculated numerical results or scoring outputs. Matches are determined by exact numerical agreement or verifiable reward signals based on the ground-truth calculation.

## Input / output format

**Input**: Clinical scenario or patient record text containing medical parameters, alongside a specific calculation task (e.g., applying a clinical formula or scoring system).

**Output**: Numerical result or calculated score, potentially with intermediate reasoning steps.

## Scoring recipe

```python
def calculate_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        try:
            pred_val = float(pred)
            gold_val = float(gold)
            if abs(pred_val - gold_val) < 1e-3:
                correct += 1
        except ValueError:
            pass
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Unit conversion mistakes between different medical measurement systems.
- Failure to correctly extract or map clinical attributes from noisy patient records.
- Incorrect application of multi-condition scoring rules (e.g., CHA2DS2-VASc).

## Evidence (verbatim from paper)

> The accuracy of most models is below 30%, with the best zero-shot performance from Qwen3-235B-A22B only reaching 31.1% on MedCalc-Eval.

## Citation

```bibtex
@misc{mao2025medcalceval,
  title={MedCalc-Eval and MedCalc-Env: Advancing Medical Calculation Capabilities of Large Language Models},
  author={Mao et al. (2025)},
  year={2025},
  note={arXiv:2510.27267}
}
```

- arXiv: 2510.27267