Wingpt 3.0 Benchmark Eval

---
name: wingpt-3.0-benchmark-eval
description: Evaluates large language models on comprehensive medical reasoning, clinical calculation, and general cognitive capabilities. It probes domain-specific knowledge application, diagnostic reasoning, and complex problem-solving in real-world clinical and academic settings. Use when the user wants to benchmark on MedCalc, MedReMCQ, CMMLU, MATH-500, MedQA-USMLE, MedMCQA, PubMedQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.17387
  bibtex_key: zhuang2025wingpt3
  confidence: high
---

# wingpt-3.0-benchmark-eval

> WiNGPT-3.0 Technical Report — Zhuang et al. (2025) (arXiv:2505.17387, 2025)

## What this evaluates

Evaluates large language models on comprehensive medical reasoning, clinical calculation, and general cognitive capabilities. It probes domain-specific knowledge application, diagnostic reasoning, and complex problem-solving in real-world clinical and academic settings.

## Datasets

- **MedCalc** — total 2811; splits: test (2811)
- **MedReMCQ** — total 700; splits: test (700)
- **CMMLU** — total 11582; splits: test (11582)
- **MATH-500** — total 500; splits: test (500)
- **MedQA-USMLE** — total 1273; splits: test (1273)
- **MedMCQA** — total 4183; splits: test (4183)
- **PubMedQA** — total 1000; splits: test (1000)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Average of 1 if the model's prediction matches the gold answer (or falls within the specified upper/lower bounds for MedCalc), else 0.
- `Micro-F1 score` — range: [0, 1]
  - Micro-averaged F1 score computed across all classes/options for the MedReMCQ dataset.

## Input / output format

**Input**: Multiple-choice questions or open-ended clinical/mathematical prompts requiring reasoning or calculation.

**Output**: Selected option letter/answer for MCQs, or computed numerical/textual answer for calculation/reasoning tasks.

## Scoring recipe

```python
def score_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred == gold: # or within bounds for MedCalc
            correct += 1
    return correct / len(golds)

def score_micro_f1(predictions, golds):
    tp = fp = fn = 0
    for pred, gold in zip(predictions, golds):
        if pred == gold: tp += 1
        elif pred != gold: fp += 1
        if gold != pred: fn += 1
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
```

## Common pitfalls

- MedCalc uses a tolerance-based accuracy metric where answers must fall within upper/lower bounds of the standard answer, not exact string match.
- Micro-F1 is used for MedReMCQ instead of standard accuracy, requiring class-level aggregation across all options.
- Benchmarks are evaluated on their test splits without explicit train/val splits mentioned in the report.

## Evidence (verbatim from paper)

> MedCalc uses accuracy as the metric which is based on whether the model’s computed answer falls within the upper and lower bounds of the standard answer. MedReMCQ is a dataset we developed in-house, tailored to medical reasoning scenarios. It is presented in a multiple-choice format and assessed using the Micro-F1 score. CMMLU, MATH-500, MedQA-USMLE, MedMCQA and PubMedQA use accuracy as the evaluation metric.

## Citation

```bibtex
@misc{zhuang2025wingpt3,
  title={WiNGPT-3.0 Technical Report},
  author={Zhuang et al. (2025)},
  year={2025},
  note={arXiv:2505.17387}
}
```

- arXiv: 2505.17387