Med Tiv Eval

---
name: med-tiv-eval
description: Evaluates a verifier model's ability to distinguish correct from erroneous reasoning traces in medical question-answering tasks. It measures how well tool-integrated reinforcement learning improves factual justification and reduces hallucination compared to static reward models. Use when the user wants to benchmark on MedQA, MedMCQA, MMLU-Med, MedXpertQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.20221
  bibtex_key: zhang2026medtiv
  confidence: high
---

# med-tiv-eval

> Scaling Medical Reasoning Verification via Tool-Integrated Reinforcement Learning — Zhang et al. (2026) (arXiv:2601.20221, 2026)

## What this evaluates

Evaluates a verifier model's ability to distinguish correct from erroneous reasoning traces in medical question-answering tasks. It measures how well tool-integrated reinforcement learning improves factual justification and reduces hallucination compared to static reward models.

## Datasets

- **MedQA** — total ?; splits: test (-1)
- **MedMCQA** — total ?; splits: test (-1)
- **MMLU-Med** — total ?; splits: test (-1)
- **MedXpertQA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - The percentage of correctly judged reasoning traces out of the total evaluated traces. Calculated as (correct predictions / total predictions) * 100.

## Input / output format

**Input**: A medical question and a candidate multi-step reasoning trace generated by a frozen generator model.

**Output**: Binary judgment (correct/incorrect) or a continuous confidence score $r_\tau \in [0,1]$ derived from softmax probabilities of the 1 and 0 tokens.

## Scoring recipe

```python
def calculate_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Averages accuracy across four benchmarks of vastly different sizes without weighting, which can skew the reported 'Avg.' score.
- Confuses the verifier's judgment accuracy with the generator's final answer accuracy; test-time search strategies (e.g., Best-of-N) further decouple verifier performance from final task performance.

## Evidence (verbatim from paper)

> We evaluated Med-TIV on four open-source medical question-answering benchmarks: MedQA*(Jin et al., [2020])*, MedMCQA*(Pal et al., [2022])*, MMLU-Med*(Hendrycks et al., [2021])*, and MedXpertQA*(Zuo et al., [2025])*, using accuracy as the evaluation metric.

## Citation

```bibtex
@misc{zhang2026medtiv,
  title={Scaling Medical Reasoning Verification via Tool-Integrated Reinforcement Learning},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2601.20221}
}
```

- arXiv: 2601.20221