Medscope Svu Eval

---
name: medscope-svu-eval
description: Evaluates multimodal models on multi-grained video description and fine-grained temporal/perceptual visual reasoning using long-form medical videos. Use when the user wants to benchmark on SVU-31K, or asks about evaluating this task. Reports CI, DO, CU, TU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.13332
  bibtex_key: li2026medscope
  confidence: high
---

# medscope-svu-eval

> MedScope: Incentivizing "Think with Videos" for Clinical Reasoning via Coarse-to-Fine Tool Calling — Li et al. (2026) (arXiv:2602.13332, 2026)

## What this evaluates

Evaluates multimodal models on multi-grained video description and fine-grained temporal/perceptual visual reasoning using long-form medical videos.

## Datasets

- **SVU-31K** — total ?; splits: test (-1)

## Metrics

- `CI, DO, CU, TU` **(primary)** — range: other
  - Scores for Correctness of Information, Detail Orientation, Contextual Understanding, and Temporal Understanding, assessed via human or LLM judges on a continuous scale.
- `BLEU-4, CIDEr, METEOR, ROUGE-L` — range: percent
  - Standard n-gram and weighted overlap metrics for evaluating generated video descriptions and reasoning text against reference captions.
- `Tool usage rate` — range: percent
  - Percentage of instances where the model invokes a tool during reasoning.

## Input / output format

**Input**: Long-form medical video paired with a text prompt asking for video description or visual reasoning.

**Output**: Text response containing video descriptions, reasoning steps, tool calls, and final answers.

## Scoring recipe

```python
def compute_text_metrics(preds, refs):
    bleu4 = nltk.bleu4(preds, refs)
    cider = compute_cider(preds, refs)
    meteor = compute_meteor(preds, refs)
    rouge_l = compute_rouge(preds, refs, 'rougeL')
    return bleu4, cider, meteor, rouge_l

def compute_tool_rate(preds):
    return sum(1 for p in preds if 'tool_call' in p) / len(preds)
```

## Common pitfalls

- CI, DO, CU, TU scores are reported as continuous values (e.g., 4.77), likely from LLM-as-judge or human raters on a scale, not standard discrete accuracy.
- Tool usage rate is reported as a percentage but the paper does not specify the exact tool API or how tool calls are parsed from the model's output.

## Evidence (verbatim from paper)

> On fine-grained reasoning, MedScope sets a new state of the art, reaching 4.56 CIDEr on temporal reasoning and 10.93 CIDEr on perceptual reasoning, with clear gains over the medical baseline SurgVidLM.

## Citation

```bibtex
@misc{li2026medscope,
  title={MedScope: Incentivizing "Think with Videos" for Clinical Reasoning via Coarse-to-Fine Tool Calling},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2602.13332}
}
```

- arXiv: 2602.13332