Surgcot Eval

---
name: surgcot-eval
description: Evaluates multi-modal large language models on hierarchical spatiotemporal reasoning in surgical videos across five clinical dimensions: causal action ordering, cue-action alignment, affordance mapping, micro-transition localization, and anomaly onset tracking. The benchmark tests a model's ability to progressively narrow its focus from global video comprehension to fine-grained frame-level localization while maintaining logical consistency through a chain-of-thought protocol. Use when the user wants to benchmark on SurgCoT, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.20319
  bibtex_key: wang2026surgcot
  confidence: high
---

# surgcot-eval

> SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark — Gui Wang et al. (2026) (arXiv:2604.20319, 2026)

## What this evaluates

Evaluates multi-modal large language models on hierarchical spatiotemporal reasoning in surgical videos across five clinical dimensions: causal action ordering, cue-action alignment, affordance mapping, micro-transition localization, and anomaly onset tracking. The benchmark tests a model's ability to progressively narrow its focus from global video comprehension to fine-grained frame-level localization while maintaining logical consistency through a chain-of-thought protocol.

## Datasets

- **SurgCoT** — total 2841; splits: test (-1); repo https://github.com/CVI-SZU/SurgCoT

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions (both main and sub-questions) per task, averaged across the five reasoning dimensions. Computed as (correct predictions / total questions) * 100.

## Input / output format

**Input**: Surgical video clips (full videos for stage 1, bounded clips for stage 2, frames/patches for stage 3) paired with a clinically meaningful question, mutually exclusive multiple-choice options, domain knowledge priors, and video-grounded clues (temporal windows, spatial ROIs, or landmarks).

**Output**: A selected option from the provided candidates, optionally accompanied by a chain-of-thought rationale following the five-tuple protocol (Question→Option→Knowledge→Clue→Answer).

## Scoring recipe

```python
correct = 0
total = 0
for instance in dataset:
    pred = model.predict(instance.video, instance.question, instance.options, instance.knowledge, instance.clue)
    if pred == instance.answer:
        correct += 1
    total += 1
accuracy = (correct / total) * 100
```

## Common pitfalls

- Models often fail to respect the progressive conditioning chain (A1→A2→A3), leading to logical inconsistencies when stage 2 or 3 ignores validated outputs from previous stages.
- Distractors are carefully designed to be clinically plausible, requiring fine-grained spatiotemporal grounding rather than relying on coarse visual recognition or general medical knowledge.
- The benchmark evaluates both main questions and sub-questions; ignoring sub-questions underestimates performance on fine-grained localization and temporal anchoring tasks.

## Evidence (verbatim from paper)

> Table 2: Evaluation of 10 MLLMs across five clinical reasoning tasks under progressive settings (BL→KE→FC) shows: 1) commercial models outperform open-source and medical-specialized counterparts, and 2) the five-tuple annotation protocol improves reasoning accuracy under both (KE) and (FC) settings.

## Citation

```bibtex
@misc{wang2026surgcot,
  title={SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark},
  author={Gui Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.20319}
}
```

- arXiv: 2604.20319