Surgveo Eval

---
name: surgveo-eval
description: Evaluates zero-shot surgical video generation models using a four-tiered Surgical Plausibility Pyramid. It probes the model's ability to maintain visual realism while correctly simulating domain-specific surgical causality, instrument handling, tissue feedback, and clinical intent over time. Use when the user wants to benchmark on SurgVeo benchmark, or asks about evaluating this task. Reports Visual Perceptual Plausibility.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.01775
  bibtex_key: chen2025surgveo
  confidence: high
---

# surgveo-eval

> How Far Are Surgeons from Surgical World Models? A Pilot Study on Zero-shot Surgical Video Generation with Expert Assessment — Chen et al. (2025) (arXiv:2511.01775, 2025)

## What this evaluates

Evaluates zero-shot surgical video generation models using a four-tiered Surgical Plausibility Pyramid. It probes the model's ability to maintain visual realism while correctly simulating domain-specific surgical causality, instrument handling, tissue feedback, and clinical intent over time.

## Datasets

- **SurgVeo benchmark** — total ?; splits: laparoscopic (-1), neurosurgery (-1); repo https://github.com/franciszchen/SurgVeo

## Metrics

- `Visual Perceptual Plausibility` **(primary)** — range: [1, 5]
  - Expert rating on a 1-5 scale assessing visual realism, texture quality, and lighting consistency of generated surgical frames.
- `Instrument Operation Plausibility` — range: [1, 5]
  - Expert rating on a 1-5 scale assessing the correctness of surgical tool manipulation, procedural actions, and anatomical targeting.
- `Environment Feedback Plausibility` — range: [1, 5]
  - Expert rating on a 1-5 scale assessing physical consistency of tissue response, fluid dynamics, and suction/irrigation behavior.
- `Surgical Intent Plausibility` — range: [1, 5]
  - Expert rating on a 1-5 scale assessing alignment with clinical goals, procedural logic, and step sequencing.

## Input / output format

**Input**: Text prompts describing surgical procedures (Baseline or Stage-aware strategy) used to condition video generation models.

**Output**: Generated surgical videos (laparoscopic or neurosurgery tracks).

## Scoring recipe

```python
def compute_surgveo_scores(generated_videos, track):
    # Two domain experts independently rate each video on 1-5 scale
    expert1_ratings = [e1_score(v, track) for v in generated_videos]
    expert2_ratings = [e2_score(v, track) for v in generated_videos]
    # Average across experts for each SPP dimension
    final_scores = [(s1 + s2) / 2 for s1, s2 in zip(expert1_ratings, expert2_ratings)]
    return final_scores
```

## Common pitfalls

- Assuming high visual fidelity implies correct surgical logic; the paper identifies a 'plausibility gap' where visual scores are high but causal/logical scores drop below 2.0.
- Expecting stage-aware prompting to significantly boost performance; results show it yields no consistent improvement over baseline prompts.
- Misclassifying error types; >93% of failures are high-level logical/surgical intent errors, not basic visual quality distortions.

## Evidence (verbatim from paper)

> Scores are reported on a 1-5 scale, where 5 is the best, and the standard deviation is calculated with the average score of two laparoscopic surgery experts. Our primary finding is a stark dichotomy in the performance of the generated surgical videos, clearly illustrated across both surgical types. The Veo-3 consistently excels at the base of the Surgical Plausibility Pyramid, achieving high scores in Visual Perceptual Plausibility.

## Citation

```bibtex
@misc{chen2025surgveo,
  title={How Far Are Surgeons from Surgical World Models? A Pilot Study on Zero-shot Surgical Video Generation with Expert Assessment},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2511.01775}
}
```

- arXiv: 2511.01775