Surglaivi Eval

---
name: surglaivi-eval
description: Evaluates zero-shot and linear-probe transferability of surgical vision-language models across laparoscopic and robotic video modalities. Probes hierarchical procedural understanding (phase, step, action) and object-centric recognition (tools, instrument-verb-target triplets) under varying temporal contexts and data regimes. Use when the user wants to benchmark on Cholec80, AutoLaparo, GraSP, SARRARP50, CholecT50, or asks about evaluating this task. Reports video-wise F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.10555
  bibtex_key: [REDACTED]
  confidence: high
---

# surglaivi-eval

> SurgLaVi: Large-Scale Hierarchical Dataset for Surgical Vision-Language Representation Learning — Alejandra Perez et al. (arXiv:2509.10555, 2025)

## What this evaluates

Evaluates zero-shot and linear-probe transferability of surgical vision-language models across laparoscopic and robotic video modalities. Probes hierarchical procedural understanding (phase, step, action) and object-centric recognition (tools, instrument-verb-target triplets) under varying temporal contexts and data regimes.

## Datasets

- **Cholec80** — total ?; splits: test (-1)
- **AutoLaparo** — total ?; splits: test (-1)
- **GraSP** — total ?; splits: test (-1)
- **SARRARP50** — total ?; splits: test (-1)
- **CholecT50** — total ?; splits: test (-1)

## Metrics

- `video-wise Accuracy` — range: [0, 1]
  - Proportion of correctly classified video segments or frames, aggregated per video instance following Twinanda et al.
- `video-wise F1-score` **(primary)** — range: [0, 1]
  - Macro-averaged F1 score computed over video-wise predictions, balancing precision and recall across all surgical classes.
- `mAP` — range: [0, 1]
  - Mean Average Precision computed by treating cosine similarity scores as class probabilities for multi-label tool presence and triplet recognition tasks.

## Input / output format

**Input**: A 16-frame (or 32-frame for linear probing) temporal window of a surgical video, encoded into a mean-pooled feature vector, compared against text embeddings of candidate classes.

**Output**: Top-similarity class label for classification tasks; similarity scores treated as class probabilities for multi-label tool presence and triplet recognition tasks.

## Scoring recipe

```python
def evaluate(predictions, gold, task_type):
    if task_type in ['phase', 'step', 'action']:
        acc = (predictions == gold).mean()
        f1 = f1_score(gold, predictions, average='macro')
        return {'accuracy': acc, 'f1': f1}
    else: # tool presence or triplets
        probs = predictions # similarity scores
        mAP = average_precision_score(gold, probs)
        return {'mAP': mAP}
```

## Common pitfalls

- Using inconsistent inference prompts across models instead of each method's original prompts, which breaks fairness.
- Confusing frame-wise similarity scores with video-wise aggregated metrics (Accuracy/F1 vs mAP) without proper temporal pooling.
- Ignoring the impact of temporal window size (1, 8, 16, 32 frames) on performance, especially for fine-grained step recognition where 8 frames peak.

## Evidence (verbatim from paper)

> We report video-wise Accuracy and F1-score following Twinanda et al. [[22]]. For tool presence and CholecT50 triplets, we treat similarity scores as class probabilities and report mAP.

## Citation

```bibtex
@misc{perez2025surglavi,
  title={SurgLaVi: Large-Scale Hierarchical Dataset for Surgical Vision-Language Representation Learning},
  author={Alejandra Perez et al.},
  year={2025},
  note={arXiv:2509.10555}
}
```

- arXiv: 2509.10555