Cataract Lmm Eval

---
name: cataract-lmm-eval
description: Evaluates a model's ability to recognize and classify temporal surgical phases in cataract surgery videos, specifically testing classification accuracy and robustness to domain shift across different clinical centers. Use when the user wants to benchmark on Cataract-LMM Phase Recognition Subset, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.16371
  bibtex_key: ahmadi2025cataractlmm
  confidence: high
---

# cataract-lmm-eval

> Cataract-LMM: Large-Scale, Multi-Source, Multi-Task Benchmark for Deep Learning in Surgical Video Analysis — Ahmadi et al. (2025) (arXiv:2510.16371, 2025)

## What this evaluates

Evaluates a model's ability to recognize and classify temporal surgical phases in cataract surgery videos, specifically testing classification accuracy and robustness to domain shift across different clinical centers.

## Datasets

- **Cataract-LMM Phase Recognition Subset** — total 150; splits: train (80), val (26), test (44); repo https://github.com/MJAHMADEE/Cataract-LMM

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy computed over the 12 effective surgical phase classes (Viscoelastic and Anterior Chamber Flushing are merged into a single class).

## Input / output format

**Input**: Video clips sampled at 4 frames per second (fps) from surgical procedures.

**Output**: Predicted surgical phase label for each input video clip.

## Scoring recipe

```python
def compute_metric(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Class imbalance is pronounced; Viscoelastic and Anterior Chamber Flushing are explicitly merged into a single class to mitigate this.
- Test set contains out-of-distribution data from a second hospital (Noor) to evaluate domain shift generalization.
- Videos are downsampled to 4 fps for evaluation, which may affect temporal resolution compared to original 30/60 fps recordings.

## Evidence (verbatim from paper)

> To rigorously assess model generalization, we partitioned the dataset based on the clinic of origin. The training set (80 videos) and validation set (26 videos) were drawn exclusively from the Farabi hospital. The test set consisted of 44 videos: 23 unseen videos from the Farabi hospital (in-distribution) and all 21 videos from the Noor hospital (out-of-distribution). To handle challenging classes, the visually similar and underrepresented phases, Viscoelastic and Anterior Chamber Flushing, were merged into a single class.

## Citation

```bibtex
@misc{ahmadi2025cataractlmm,
  title={Cataract-LMM: Large-Scale, Multi-Source, Multi-Task Benchmark for Deep Learning in Surgical Video Analysis},
  author={Ahmadi et al. (2025)},
  year={2025},
  note={arXiv:2510.16371}
}
```

- arXiv: 2510.16371