Medvidbench Eval

---
name: medvidbench-eval
description: Evaluates heterogeneous medical video understanding across classification, grounding, and captioning tasks. Probes a model's ability to perform clinical safety checks, predict surgical actions, assess skills, localize temporal/spatial events, and generate precise medical video descriptions. Use when the user wants to benchmark on MedVidBench (Standard), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.06581
  bibtex_key: su2025medgrpo
  confidence: high
---

# medvidbench-eval

> MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understanding — Su et al. (2025) (arXiv:2512.06581, 2025)

## What this evaluates

Evaluates heterogeneous medical video understanding across classification, grounding, and captioning tasks. Probes a model's ability to perform clinical safety checks, predict surgical actions, assess skills, localize temporal/spatial events, and generate precise medical video descriptions.

## Datasets

- **MedVidBench (Standard)** — total 51505; splits: train (45260), test (6245)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correct predictions for classification tasks (CVS, NA, SA). Computed as correct predictions divided by total samples.
- `mIoU` — range: [0, 1]
  - Mean Intersection over Union for temporal (TAG) and spatiotemporal (STG) grounding tasks. Computed as the intersection of predicted and ground-truth bounding boxes divided by their union, averaged across samples at IoU thresholds of 0.3 and 0.5.
- `F1 score` — range: [0, 1]
  - Harmonic mean of precision and recall for Dense Video Captioning (DVC), evaluating the overlap between generated and ground-truth captions.
- `LLM judge score` — range: other
  - Comparative similarity score generated by a medical LLM judge evaluating five clinical dimensions: terminology precision, instrument/anatomy identification, specificity, procedural context, and action accuracy.

## Input / output format

**Input**: Video frames (sampled at 1 FPS) paired with a text prompt/question specifying the task (e.g., region captioning, action prediction, safety assessment).

**Output**: Text response: clinical description, action label, safety status, or structured caption depending on the task.

## Scoring recipe

```python
def compute_metric(task, pred, gold):
    if task in ['CVS', 'NA', 'SA']:
        return 'accuracy', 1.0 if pred == gold else 0.0
    elif task in ['STG', 'TAG']:
        iou = intersection(pred_box, gold_box) / union(pred_box, gold_box)
        return 'mIoU', iou
    elif task == 'DVC':
        return 'F1 score', f1_score(pred_caption, gold_caption)
    elif task in ['DVC', 'VS', 'RC']:
        return 'LLM judge score', medical_llm_judge.evaluate(pred_caption, gold_caption)
    return None, None
```

## Common pitfalls

- Removing cross-dataset reward normalization causes catastrophic training collapse due to high magnitude differences between tasks.
- Using standard semantic similarity metrics for captioning inflates scores without capturing clinical terminology precision or anatomical accuracy.
- The LargeScale version has natural task imbalance favoring captioning, which can skew scaling law experiments if not accounted for.

## Evidence (verbatim from paper)

> Video-level tasks include Video Summarization (VS) measured by LLM judge score; Critical View of Safety (CVS) assessed by accuracy; Next Action Prediction (NA) evaluated by accuracy; and Skill Assessment (SA) measured by accuracy. Segment-level tasks include Temporal Action Grounding (TAG) measured by mean IoU at thresholds 0.3 and 0.5; Dense Video Captioning (DVC) evaluated by LLM judge score and F1 score; and Region Captioning (RC) assessed by LLM judge score. Frame-level tasks include Spatiotemporal Grounding (STG) measured by mIoU.

## Citation

```bibtex
@misc{su2025medgrpo,
  title={MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understanding},
  author={Su et al. (2025)},
  year={2025},
  note={arXiv:2512.06581}
}
```

- arXiv: 2512.06581