Video Oasis Eval

---
name: video-oasis-eval
description: This protocol audits video understanding benchmarks to measure genuine spatio-temporal reasoning versus shortcut reliance. It filters out samples solvable without video context and evaluates models under diagnostic conditions (e.g., blind, audio-only, center-frame) to quantify performance degradation and dependency on actual video content. Use when the user wants to benchmark on EgoSchema, ImplicitQA, VSI-Bench, TVBench, VCR-Bench, RTV-Bench, Video-Holmes, MINERVA, MMR-V, VideoMME, MVBench, LVBench, LongVideoBench, MLVU, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.29616
  bibtex_key: lim2026videooasis
  confidence: high
---

# video-oasis-eval

> Video-Oasis: Rethinking Evaluation of Video Understanding — Lim et al. (2026) (arXiv:2603.29616, 2026)

## What this evaluates

This protocol audits video understanding benchmarks to measure genuine spatio-temporal reasoning versus shortcut reliance. It filters out samples solvable without video context and evaluates models under diagnostic conditions (e.g., blind, audio-only, center-frame) to quantify performance degradation and dependency on actual video content.

## Datasets

- **EgoSchema** — total 500; splits: test (-1)
- **ImplicitQA** — total 766; splits: test (-1)
- **VSI-Bench** — total 2490; splits: test (-1)
- **TVBench** — total 2205; splits: test (-1)
- **VCR-Bench** — total 511; splits: test (-1)
- **RTV-Bench** — total 4608; splits: test (-1)
- **Video-Holmes** — total 1837; splits: test (-1)
- **MINERVA** — total 1358; splits: test (-1)
- **MMR-V** — total 1257; splits: test (-1)
- **VideoMME** — total 2700; splits: test (-1)
- **MVBench** — total 3000; splits: test (-1)
- **LVBench** — total 1345; splits: test (-1)
- **LongVideoBench** — total 1337; splits: test (-1)
- **MLVU** — total 502; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions on the filtered dataset or under specific diagnostic conditions. Calculated as (correct predictions / total evaluated samples) * 100.
- `filtering_ratio` — range: percent
  - Proportion of samples removed from the original benchmark because they are identified as shortcut-prone or solvable without video context. Calculated as (removed samples / original samples) * 100.
- `performance_gap` — range: percent
  - Absolute difference in accuracy between the original benchmark set and the Video-Oasis filtered set, indicating how much performance drops when shortcuts are removed.

## Input / output format

**Input**: Video clips with associated multiple-choice or open-ended questions. For diagnostic tests, inputs are modified: visual input removed (Blind), audio transcript only (Audio), concatenated captions (Narrative), single center frame (Center-Frame), randomly shuffled frames (Frame Shuffling), or independent frame processing (Bag-of-Frames).

**Output**: Model predictions (selected option or generated answer) for each question.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

def compute_filtering_ratio(original_count, remaining_count):
    removed = original_count - remaining_count
    return (removed / original_count) * 100

def compute_performance_gap(acc_original, acc_remaining):
    return abs(acc_original - acc_remaining)
```

## Common pitfalls

- Models may achieve high accuracy on original benchmarks by relying on textual priors or static frame cues rather than actual video understanding.
- Diagnostic tests like the Audio or Narrative tests can yield high scores if benchmarks contain redundant textual information, masking poor spatio-temporal reasoning.
- Filtering ratio does not strictly correlate with original accuracy, meaning high-performing benchmarks can still contain many shortcut samples.

## Evidence (verbatim from paper)

> MVBench achieves a higher original accuracy (71.2) than EgoSchema (62.4), yet EgoSchema exhibits both a higher filtering ratio (75.8 vs. 66.0) and a substantially larger performance gap (40.5 vs. 20.7).

## Citation

```bibtex
@misc{lim2026videooasis,
  title={Video-Oasis: Rethinking Evaluation of Video Understanding},
  author={Lim et al. (2026)},
  year={2026},
  note={arXiv:2603.29616}
}
```

- arXiv: 2603.29616