Video Panels Eval

---
name: video-panels-eval
description: Evaluates the ability of vision-language models to understand long videos using a training-free visual prompting strategy that combines consecutive frames into multi-frame 'panels'. It probes temporal reasoning, needle-in-a-haystack retrieval, and question-answering capabilities under varying context window constraints. Use when the user wants to benchmark on VideoMME, TimeScope, MLVU, MF2, VNBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.23724
  bibtex_key: [REDACTED]
  confidence: high
---

# video-panels-eval

> Video Panels for Long Video Understanding — Doorenbos et al. (2025) (arXiv:2509.23724, 2025)

## What this evaluates

Evaluates the ability of vision-language models to understand long videos using a training-free visual prompting strategy that combines consecutive frames into multi-frame 'panels'. It probes temporal reasoning, needle-in-a-haystack retrieval, and question-answering capabilities under varying context window constraints.

## Datasets

- **VideoMME** — total ?; splits: short (-1), medium (-1), long (-1)
- **TimeScope** — total 1950; splits: short (1500), long (450)
- **MLVU** — total 2593; splits: dev (2593)
- **MF2** — total 850; splits: (unstated)
- **VNBench** — total 5400; splits: (unstated)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions out of the total number of questions in the benchmark split.

## Input / output format

**Input**: Multi-frame images (panels) created by combining consecutive video frames, accompanied by the question and multiple-choice options formatted as expected by the VLM.

**Output**: A single uppercase letter corresponding to the correct option (e.g., 'A', 'B', 'C', 'D'), as instructed by 'Answer with the option’s letter from the given choices directly.'

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if pred.strip().upper() == gold.strip().upper():
            correct += 1
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Paneling reduces spatial resolution to increase temporal coverage, which can hurt performance on tasks requiring fine-grained visual details.
- The gamma hyperparameter triggers paneling based on video FPS; applying a fixed gamma across datasets with different frame rates may cause suboptimal performance on short videos.
- Models often output full sentences or explanations instead of the required single-letter answer, necessitating strict prompt adherence or post-processing to match the evaluation format.

## Evidence (verbatim from paper)

> We report the accuracy over all the questions for each benchmark.

## Citation

```bibtex
@misc{doorenbos2025videopanelslongvideo,
  title={Video Panels for Long Video Understanding},
  author={Doorenbos et al. (2025)},
  year={2025},
  note={arXiv:2509.23724}
}
```

- arXiv: 2509.23724