Shotbench Eval

---
name: shotbench-eval
description: Evaluates vision-language models on expert-level cinematic understanding by testing their ability to reason about shot-level visual properties. It probes fine-grained visual reasoning and spatial cognition across eight cinematography dimensions such as shot size, framing, camera angle, lens, lighting, composition, and movement. Use when the user wants to benchmark on ShotBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.21356
  bibtex_key: liu2025shotbench
  confidence: high
---

# shotbench-eval

> ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models — Liu et al. (2025) (arXiv:2506.21356, 2025)

## What this evaluates

Evaluates vision-language models on expert-level cinematic understanding by testing their ability to reason about shot-level visual properties. It probes fine-grained visual reasoning and spatial cognition across eight cinematography dimensions such as shot size, framing, camera angle, lens, lighting, composition, and movement.

## Datasets

- **ShotBench** — total 3500; splits: test (3500)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions. Computed per cinematography dimension and averaged across all eight dimensions (shot size, framing, camera angle, lens, lighting, composition, movement, camera movement).

## Input / output format

**Input**: A cinematic image (or frame) paired with a multiple-choice question asking to identify a specific cinematography property (e.g., shot size, camera angle, lighting).

**Output**: A single letter or text corresponding to the correct multiple-choice option.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().upper() == gold.strip().upper():
        correct += 1
accuracy = (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Confusing ShotBench (the 3.5k evaluation benchmark) with ShotQA (the 70k training dataset used for SFT/GRPO).
- Assuming the benchmark requires video input; the ablation study notes experiments were conducted on static images, yet it still evaluates camera movement understanding.
- Misinterpreting the score scale as 0-1 instead of 0-100 percent, as reported in the tables.

## Evidence (verbatim from paper)

> Evaluation of 24 VLMs reveals a critical gap: even top models like GPT-4o achieve <60% accuracy, with severe weaknesses in fine-grained visual reasoning and spatial cognition, especially in camera movement.

## Citation

```bibtex
@misc{liu2025shotbench,
  title={ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2506.21356}
}
```

- arXiv: 2506.21356