Spookybench Eval

---
name: spookybench-eval
description: Evaluates video-language models' ability to recognize and report content encoded purely in temporal sequences of noise-like frames, probing their temporal pattern recognition and susceptibility to 'time-blindness' despite strong spatial reasoning. Use when the user wants to benchmark on SpookyBench, or asks about evaluating this task. Reports exact-match accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.24867
  bibtex_key: [REDACTED]
  confidence: high
---

# spookybench-eval

> Time Blindness: Why Video-Language Models Can't See What Humans Can? — Upadhyay et al. (2025) (arXiv:2505.24867, 2025)

## What this evaluates

Evaluates video-language models' ability to recognize and report content encoded purely in temporal sequences of noise-like frames, probing their temporal pattern recognition and susceptibility to 'time-blindness' despite strong spatial reasoning.

## Datasets

- **SpookyBench** — total ?; splits: test (-1)

## Metrics

- `exact-match accuracy` **(primary)** — range: percent
  - Accuracy is calculated as the fraction of correctly identified videos: 1/N * sum(1(r_i in L_i)), where r_i is the model response, L_i is the ground truth label (or set of acceptable labels for ambiguous categories), and 1 is the indicator function.

## Input / output format

**Input**: Video sequences (or sampled frames) with a prompt instructing the model to respond with 1-5 words identifying the content.

**Output**: 1-5 words identifying the content.

## Scoring recipe

```python
correct = 0
for i in range(N):
    if predictions[i] in labels[i]:
        correct += 1
accuracy = (correct / N) * 100
```

## Common pitfalls

- Models are constrained to output only 1-5 words, but many fail to produce any valid response or hallucinate unrelated content.
- For Object Images and Dynamic Scenes categories, multiple semantically valid responses are accepted, yet all tested VLMs still achieved 0% accuracy, indicating a fundamental temporal processing flaw rather than just prompt sensitivity.

## Evidence (verbatim from paper)

> We evaluate model performance using exact match accuracy between model responses and our labels. For the Text and Shapes categories, each video has a single correct label $y_{i}$. For Object Images and Dynamic Scenes categories, we define a set of acceptable labels $Y_{i}\={y_{i1},y_{i2},\ldots,y_{in}}$ to account for semantic ambiguity. For example, a video showing “a man playing basketball” accepts responses such as “playing basketball,” “man”, “human”, or “woman playing basketball” as correct. Formally, for each video $i$, given a model response $r_{i}$ and corresponding label or set of labels $L_{i}$ (where $L_{i}\=y_{i}$ for Text and Shapes, or $L_{i}\=Y_{i}$ for objects and dynamic scenes), we calculate the accuracy as: $\text{Accuracy}\=\frac{1}{N}\sum_{i\=1}^{N}\mathbbm{1}(r_{i}\in L_{i})$

## Citation

```bibtex
@misc{upadhyay2025timeblindness,
  title={Time Blindness: Why Video-Language Models Can't See What Humans Can?},
  author={Upadhyay et al. (2025)},
  year={2025},
  note={arXiv:2505.24867}
}
```

- arXiv: 2505.24867