Perception Test Eval

---
name: perception-test-eval
description: Evaluates multimodal video models on core perception skills and reasoning types across six computational tasks, including tracking, temporal localization, and video question answering. It probes zero-shot and few-shot generalization on real-world videos with dense annotations. Use when the user wants to benchmark on Perception Test, or asks about evaluating this task. Reports top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.13786
  bibtex_key: patraucean2023perceptiontest
  confidence: high
---

# perception-test-eval

> Perception Test: A Diagnostic Benchmark for Multimodal Video Models — Pătrăucean et al. (2023) (arXiv:2305.13786, 2023)

## What this evaluates

Evaluates multimodal video models on core perception skills and reasoning types across six computational tasks, including tracking, temporal localization, and video question answering. It probes zero-shot and few-shot generalization on real-world videos with dense annotations.

## Datasets

- **Perception Test** — total 11600; splits: validation (-1); repo https://github.com/deepmind/perception_test

## Metrics

- `top-1 accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted answers out of the total number of multiple-choice questions.
- `Avg. IoU` — range: [0, 1]
  - Average Intersection over Union between predicted and ground-truth bounding box trajectories.
- `Avg. Jaccard` — range: [0, 1]
  - Average Jaccard index for point tracking trajectories.
- `mAP` — range: [0, 100]
  - mean Average Precision for temporal action or sound localization segments.
- `HOTA` — range: [0, 1]
  - Higher Order Tracking Accuracy for grounded video question answering bounding box tracks.

## Input / output format

**Input**: Video with audio, plus task-specific instructions (e.g., bounding box coordinates for tracking, or a natural language question with multiple-choice options for videoQA).

**Output**: Task-specific predictions: bounding box or point trajectories, lists of temporal segments, or a single selected answer choice.

## Scoring recipe

```python
def compute_top1_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

# For tracking/localization, compute per-frame IoU/Jaccard/mAP
# and average across the dataset as specified in Table 4.
```

## Common pitfalls

- Models often fail on counterfactual questions by latching onto visible video elements instead of imagining alternate realities.
- Fine-tuning is currently required for action and sound localization tasks because open-vocabulary models do not yet exist, violating the intended zero-shot evaluation setting.
- Hard negative options and adversarial actions in the dataset cause significant performance drops compared to standard benchmarks like NExT-QA.

## Evidence (verbatim from paper)

> For all the tasks, the video and audio are available as inputs, together with a task specification where applicable, e.g. the coordinates of a box to track for object tracking, or a language question and options for multiple-choice videoQA. multiple-choice videoQA | answer (1 out of 3) | top-1 accuracy | SeViLA | 46.2

## Citation

```bibtex
@misc{patraucean2023perceptiontest,
  title={Perception Test: A Diagnostic Benchmark for Multimodal Video Models},
  author={Pătrăucean et al. (2023)},
  year={2023},
  note={arXiv:2305.13786}
}
```

- arXiv: 2305.13786