Video Reality Test Eval

---
name: video-reality-test-eval
description: This benchmark probes the ability of video-language models and humans to distinguish real ASMR videos from AI-generated ones, evaluating perceptual realism and audio-visual consistency. It also measures how effectively video generation models can deceive video understanding models by producing indistinguishable synthetic content. Use when the user wants to benchmark on Video Reality Test, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.13281
  bibtex_key: wang2025videorealitytest
  confidence: high
---

# video-reality-test-eval

> Video Reality Test: Can AI-Generated ASMR Videos fool VLMs and Humans? — Wang et al. (2025) (arXiv:2512.13281, 2025)

## What this evaluates

This benchmark probes the ability of video-language models and humans to distinguish real ASMR videos from AI-generated ones, evaluating perceptual realism and audio-visual consistency. It also measures how effectively video generation models can deceive video understanding models by producing indistinguishable synthetic content.

## Datasets

- **Video Reality Test** — total 100; splits: test (100)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly classified videos (real vs. fake) out of the total evaluated. For preference settings, it is the percentage of times the model correctly selects the real video from a real-fake pair.

## Input / output format

**Input**: Videos sampled at 8 equally interleaved frames, optionally accompanied by audio tracks and/or text prompts. Models receive either a single video for binary classification or a pair of videos (one real, one generated) for preference selection.

**Output**: Binary label ('real' or 'fake') for single-video evaluation, or selection of the real video from a pair for preference evaluation.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Models often latch onto watermarks (e.g., from Sora2) as a shortcut for detection rather than analyzing video content, artificially inflating accuracy.
- VLMs exhibit a strong conservative bias towards predicting 'real', leading to high false-negative rates for generated videos.
- Audio-visual misalignment in generated videos can disproportionately affect detection accuracy, especially when audio is included in the input modality.

## Evidence (verbatim from paper)

> We then ask VLM reviewers to identify the fake video and compute the accuracy of correctly detected fakes. A lower detection rate indicates that the generated videos are harder to distinguish from real ones, reflecting stronger generation capability.

## Citation

```bibtex
@misc{wang2025videorealitytest,
  title={Video Reality Test: Can AI-Generated ASMR Videos fool VLMs and Humans?},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2512.13281}
}
```

- arXiv: 2512.13281