Vane Bench Eval

---
name: vane-bench-eval
description: Evaluates the ability of video-language models to detect subtle, rapid, and contextually nuanced anomalies in both real-world surveillance footage and high-fidelity AI-generated videos. It probes fine-grained temporal reasoning, visual grounding, and robustness to synthetic artifacts through a multiple-choice question-answering format. Use when the user wants to benchmark on VANE-Bench, or asks about evaluating this task. Reports MC-Video QA accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.10326
  bibtex_key: gani2024vanebench
  confidence: high
---

# vane-bench-eval

> VANE-Bench: Video Anomaly Evaluation Benchmark for Conversational LMMs — Gani et al. (2024) (arXiv:2406.10326, 2024)

## What this evaluates

Evaluates the ability of video-language models to detect subtle, rapid, and contextually nuanced anomalies in both real-world surveillance footage and high-fidelity AI-generated videos. It probes fine-grained temporal reasoning, visual grounding, and robustness to synthetic artifacts through a multiple-choice question-answering format.

## Datasets

- **VANE-Bench** — total 325; splits: test (325); repo https://github.com/rohit901/VANE-Bench

## Metrics

- `MC-Video QA accuracy` **(primary)** — range: [0, 1]
  - Standard multiple-choice accuracy: the fraction of questions where the model's selected option exactly matches the ground-truth answer. Calculated as correct predictions divided by total questions.

## Input / output format

**Input**: A video clip accompanied by a natural language question and a set of multiple-choice options.

**Output**: The model must select exactly one option from the provided multiple-choice list.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, golds):
    if pred == gold:
        correct += 1
accuracy = correct / len(golds)
```

## Common pitfalls

- Anomalies are often subtle and rapid, requiring fine-grained temporal reasoning rather than relying on static frame-level inspection.
- AI-generated videos (e.g., SORA) have high visual fidelity, making anomalies extremely difficult to detect and potentially causing models to hallucinate or guess.
- Multiple-choice options may contain plausible distractors, so high scores can sometimes reflect guessing or textual priors rather than true visual understanding.

## Evidence (verbatim from paper)

> We adapted the VAD problem into a Multiple-Choice Video Question Answering (MC-Video QA) task to facilitate the evaluation of LMMs, allowing for a more granular assessment of their video content understanding.

## Citation

```bibtex
@misc{gani2024vanebench,
  title={VANE-Bench: Video Anomaly Evaluation Benchmark for Conversational LMMs},
  author={Gani et al. (2024)},
  year={2024},
  note={arXiv:2406.10326}
}
```

- arXiv: 2406.10326