Video Bench Eval

---
name: video-bench-eval
description: Evaluates video generation models across two core dimensions: video-condition alignment (how well the generated video matches the text prompt in terms of objects, actions, colors, scenes, and overall consistency) and video quality (technical fidelity, aesthetics, temporal consistency, and motion quality). Use when the user wants to benchmark on Video-Bench, or asks about evaluating this task. Reports Video-text Consistency.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.04907
  bibtex_key: han2025videobench
  confidence: high
---

# video-bench-eval

> Video-Bench: Human-Aligned Video Generation Benchmark — Han et al. (2025) (arXiv:2504.04907, 2025)

## What this evaluates

Evaluates video generation models across two core dimensions: video-condition alignment (how well the generated video matches the text prompt in terms of objects, actions, colors, scenes, and overall consistency) and video quality (technical fidelity, aesthetics, temporal consistency, and motion quality).

## Datasets

- **Video-Bench** — total 419; splits: test (419); repo https://github.com/Video-Bench/Video-Bench.git

## Metrics

- `Object Class Consistency` — range: 1-3
  - Evaluates if objects match the text prompt, are identifiable, and move without abnormal deformation. Scored on a 3-point scale.
- `Action Consistency` — range: 1-3
  - Assesses if actions accurately reflect the text prompt, conforming to reality and cognition. Scored on a 3-point scale.
- `Color Consistency` — range: 1-3
  - Measures if object colors match the prompt and remain consistent without sudden changes. Scored on a 3-point scale.
- `Scene Consistency` — range: 1-3
  - Evaluates alignment of generated scene with prompt, ensuring logical arrangement and reality consistency. Scored on a 3-point scale.
- `Video-text Consistency` **(primary)** — range: 1-5
  - Assesses overall consistency between video and prompt across core elements (humans, actions, objects, scenes, style, etc.). Scored on a 5-point scale.
- `Imaging Quality` — range: 1-5
  - Focuses on technical frame quality, assessing noise, blur, overexposure, and artifacts. Scored on a 5-point scale.
- `Aesthetic Quality` — range: 1-5
  - Evaluates artistic appeal, composition, and visual coherence against human perceptual expectations. Scored on a 5-point scale.
- `Temporal Consistency` — range: 1-5
  - Assesses smooth transitions in visual features (color, brightness, texture) and semantic consistency of objects/scenes across frames. Scored on a 5-point scale.
- `Motion Quality` — range: 1-5
  - Evaluates motion rationality (adherence to physical laws) and motion amplitude (appropriateness of movement extent). Scored on a 5-point scale.

## Input / output format

**Input**: Text prompt and the corresponding generated video (each prompt sampled 3 times).

**Output**: Numerical score (1–3 or 1–5) per evaluation dimension, generated by an MLLM using few-shot scoring and chain-of-query.

## Scoring recipe

```python
def compute_scores(prompts, videos, dimensions):
    scores = {}
    for dim in dimensions:
        dim_scores = []
        for prompt in prompts:
            for video in sample(videos[prompt], k=3):
                score = mllm.evaluate(prompt, video, dim, scale=3 if dim in ALIGNMENT_DIMS else 5)
                dim_scores.append(score)
        scores[dim] = mean(dim_scores)
    return scores
```

## Common pitfalls

- Text-based guidelines are inherently ambiguous, requiring MLLM few-shot prompting to stabilize scoring.
- Video generation models have sampling randomness; failing to average over 3 samples per prompt introduces bias.
- Dimensions use mixed scales (3-point for alignment, 5-point for quality); aggregating them without normalization skews results.

## Evidence (verbatim from paper)

> Object Class Consistency This metric evaluates whether the objects presented in the video match those described in the text prompt. The focus is on whether the objects are generated correctly, are clearly identifiable, and whether their appearance and structure align with objective reality and human perception.

## Citation

```bibtex
@misc{han2025videobench,
  title={Video-Bench: Human-Aligned Video Generation Benchmark},
  author={Han et al. (2025)},
  year={2025},
  note={arXiv:2504.04907}
}
```

- arXiv: 2504.04907