Vbench Eval

---
name: vbench-eval
description: Evaluates the visual quality, semantic alignment, temporal consistency, and aesthetic fidelity of text-to-video generation models. It probes the model's ability to produce coherent, high-fidelity videos that match textual prompts across multiple perceptual and technical dimensions. Use when the user wants to benchmark on VBench, or asks about evaluating this task. Reports VBench Total Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.13343
  bibtex_key: wu2025taming
  confidence: high
---

# vbench-eval

> Taming Diffusion Transformer for Efficient Mobile Video Generation in Seconds — Wu et al. (2025) (arXiv:2507.13343, 2025)

## What this evaluates

Evaluates the visual quality, semantic alignment, temporal consistency, and aesthetic fidelity of text-to-video generation models. It probes the model's ability to produce coherent, high-fidelity videos that match textual prompts across multiple perceptual and technical dimensions.

## Datasets

- **VBench** — total 1000; splits: test (1000)

## Metrics

- `VBench Total Score` **(primary)** — range: percent
  - Composite score aggregating multiple sub-dimensions including Quality, Semantic, Aesthetics, Imaging, Object Class, Scene, Consistency, and Flickering. Computed by generating 5 videos per prompt across a 1K prompt set and averaging the VBench evaluator outputs.

## Input / output format

**Input**: Text prompt (encoded via T5 for server evaluation, CLIP for mobile evaluation)

**Output**: Video sequence (121 frames at 576x1024 for server; 49 frames at 384x512 for mobile), saved at 24 FPS for 5 seconds

## Scoring recipe

```python
prompts = load_vbench_1k_prompts()
all_scores = []
for prompt in prompts:
    videos = [generate_video(prompt, steps=4, cfg=None, seed=s) for s in range(5)]
    scores = vbench_evaluate(videos)
    all_scores.append(scores)
final_metrics = {k: mean([s[k] for s in all_scores]) for k in all_scores[0].keys()}
return final_metrics
```

## Common pitfalls

- Using classifier-free guidance (CFG) during evaluation, which contradicts the paper's 4-step distillation setup that explicitly runs without CFG.
- Evaluating with different video resolutions or frame counts than specified (576x1024x121 for server, 384x512x49 for mobile), as VBench scores are sensitive to resolution and temporal length.
- Averaging scores over fewer than 5 videos per prompt or using a different prompt subset than the official 1K VBench set, which can cause high variance (authors note ΔVBench score < ±0.2).

## Evidence (verbatim from paper)

> Our models are evaluated following the standard Vbench*(Huang et al., [2024])* setting, that is, we generate 5 videos for each prompt, and test the scores over the 1K prompt set. Both server and mobile-deployed models are step-distilled and evaluated with 4-step generation. The server model generates 121-frame horizontal videos at a resolution of 576	imes 1024, without classifier-free guidance.

## Citation

```bibtex
@misc{wu2025taming,
  title={Taming Diffusion Transformer for Efficient Mobile Video Generation in Seconds},
  author={Wu et al. (2025)},
  year={2025},
  note={arXiv:2507.13343}
}
```

- arXiv: 2507.13343