Videodpo Eval

---
name: videodpo-eval
description: Evaluates text-to-video diffusion models on visual quality and semantic alignment with input prompts. It measures intra-frame fidelity, aesthetic appeal, and inter-frame temporal consistency using automated benchmarks and human-preference predictors. Use when the user wants to benchmark on VBench, or asks about evaluating this task. Reports VBench.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.14167
  bibtex_key: liu2024videodpo
  confidence: high
---

# videodpo-eval

> VideoDPO: Omni-Preference Alignment for Video Diffusion Generation — Liu et al. (2024) (arXiv:2412.14167, 2024)

## What this evaluates

Evaluates text-to-video diffusion models on visual quality and semantic alignment with input prompts. It measures intra-frame fidelity, aesthetic appeal, and inter-frame temporal consistency using automated benchmarks and human-preference predictors.

## Datasets

- **VBench** — total ?; splits: test (-1)

## Metrics

- `VBench` **(primary)** — range: percent
  - Assesses quality and semantic alignment across 16 hierarchical dimensions, providing fine-grained evaluation scores for video generation.
- `HPS (V)` — range: other
  - Predicts scores of human preference for generated videos, trained on large-scale human preference datasets.
- `PickScore` — range: other
  - Predicts scores of human preference for generated videos, trained on large-scale human preference datasets.

## Input / output format

**Input**: Text prompt conditioning a video diffusion model.

**Output**: Generated video.

## Scoring recipe

```python
def evaluate_model(model, prompts):
    scores = {'VBench': [], 'HPS_V': [], 'PickScore': []}
    for prompt in prompts:
        videos = model.generate(prompt, num_samples=4)
        for v in videos:
            scores['VBench'].append(vbench_score(v))
            scores['HPS_V'].append(hpsv_score(v))
            scores['PickScore'].append(pickscore_score(v))
    return {k: sum(v)/len(v) for k, v in scores.items()}
```

## Common pitfalls

- HPS(V) scores can be insensitive to certain models (e.g., CogVideo) due to low generation quality or early release dates.
- Metrics are fully automated and may not perfectly correlate with human subjective judgments of video quality or alignment.

## Evidence (verbatim from paper)

> To evaluate our method and the baselines, we use the following metrics: VBench, a widely recognized benchmark that assesses both quality and semantic alignment in video generation across 16 hierarchical dimensions, providing fine-grained evaluation. HPS (V) and PickScore are also included as metrics; both are trained on large-scale human preference datasets and are designed to predict scores of human preference for generated videos.

## Citation

```bibtex
@misc{liu2024videodpo,
  title={VideoDPO: Omni-Preference Alignment for Video Diffusion Generation},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2412.14167}
}
```

- arXiv: 2412.14167