Vmbench Eval

---
name: vmbench-eval
description: Evaluates how well text-to-video models generate videos that align with human perceptual preferences across five motion dimensions: object integrity, motion smoothness, commonsense adherence, perceptible amplitude, and temporal coherence. Use when the user wants to benchmark on MMPG-set, or asks about evaluating this task. Reports Spearman correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.10076
  bibtex_key: ling2025vmbench
  confidence: high
---

# vmbench-eval

> VMBench: A Benchmark for Perception-Aligned Video Motion Generation — Ling et al. (2025) (arXiv:2503.10076, 2025)

## What this evaluates

Evaluates how well text-to-video models generate videos that align with human perceptual preferences across five motion dimensions: object integrity, motion smoothness, commonsense adherence, perceptible amplitude, and temporal coherence.

## Datasets

- **MMPG-set** — total 1050; splits: test (1050); repo https://github.com/GD-AIGC/VMBench

## Metrics

- `Spearman correlation` **(primary)** — range: percent
  - Spearman’s rank correlation coefficient ($\rho$) measures the monotonic relationship between two ranked variables. The paper reports $\rho \times 100$ to express alignment strength as a percentage.
- `Accuracy` — range: percent
  - Percentage of video pairs where the metric's preference matches the human preference. Ties are explicitly excluded to prioritize unambiguous decisions.

## Input / output format

**Input**: Text prompts (1,050 total) fed to text-to-video models to generate videos. For evaluation, videos are uniformly sampled at 2 frames per second and processed by rule-based algorithms or Multimodal Large Language Models (MLLMs).

**Output**: Per-dimension scores (1–5 Likert scale) from MLLMs or rule-based metrics, aggregated into PMM scores. Final outputs are Spearman correlation coefficients and alignment accuracy percentages against human expert ratings.

## Scoring recipe

```python
def compute_spearman(metric_scores, human_scores):
    metric_ranks = rank(metric_scores)
    human_ranks = rank(human_scores)
    rho = spearmanr(metric_ranks, human_ranks).correlation
    return rho * 100

def compute_accuracy(metric_scores, human_scores):
    pairs = list(itertools.combinations(range(len(metric_scores)), 2))
    matches = 0
    total = 0
    for i, j in pairs:
        if human_scores[i] == human_scores[j]: continue
        total += 1
        if (metric_scores[i] > metric_scores[j]) == (human_scores[i] > human_scores[j]):
            matches += 1
    return (matches / total) * 100
```

## Common pitfalls

- Ties in human preference pairs are explicitly excluded when computing alignment accuracy, which reduces the effective sample size and must be handled correctly.
- MLLM evaluation requires fixed frame sampling (2 fps) to manage computational limits; varying this rate changes the input distribution and breaks comparability.
- Rule-based metrics like RAFT and SSIM can yield negative correlations with human scores for certain dimensions (e.g., PAS), indicating counterintuitive alignment rather than better performance.

## Evidence (verbatim from paper)

> Spearman correlation is nonparametric and can effectively evaluate associations in datasets where variables may not follow a normal distribution. Unlike Pearson correlation, which captures linear relationships, Spearman correlation focuses on rank-based associations, making it more robust to outliers and suitable for ordinal data or scenarios with nonlinear dependencies. Alignment accuracy is quantified as the percentage of pairs where PMM preferences match human judgments, with ties excluded to prioritize unambiguous decisions.

## Citation

```bibtex
@misc{ling2025vmbench,
  title={VMBench: A Benchmark for Perception-Aligned Video Motion Generation},
  author={Ling et al. (2025)},
  year={2025},
  note={arXiv:2503.10076}
}
```

- arXiv: 2503.10076