Waver Bench Eval

---
name: waver-bench-eval
description: Evaluates a video generation model's capacity to synthesize physically coherent motion, high-fidelity visuals, and strict adherence to textual or image prompts across diverse scenarios. Use when the user wants to benchmark on Waver-Bench 1.0, Hermes Motion Testset, or asks about evaluating this task. Reports Human Preference Win Rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.15761
  bibtex_key: zhang2025waver
  confidence: high
---

# waver-bench-eval

> Waver: Wave Your Way to Lifelike Video Generation — Zhang et al. (2025) (arXiv:2508.15761, 2025)

## What this evaluates

Evaluates a video generation model's capacity to synthesize physically coherent motion, high-fidelity visuals, and strict adherence to textual or image prompts across diverse scenarios.

## Datasets

- **Waver-Bench 1.0** — total 304; splits: test (304)
- **Hermes Motion Testset** — total 96; splits: test (96)

## Metrics

- `Human Preference Win Rate` **(primary)** — range: [0, 1]
  - Proportion of human raters who prefer the model's generated video over a competitor's in side-by-side comparisons, calculated as (wins + 0.5 * ties) / total_votes. Aggregated across Motion Quality, Visual Quality, and Prompt Following dimensions.
- `motion quality` — range: percent
  - Human-rated category assessing physical realism and coherence of movement, including Action, Interaction, and Distortion sub-dimensions.

## Input / output format

**Input**: Text prompt (T2V) or reference image + text prompt (I2V)

**Output**: Generated video clip

## Scoring recipe

```python
def compute_win_rate(wins, losses, ties):
    total = wins + losses + ties
    if total == 0:
        return 0.0
    return (wins + 0.5 * ties) / total
```

## Common pitfalls

- Side-by-side human comparisons are susceptible to left/right position bias.
- Benchmarks are proprietary and not publicly released, limiting independent reproduction.
- Artificial Analysis Elo scores depend on user-submitted prompts, making rankings non-deterministic and sensitive to prompt distribution.

## Evidence (verbatim from paper)

> On these benchmarks, we conducted extensive human evaluations to assess our model’s performance against leading competitors (Veo3, Kling 2.0, and Wan 2.1 14B). In a side-by-side comparison format, human raters were shown videos generated by our model (Waver) and a competitor, then asked to judge which video was superior. The judgment was based on the comprehensive criteria defined above, including motion quality, visual quality, and prompt following.

## Citation

```bibtex
@misc{zhang2025waver,
  title={Waver: Wave Your Way to Lifelike Video Generation},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2508.15761}
}
```

- arXiv: 2508.15761