Wowbench Eval

---
name: wowbench-eval
description: Evaluates embodied world models on conditional video generation from an initial image and text instruction. It probes instruction understanding, long-horizon planning, physical/causal reasoning, and temporal consistency in robotic interaction scenarios. Use when the user wants to benchmark on WoWBench, or asks about evaluating this task. Reports Planning Score ($S_{plan}$), Overall Benchmark Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.22642
  bibtex_key: chi2025wow
  confidence: high
---

# wowbench-eval

> WoW: Towards a World omniscient World model Through Embodied Interaction — Chi et al. (2025) (arXiv:2509.22642, 2025)

## What this evaluates

Evaluates embodied world models on conditional video generation from an initial image and text instruction. It probes instruction understanding, long-horizon planning, physical/causal reasoning, and temporal consistency in robotic interaction scenarios.

## Datasets

- **WoWBench** — total ?; splits: test (-1)

## Metrics

- `FVD` — range: other
  - Fréchet Video Distance measuring distributional difference between generated and real video feature embeddings.
- `Mask-guided Regional Consistency` — range: [0, 1]
  - Cosine similarity of region-specific embeddings (DINOv3) computed separately for robot arm, manipulated object, and background across frames.
- `Sequence Match Score` — range: percent
  - LLM-evaluated score measuring the correct ordering of actions in the generated video relative to the instruction.
- `Execution Quality Score` — range: other
  - LLM-evaluated score on a 1–5 scale assessing how well the generated video fulfills the instruction.
- `Trajectory Consistency` — range: other
  - Combines Mean Euclidean Distance (MED), Dynamic Time Warping (DTW), and Fréchet Distance to compare tracked end-effector/object trajectories between generated and ground-truth videos.
- `Physical Common Sense Score` — range: other
  - 1–5 scale rating across six dimensions (object interaction, properties, temporal consistency, lighting, fluid dynamics, local anomalies) via a fine-tuned Qwen-2.5-VL evaluator.
- `Planning Score ($S_{plan}$)` **(primary)** — range: [0, 1]
  - $S_{plan} = (0.5 \times R_k + 0.5 \times R_s) \times P_k$, where $R_k$ is key-step recall, $R_s$ is sequential consistency, and $P_k$ is key-step precision, computed by comparing DAGs of predicted vs. ground-truth action plans.
- `Overall Benchmark Score` **(primary)** — range: other
  - Weighted arithmetic mean of per-metric desirability scores $s_{i,m} \in (0,100)$, where raw metrics are pre-scaled to [0,1] using absolute anchors, transformed via monotone mappings, and aggregated by group and overall.

## Input / output format

**Input**: Initial image and a natural language instruction specifying the desired action/task.

**Output**: A generated video sequence depicting the execution of the instruction from the initial state.

## Scoring recipe

```python
# 1. Compute raw metrics per model i and metric m: x_i,m
# 2. Pre-scale to [0,1] using absolute anchors L_m, U_m
if higher_is_better:
    x_hat = clip(x_i,m, L_m, U_m)
    x_hat = (x_hat - L_m) / (U_m - L_m)
else:
    x_hat = 1 - (clip(x_i,m, L_m, U_m) - L_m) / (U_m - L_m)
# 3. Apply monotone mapping f_m (e.g., power, logit, tanh) and scale to (0,100)
s_i,m = 100 * f_m(x_hat; theta_m)
# 4. Group averaging (uniform)
G_i,g = mean(s_i,m for m in group_g)
# 5. Overall aggregation (weighted mean)
O_i = sum(W_g * G_i,g for g in groups) / sum(W_g for g in groups)
return O_i
```

## Common pitfalls

- Relies on LLM/VLM evaluators (GPT-4o, Gemini, Qwen-2.5-VL) for instruction, planning, and physical common sense scores, which may introduce systematic bias or inconsistency compared to human judgment.
- The overall benchmark score uses dataset-tuned monotone mappings and absolute anchors, which can compress or distort raw metric differences and make cross-benchmark comparisons difficult.
- Trajectory consistency metrics depend on SAM2 tracking, which can degrade significantly under heavy occlusion or rapid motion, skewing physical reasoning scores.

## Evidence (verbatim from paper)

> We first parse the natural language instruction and ground-truth video into a ground-truth plan DAG, where nodes are atomic actions and edges represent dependencies. This representation flexibly handles non-unique but valid action orderings. We then compare the model-generated plan (which also uses the same approach to infer from the video) to the ground-truth DAG using three scores: Key-step Recall, Sequential Consistency, and Key-step Precision. The final planning score $S_{plan}$ integrates these aspects to reward both completeness and correctness: $S_{plan} = (0.5 \times R_k + 0.5 \times R_s) \times P_k$

## Citation

```bibtex
@misc{chi2025wow,
  title={WoW: Towards a World omniscient World model Through Embodied Interaction},
  author={Chi et al. (2025)},
  year={2025},
  note={arXiv:2509.22642}
}
```

- arXiv: 2509.22642