Worldmark Eval

---
name: worldmark-eval
description: This benchmark evaluates interactive Image-to-Video world models by measuring their ability to generate temporally coherent videos in response to standardized action commands. It probes three core capabilities: visual fidelity, precise camera/object control alignment, and long-horizon world consistency across different perspectives and visual styles. Use when the user wants to benchmark on WorldMark Image Suite, or asks about evaluating this task. Reports Aesthetic Quality.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.21686
  bibtex_key: xu2026worldmark
  confidence: high
---

# worldmark-eval

> WorldMark: A Unified Benchmark Suite for Interactive Video World Models — Xu et al. (2026) (arXiv:2604.21686, 2026)

## What this evaluates

This benchmark evaluates interactive Image-to-Video world models by measuring their ability to generate temporally coherent videos in response to standardized action commands. It probes three core capabilities: visual fidelity, precise camera/object control alignment, and long-horizon world consistency across different perspectives and visual styles.

## Datasets

- **WorldMark Image Suite** — total 50; splits: First-Person Real (-1), First-Person Stylized (-1), Third-Person Real (-1), Third-Person Stylized (-1)

## Metrics

- `Aesthetic Quality` **(primary)** — range: other
  - Automated score measuring visual appeal and artistic merit of generated frames. Scale unspecified in text.
- `Imaging Quality` — range: other
  - Automated score measuring technical rendering fidelity and clarity of generated frames.
- `Translation Error` — range: other
  - Geometric deviation between intended and actual camera/object translation in the generated video.
- `Rotation Error` — range: other
  - Angular deviation between intended and actual camera/object rotation in the generated video.
- `Reprojection Error` — range: other
  - Error in projecting 3D scene structure back to 2D frames, measuring spatial consistency.
- `State Consistency` — range: other
  - Score measuring temporal stability of object states and physics across the video sequence.
- `Content Consistency` — range: other
  - Score measuring preservation of scene elements and objects without hallucination or disappearance.
- `Style Consistency` — range: other
  - Score measuring maintenance of the original visual style and artistic tone throughout the generation.

## Input / output format

**Input**: A reference image (first- or third-person view) paired with a sequence of 5 standardized WASD-style action commands, translated via a unified interface into the model's native control format (e.g., captions, pose vectors, or gamepad signals).

**Output**: A generated video sequence corresponding to the input action sequence, rendered in the specified perspective and style.

## Scoring recipe

```python
def evaluate_model(model, test_suite):
    metrics = {'Aesthetic': [], 'Imaging': [], 'TransErr': [], 'RotErr': [], 'ReprojErr': [], 'StateCon': [], 'ContentCon': [], 'StyleCon': []}
    for img, actions in test_suite:
        video = model.generate(img, actions)
        metrics['Aesthetic'].append(compute_aesthetic_score(video))
        metrics['Imaging'].append(compute_imaging_score(video))
        metrics['TransErr'].append(compute_translation_error(video, actions))
        metrics['RotErr'].append(compute_rotation_error(video, actions))
        metrics['ReprojErr'].append(compute_reprojection_error(video))
        metrics['StateCon'].append(compute_state_consistency(video))
        metrics['ContentCon'].append(compute_content_consistency(video))
        metrics['StyleCon'].append(compute_style_consistency(video))
    return {k: mean(v) for k, v in metrics.items()}
```

## Common pitfalls

- Models require a unified action-mapping layer to translate standardized commands into native interfaces; skipping this step breaks cross-model comparability.
- Third-person evaluation is only valid for models natively supporting that perspective (Matrix-Game 2.0, HY-World 1.5, Genie 3); forcing unsupported models yields invalid results.
- High visual quality does not guarantee world consistency or control alignment; these dimensions are largely uncorrelated and must be evaluated independently.

## Evidence (verbatim from paper)

> Regarding control alignment, the results indicate a divergence in model capabilities. HY-Game exhibits robust translation control, achieving the lowest Translation Error in both Real (0.159) and Stylized (0.116) scenarios, as well as the lowest Rotation Error (0.932) in the Stylized setting.

## Citation

```bibtex
@misc{xu2026worldmark,
  title={WorldMark: A Unified Benchmark Suite for Interactive Video World Models},
  author={Xu et al. (2026)},
  year={2026},
  note={arXiv:2604.21686}
}
```

- arXiv: 2604.21686