Omniworld Game Eval

---
name: omniworld-game-eval
description: Evaluates 3D geometric foundation models on monocular and video depth estimation, and tests camera-controlled video generation models on their ability to follow camera trajectories while maintaining video quality across diverse, dynamic environments. Use when the user wants to benchmark on OmniWorld-Game, or asks about evaluating this task. Reports FVD.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.12201
  bibtex_key: [REDACTED]
  confidence: high
---

# omniworld-game-eval

> OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling — Zhou et al. (2025) (arXiv:2509.12201, 2025)

## What this evaluates

Evaluates 3D geometric foundation models on monocular and video depth estimation, and tests camera-controlled video generation models on their ability to follow camera trajectories while maintaining video quality across diverse, dynamic environments.

## Datasets

- **OmniWorld-Game** — total ?; splits: test (-1); repo https://github.com/yangzhou24/OmniWorld

## Metrics

- `Abs Rel` — range: other
  - Absolute relative error between predicted and ground truth depth. Lower is better.
- `δ<1.25` — range: percent
  - Percentage of pixels where the ratio of predicted to ground truth depth is within 1.25. Higher is better.
- `TransErr` — range: other
  - Translation error between predicted and ground truth camera poses. Lower is better.
- `RotErr` — range: other
  - Rotation error between predicted and ground truth camera poses. Lower is better.
- `CamMC` — range: other
  - Camera Motion Consistency metric quantifying adherence to camera commands. Lower is better.
- `FVD` **(primary)** — range: other
  - Fréchet Video Distance measuring perceptual realism between generated and real video distributions. Lower is better.

## Input / output format

**Input**: Monocular depth: single RGB image. Video depth: video sequence. Video generation: text prompt or reference image plus camera trajectory parameters.

**Output**: Depth maps or video depth sequences for geometric tasks; generated video frames for video generation tasks.

## Scoring recipe

```python
abs_rel = np.mean(np.abs(pred_depth - gt_depth) / gt_depth)
delta1 = np.mean((pred_depth / gt_depth < 1.25) & (gt_depth / pred_depth < 1.25))
trans_err = np.linalg.norm(pred_cam_t - gt_cam_t)
rot_err = np.linalg.norm(R_log(pred_cam_R) - R_log(gt_cam_R))
cam_mc = compute_camera_motion_consistency(pred_traj, gt_traj)
fvd = frechet_video_distance(real_videos, generated_videos)
```

## Common pitfalls

- Models often fail to maintain temporal consistency over long sequences (up to 384 frames).
- Scale-and-shift alignment drastically changes depth metric scores compared to scale-only alignment.
- Video generation models struggle to simultaneously achieve high camera control accuracy and perceptual video quality.

## Evidence (verbatim from paper)

> Following CAMI2V (Zheng et al., [2024]), metrics include Camera Parameter Metrics (RotError, TransError, and CamMC) to quantify adherence to camera commands, and Fréchet Video Distance (FVD) (Unterthiner et al., [2018]) to assess perceptual realism.

## Citation

```bibtex
@misc{zhou2025omniworld,
  title={OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling},
  author={Zhou et al. (2025)},
  year={2025},
  note={arXiv:2509.12201}
}
```

- arXiv: 2509.12201