Worldlens Eval

---
name: worldlens-eval
description: Evaluates driving world models across five dimensions: generation quality, 3D/4D reconstruction coherence, action-following capability in closed-loop simulation, downstream perception task utility, and alignment with human preference. It probes geometric consistency, physical plausibility, and functional reliability of synthesized driving scenes. Use when the user wants to benchmark on WorldLens, or asks about evaluating this task. Reports Route Completion (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.10958
  bibtex_key: liang2025worldlens
  confidence: medium
---

# worldlens-eval

> WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World — Liang et al. (2025) (arXiv:2512.10958, 2025)

## What this evaluates

Evaluates driving world models across five dimensions: generation quality, 3D/4D reconstruction coherence, action-following capability in closed-loop simulation, downstream perception task utility, and alignment with human preference. It probes geometric consistency, physical plausibility, and functional reliability of synthesized driving scenes.

## Datasets

- **WorldLens** — total ?; splits: test (-1); repo https://github.com/worldbench/WorldLens

## Metrics

- `FID (Subject Fidelity)` — range: [0, inf)
  - Fréchet Inception Distance measuring the distance between feature distributions of real and generated driving frames. Lower is better.
- `Perceptual Discrepancy` — range: [0, 1] or dB
  - Likely LPIPS or PSNR-based metric quantifying visual similarity between generated and ground-truth frames. Lower is better.
- `Geometric Discrepancy / Photometric Error` — range: [0, inf)
  - Errors computed after reconstructing generated videos into 4D Gaussian Fields. Measures floaters, structural instability, and pixel-level reconstruction loss.
- `Route Completion (%)` **(primary)** — range: [0, 100] percent
  - Percentage of simulation episodes where a closed-loop planner successfully navigates the route without collisions or off-road drift.
- `Human Preference Score` — range: [1, 10]
  - Subjective ratings on a 1–10 scale across World Realism, Physical Plausibility, 3D & 4D Consistency, and Behavioral Safety.

## Input / output format

**Input**: Synthesized video sequences or frame-by-frame predictions from driving world models, typically conditioned on ground-truth frames, ego-vehicle states, or action commands.

**Output**: Quantitative metric scores per evaluation aspect, plus human preference ratings on a 1–10 scale and textual rationales for the WorldLens-Agent auto-evaluator.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    scores = {}
    scores['FID'] = frechet_inception_distance(predictions, gold)
    scores['Perceptual'] = perceptual_similarity(predictions, gold)
    scores['Geo_Error'] = reconstruct_4d_gaussians(predictions).compute_error(gold)
    scores['Route_Completion'] = closed_loop_planner(predictions).success_rate()
    scores['Human_Pref'] = human_annotators.score(predictions, scale=10)
    return scores
```

## Common pitfalls

- Relying solely on perceptual metrics (e.g., FID/LPIPS) without verifying geometric or physical consistency, as texture-rich models often violate physics.
- Evaluating only in open-loop settings; closed-loop simulation exposes severe planner failures (collisions, off-road drift) that open-loop metrics miss.
- Assuming high perceptual quality guarantees downstream task usability; multi-domain training can actually hinder task-specific adaptation.

## Evidence (verbatim from paper)

> As summarized in Table [1], all existing models remain notably below the ‘Empirical Max’, indicating substantial room for improving the visual and temporal realism of driving world models. Although DiST-4D achieves the lowest Perceptual Discrepancy, it underperforms OpenDWM in Subject Fidelity and View Consistency, demonstrating that perceptual metrics alone are insufficient for assessing physically coherent scene generation.

## Citation

```bibtex
@misc{liang2025worldlens,
  title={WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World},
  author={Liang et al. (2025)},
  year={2025},
  note={arXiv:2512.10958}
}
```

- arXiv: 2512.10958