Drivinggen Eval

---
name: drivinggen-eval
description: Evaluates generative video world models for autonomous driving by jointly assessing visual realism, trajectory plausibility, temporal and agent-level consistency, and ego-conditioned motion controllability over a 100-frame prediction horizon. It benchmarks both general-purpose and driving-specific models to reveal trade-offs between photorealism and physical motion fidelity. Use when the user wants to benchmark on DrivingGen, or asks about evaluating this task. Reports Avg. Rank.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.01528
  bibtex_key: zhou2026drivinggen
  confidence: high
---

# drivinggen-eval

> DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving — Zhou et al. (2026) (arXiv:2601.01528, 2026)

## What this evaluates

Evaluates generative video world models for autonomous driving by jointly assessing visual realism, trajectory plausibility, temporal and agent-level consistency, and ego-conditioned motion controllability over a 100-frame prediction horizon. It benchmarks both general-purpose and driving-specific models to reveal trade-offs between photorealism and physical motion fidelity.

## Datasets

- **DrivingGen** — total ?; splits: test (-1)

## Metrics

- `Avg. Rank` **(primary)** — range: integer rank
  - Models are ranked on each individual metric (1 = best). The average of these ranks across all reported metrics serves as a quick summary score.
- `FVD` — range: arbitrary (lower is better)
  - Fréchet Video Distance; measures distribution-level similarity between generated and real video frames.
- `FTD` — range: arbitrary (lower is better)
  - Fréchet Trajectory Distance; measures distribution-level similarity of predicted trajectories.
- `ADE` — range: arbitrary (lower is better)
  - Average Displacement Error; measures trajectory alignment under ego-conditioning.
- `DTW` — range: arbitrary (lower is better)
  - Dynamic Time Warping distance; measures temporal alignment and adherence to commanded paths.

## Input / output format

**Input**: Initial driving scene video frame(s) and ego-vehicle trajectory commands/conditions.

**Output**: A generated video sequence of 100 frames conditioned on the input trajectory.

## Scoring recipe

```python
def evaluate(predictions, gold):
    # predictions: generated 100-frame videos
    # gold: ground truth videos & trajectory conditions
    scores = {}
    scores['FVD'] = frechet_video_distance(predictions, gold)
    scores['ADE'] = average_displacement_error(predictions, gold)
    scores['DTW'] = dtw_distance(predictions, gold)
    # Compute remaining metrics (FTD, SubjectiveQuality, VideoConsist, etc.)
    # Rank models per metric (1 = best)
    ranks = {m: rank_models(scores[m]) for m in scores}
    scores['Avg. Rank'] = mean([ranks[m] for m in ranks])
    return scores
```

## Common pitfalls

- Relying solely on distribution-level metrics like FVD/FTD can mask physically implausible behaviors (e.g., stop-go jitter, agent disappearance).
- High objective quality scores (e.g., low flicker) may coexist with poor subjective quality or unstable agent behavior.
- Treating the average rank as a definitive score rather than a quick summary metric.

## Evidence (verbatim from paper)

> Existing benchmarks often rely solely on distribution-level metrics such as FVD to evaluate generated driving videos. While useful for assessing overall distribution similarity, good FVD/FTD alone does not necessarily imply plausible driving—videos can appear distribution-close yet exhibit stop–go jitter, identity drift, or non-physical disappearances.

## Citation

```bibtex
@misc{zhou2026drivinggen,
  title={DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving},
  author={Zhou et al. (2026)},
  year={2026},
  note={arXiv:2601.01528}
}
```

- arXiv: 2601.01528