Urbanverse Eval

---
name: urbanverse-eval
description: Evaluates the fidelity of automated urban scene reconstruction from city-tour videos and the generalization capability of reinforcement learning navigation policies trained in these simulations. It measures how well generated scenes match real-world semantics and layouts, and how effectively policies transfer to unseen simulated and real-world environments. Use when the user wants to benchmark on KITTI-360, CraftBench, AutoBench, or asks about evaluating this task. Reports success_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.15018
  bibtex_key: liu2025urbanverse
  confidence: high
---

# urbanverse-eval

> UrbanVerse: Scaling Urban Simulation by Watching City-Tour Videos — Liu et al. (2025) (arXiv:2510.15018, 2025)

## What this evaluates

Evaluates the fidelity of automated urban scene reconstruction from city-tour videos and the generalization capability of reinforcement learning navigation policies trained in these simulations. It measures how well generated scenes match real-world semantics and layouts, and how effectively policies transfer to unseen simulated and real-world environments.

## Datasets

- **KITTI-360** — total 45; splits: test (45)
- **CraftBench** — total 10; splits: test (10)
- **AutoBench** — total 10; splits: test (10)

## Metrics

- `success_rate` **(primary)** — range: percent
  - Percentage of navigation attempts where the agent reaches the goal within a predefined distance tolerance without colliding. Calculated as (successful_trials / total_trials) * 100.
- `route_completion` — range: percent
  - Percentage of the total route distance successfully traversed by the agent before termination or collision.
- `collision_times` — range: count
  - Total number of collisions with obstacles or off-traversable surfaces during a navigation attempt.
- `mAP25` — range: [0, 1]
  - 3D object detection mean Average Precision at an IoU threshold of 0.25, used to evaluate scene reconstruction fidelity against ground-truth annotations.

## Input / output format

**Input**: RGB camera observations, relative position to the goal, and GPS-derived waypoints sampled every 5 meters projected into a local metric frame. No global map or semantic map is provided.

**Output**: Continuous or discrete navigation actions (e.g., velocity and steering commands) to move the agent toward the goal while avoiding collisions.

## Scoring recipe

```python
def compute_sr(predictions, gold):
    successful = 0
    for pred_pos, goal_info in zip(predictions, gold):
        dist = np.linalg.norm(np.array(pred_pos) - np.array(goal_info['target']))
        if dist <= goal_info['tolerance'] and not goal_info['collided']:
            successful += 1
    return (successful / len(gold)) * 100
```

## Common pitfalls

- Evaluating on training scenes instead of the specified unseen AutoBench/CraftBench test sets, which inflates generalization scores.
- Assuming sim-to-real transfer requires domain adaptation or fine-tuning; the protocol explicitly tests zero-shot deployment on physical robots.
- Confusing layout diversity (number of unique city-tour videos) with intra-layout diversity (number of digital cousins), which have distinct scaling effects on policy performance.

## Evidence (verbatim from paper)

> Policy performance is measured by success rate (SR), route completion (RC), and collision times (CT), with all evaluations conducted in unseen environments from AutoBench and CraftBench.

## Citation

```bibtex
@misc{liu2025urbanverse,
  title={UrbanVerse: Scaling Urban Simulation by Watching City-Tour Videos},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2510.15018}
}
```

- arXiv: 2510.15018