Target Bench Eval

---
name: target-bench-eval
description: Evaluates whether world models can perform mapless path planning toward semantic targets in real-world environments. It probes spatio-temporal consistency, trajectory accuracy, and the ability to reason about explicit versus implicit goals without prior map information. Use when the user wants to benchmark on Target-Bench, or asks about evaluating this task. Reports WO.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.17792
  bibtex_key: wang2025targetbench
  confidence: high
---

# target-bench-eval

> Target-Bench: Can World Models Achieve Mapless Path Planning with Semantic Targets? — Wang et al. (2025) (arXiv:2511.17792, 2025)

## What this evaluates

Evaluates whether world models can perform mapless path planning toward semantic targets in real-world environments. It probes spatio-temporal consistency, trajectory accuracy, and the ability to reason about explicit versus implicit goals without prior map information.

## Datasets

- **Target-Bench** — total 450; splits: train (325), test (-1)

## Metrics

- `FDE` — range: meters
  - Final Displacement Error: Euclidean distance between the predicted and ground-truth final positions. Exact threshold for Miss Rate is defined in Sec. 3.2.2.
- `ADE` — range: meters
  - Average Displacement Error: Mean Euclidean distance between predicted and ground-truth positions across the trajectory.
- `MR` — range: percent
  - Miss Rate: Percentage of trajectories where the final position exceeds a predefined error threshold.
- `SE` — range: [0, 1]
  - Soft Endpoint: Accuracy/probability score measuring how closely the predicted endpoint matches the target.
- `AC` — range: [0, 1]
  - Approach Consistency: Measures directional alignment between the predicted trajectory and the ground-truth path.
- `WO` **(primary)** — range: [0, 1]
  - Weighted Overall score: A composite metric aggregating FDE, ADE, MR, SE, and AC to provide a single performance indicator.

## Input / output format

**Input**: Robot-collected video sequences (720p/1080p, 5–10s duration) paired with explicit or implicit semantic target descriptions.

**Output**: Generated video sequences depicting the predicted trajectory toward the target.

## Scoring recipe

```python
def compute_metrics(pred_video, gt_video, tool='VGGT'):
    pred_traj = reconstruct_trajectory(pred_video, tool)
    gt_traj = extract_trajectory(gt_video)
    fde = distance(pred_traj[-1], gt_traj[-1])
    ade = mean([distance(p, g) for p, g in zip(pred_traj, gt_traj)])
    mr = 1 if fde > threshold else 0
    se = endpoint_similarity(pred_traj[-1], gt_traj[-1])
    ac = directional_alignment(pred_traj, gt_traj)
    wo = weighted_average([fde, ade, mr, se, ac])
    return {'FDE': fde, 'ADE': ade, 'MR': mr, 'SE': se, 'AC': ac, 'WO': wo}
```

## Common pitfalls

- Scores vary significantly depending on the spatio-temporal reconstruction tool used (VGGT vs. SpaTracker vs. ViPE).
- Performance is highly sensitive to the planning horizon length; shorter horizons (4s) consistently yield higher scores than longer ones (8s).
- Explicit and implicit targets produce similar results, but model-specific fluctuations can mislead comparisons if not averaged across both.

## Evidence (verbatim from paper)

> Among all evaluated off-the-shelf models, Wan2.2-Flash achieves the best overall performance with a weighted overall score of 0.299. Specifically, it obtains the lowest errors in FDE (1.362m), ADE (1.005m), and MR (38.75%), while achieving the highest SE (0.292).

## Citation

```bibtex
@misc{wang2025targetbench,
  title={Target-Bench: Can World Models Achieve Mapless Path Planning with Semantic Targets?},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2511.17792}
}
```

- arXiv: 2511.17792