Robotracer Spatial Eval

---
name: robotracer-spatial-eval
description: Evaluates vision-language models' ability to perform spatial understanding, metric measuring, 2D/3D referring, and multi-step visual tracing in cluttered environments. It probes geometric reasoning, depth estimation, and collision-free path planning for robotic manipulation. Use when the user wants to benchmark on CV-Bench, BLINK_val, RoboSpatial, Embspacial, Q-spatial, MSMU, Where2Place, RefSpatial-Bench, ShareRobot-Bench, VABench-V, TraceSpatial-Bench, RoboTwin, MMEtest, MMBenchdev, OK-VQA, POPE, or asks about evaluating this task. Reports Top-1 success rate (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.13660
  bibtex_key: zhou2025robotracer
  confidence: high
---

# robotracer-spatial-eval

> RoboTracer: Mastering Spatial Trace with Reasoning in Vision-Language Models for Robotics — Enshen Zhou et al. (2025) (arXiv:2512.13660, 2025)

## What this evaluates

Evaluates vision-language models' ability to perform spatial understanding, metric measuring, 2D/3D referring, and multi-step visual tracing in cluttered environments. It probes geometric reasoning, depth estimation, and collision-free path planning for robotic manipulation.

## Datasets

- **CV-Bench** — total ?; splits: test (-1)
- **BLINK_val** — total ?; splits: val (-1)
- **RoboSpatial** — total ?; splits: test (-1)
- **Embspacial** — total ?; splits: test (-1)
- **Q-spatial** — total ?; splits: test (-1)
- **MSMU** — total ?; splits: test (-1)
- **Where2Place** — total ?; splits: test (-1)
- **RefSpatial-Bench** — total ?; splits: test (-1)
- **ShareRobot-Bench** — total ?; splits: test (-1)
- **VABench-V** — total ?; splits: test (-1)
- **TraceSpatial-Bench** — total 100; splits: test (100)
- **RoboTwin** — total 19; splits: test (19)
- **MMEtest** — total ?; splits: test (-1)
- **MMBenchdev** — total ?; splits: dev (-1)
- **OK-VQA** — total ?; splits: test (-1)
- **POPE** — total ?; splits: test (-1)

## Metrics

- `Top-1 success rate (%)` **(primary)** — range: percent
  - Percentage of instances where the predicted spatial trace or referring point exactly matches the ground truth within a defined tolerance, with top-1 indicating the single best prediction.
- `Discrete Fréchet Distance (DFD)` — range: other
  - Measures the similarity between two trajectory curves (predicted vs. ground truth) in a discrete space. Lower values indicate better alignment.
- `Hausdorff Distance (HD)` — range: other
  - The maximum distance from a point in one set to the closest point in the other set. Lower values indicate tighter trace alignment.
- `Root Mean Square Error (RMSE)` — range: other
  - Square root of the average of squared differences between predicted and ground truth coordinates. Lower values indicate higher precision.

## Input / output format

**Input**: RGB images, optionally augmented with camera intrinsics and absolute depth maps, paired with natural language instructions describing the spatial task.

**Output**: Predicted 2D masks/points, 3D bounding boxes or coordinates, and trajectory paths; evaluated as success/failure flags or distance metrics.

## Scoring recipe

```python
def evaluate(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if is_within_tolerance(p, g))
    success_rate = (correct / len(predictions)) * 100
    dfd = discrete_frechet_distance(predictions, gold)
    hd = hausdorff_distance(predictions, gold)
    rmse = np.sqrt(np.mean((np.array(predictions) - np.array(gold))**2))
    multi_step_success = all(is_collision_free(p) and is_correct_start_end(p, g) for p, g in zip(predictions, gold))
    return success_rate, dfd, hd, rmse, multi_step_success
```

## Common pitfalls

- Models often produce 'floating' or colliding traces in 3D due to inaccurate depth estimation when only RGB inputs are provided.
- Evaluating 2D referring tasks without accounting for the decoupled point formulation or dimensionality reduction used in training can lead to unfair comparisons.
- Assuming success in 2D spatial referring implies success in 3D multi-step tracing, which requires explicit geometric reasoning and collision-free path validation.

## Evidence (verbatim from paper)

> Top-1/-2 success rate (%) are indicated by bold/underlined text. DFD/HD/RMSE are Discrete Fréchet Distance, Hausdorff Distance, Root Mean Square Error.

## Citation

```bibtex
@misc{zhou2025robotracer,
  title={RoboTracer: Mastering Spatial Trace with Reasoning in Vision-Language Models for Robotics},
  author={Enshen Zhou et al. (2025)},
  year={2025},
  note={arXiv:2512.13660}
}
```

- arXiv: 2512.13660