Robo Refer Spatial Eval

---
name: robo-refer-spatial-eval
description: Evaluates vision-language models' ability to perform single-step and multi-step spatial understanding and referring tasks in robotics contexts. It probes capabilities like 2D/3D relation reasoning, depth perception, and complex compositional spatial constraints in cluttered scenes. Use when the user wants to benchmark on CV-Bench, BLINK, RoboSpatial, RefSpatial-Bench, RefCOCO, or asks about evaluating this task. Reports Top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.04308
  bibtex_key: zhou2025roborefer
  confidence: high
---

# robo-refer-spatial-eval

> RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics — Zhou et al. (2025) (arXiv:2506.04308, 2025)

## What this evaluates

Evaluates vision-language models' ability to perform single-step and multi-step spatial understanding and referring tasks in robotics contexts. It probes capabilities like 2D/3D relation reasoning, depth perception, and complex compositional spatial constraints in cluttered scenes.

## Datasets

- **CV-Bench** — total ?; splits: (unstated)
- **BLINK** — total ?; splits: val (-1)
- **RoboSpatial** — total ?; splits: (unstated)
- **RefSpatial-Bench** — total 200; splits: Location (100), Placement (100)
- **RefCOCO** — total ?; splits: val (-1), testA (-1), testB (-1)

## Metrics

- `Top-1 accuracy` **(primary)** — range: [0, 1]
  - Calculated as the proportion of correctly predicted instances out of the total. For referring tasks, a prediction is correct if the predicted point lies within the ground-truth mask or bounding box.
- `Success rate` — range: percent
  - Average success rate of predicted points within the ground-truth mask or bounding box, reported as a percentage.

## Input / output format

**Input**: RGB or RGB-D image paired with a natural language spatial query or instruction.

**Output**: Predicted 2D point coordinates (x, y) or bounding box, or a multiple-choice label.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if isinstance(gold, dict) and 'mask' in gold:
            if point_in_mask(pred, gold['mask']): correct += 1
        elif isinstance(gold, dict) and 'bbox' in gold:
            if point_in_bbox(pred, gold['bbox']): correct += 1
        else:
            if pred == gold: correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Evaluating point predictions against bounding boxes using IoU instead of point-in-box containment.
- Ignoring the impact of RGB vs. RGB-D inputs, which drastically changes 3D spatial benchmark performance.
- Assuming single-step accuracy generalizes to multi-step reasoning, which requires sequential refinement.

## Evidence (verbatim from paper)

> For metrics, we report the average success rate of predicted points within the mask. ... Since our model predicts a single point, we deem a prediction correct if the point lies within the ground-truth bounding box.

## Citation

```bibtex
@misc{zhou2025roborefer,
  title={RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics},
  author={Zhou et al. (2025)},
  year={2025},
  note={arXiv:2506.04308}
}
```

- arXiv: 2506.04308