Scanreason Eval

---
name: scanreason-eval
description: Evaluates a model's ability to perform 3D visual grounding by jointly reasoning about implicit human instructions and localizing target objects in 3D scenes. It probes spatial, functional, logical, emotional, and safety-related reasoning capabilities alongside precise 3D bounding box localization. Use when the user wants to benchmark on ScanReason, or asks about evaluating this task. Reports matching score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.01525
  bibtex_key: [REDACTED]
  confidence: medium
---

# scanreason-eval

> ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilities — Zhu et al. (2024) (arXiv:2407.01525, 2024)

## What this evaluates

Evaluates a model's ability to perform 3D visual grounding by jointly reasoning about implicit human instructions and localizing target objects in 3D scenes. It probes spatial, functional, logical, emotional, and safety-related reasoning capabilities alongside precise 3D bounding box localization.

## Datasets

- **ScanReason** — total 10000; splits: test (-1)

## Metrics

- `matching score` **(primary)** — range: [0, 1]
  - Not explicitly defined in the provided Method section. The prediction head outputs a matching score used to evaluate localization quality and filter successful object locations in the Chain-of-Grounding mechanism.

## Input / output format

**Input**: 3D scene representation (point cloud features or multi-view 2D images back-projected to 3D) combined with a natural language instruction or question.

**Output**: Text answer and/or 3D bounding box prediction(s). For grounding tasks, the model outputs a special <LOC> token followed by the predicted 3D box coordinates.

## Scoring recipe

```python
def evaluate(predictions, gold):
    # Compute IoU between predicted and ground-truth 3D boxes
    iou = compute_iou(predictions.bbox, gold.bbox)
    # Use matching score or confidence threshold for filtering
    if predictions.confidence > threshold:
        recall_at_k = calculate_recall(iou, k=10)
    return recall_at_k
```

## Common pitfalls

- Models may struggle with implicit instructions that require multi-step spatial or functional reasoning rather than direct object matching.
- The Chain-of-Grounding mechanism requires careful threshold tuning for confidence scores to avoid error propagation across interleaved reasoning and grounding steps.

## Evidence (verbatim from paper)

> Finally, the prediction head takes the updated object queries as input and predicts the final 3D locations and matching score.

## Citation

```bibtex
@misc{zhu2024scanreason,
  title={ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilities},
  author={Zhu et al. (2024)},
  year={2024},
  note={arXiv:2407.01525}
}
```

- arXiv: 2407.01525