Mmscan Visual Grounding Eval

---
name: mmscan-visual-grounding-eval
description: Evaluates a model's ability to localize specific 3D objects or regions within a large-scale scene based on complex natural language prompts. It probes spatial reasoning, attribute understanding, and multi-target grounding capabilities in 3D point cloud environments. Use when the user wants to benchmark on MMScan (3D Visual Grounding), or asks about evaluating this task. Reports gTop-k.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.09401
  bibtex_key: lyu2024mmscan
  confidence: high
---

# mmscan-visual-grounding-eval

> MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations — Lyu et al. (2024) (arXiv:2406.09401, 2024)

## What this evaluates

Evaluates a model's ability to localize specific 3D objects or regions within a large-scale scene based on complex natural language prompts. It probes spatial reasoning, attribute understanding, and multi-target grounding capabilities in 3D point cloud environments.

## Datasets

- **MMScan (3D Visual Grounding)** — total 1275586; splits: train (848867), val (217002), test (209717); repo https://github.com/OpenRobotLab/EmbodiedScan

## Metrics

- `gTop-k` **(primary)** — range: [0, 1]
  - Generalized Top-k score for multi-target grounding: gTop(k) = (1/N) * sum_{i=1}^{N} [Hit(min(i*k, M)) >= i]. N is the number of GT boxes, M is the number of predicted boxes, and Hit(X) counts how many of the top-X scored predictions match any GT box at a given IoU threshold.
- `AP_sample` — range: [0, 1]
  - Average Precision computed across all prompt samples, treating each sample as an independent detection task.
- `AP_box` — range: [0, 1]
  - Average Precision computed across all predicted bounding boxes, aggregating confidence scores across samples within the same subclass.
- `AR` — range: [0, 1]
  - Average Recall measuring the proportion of GT boxes successfully matched by predictions at a given IoU threshold.

## Input / output format

**Input**: 3D scene representation (reconstructed point clouds or RGB-D features) paired with a natural language prompt describing one or more target objects/regions (e.g., single-target attribute, object-object attribute, object-region).

**Output**: A ranked list of predicted 3D bounding boxes with associated confidence scores for each input prompt.

## Scoring recipe

```python
def compute_gTop_k(pred_boxes, gt_boxes, k, iou_thresh):
    N = len(gt_boxes)
    M = len(pred_boxes)
    score = 0.0
    for i in range(1, N + 1):
        top_x = min(i * k, M)
        top_preds = pred_boxes[:top_x]
        matches = sum(1 for p in top_preds if any(box_iou(p, g) >= iou_thresh for g in gt_boxes))
        if matches >= i:
            score += 1.0
    return score / N
```

## Common pitfalls

- AP_box aggregates confidence scores across all samples in a subclass, which can unfairly penalize models whose confidence distributions are not well-calibrated across different prompts, whereas AP_sample evaluates each prompt independently.
- Traditional Top-k metrics assume exactly one ground truth target; they fail when prompts require grounding multiple objects, necessitating the use of gTop-k.
- Evaluations in the paper report results on the validation set after training on only 20% of the training data to save compute, which may not reflect full-data training performance.

## Evidence (verbatim from paper)

> Therefore, we propose gTop-k (generalized Top-k score), which extends the Top-k metric to multi-target cases. Specifically, we define it as: |  | $gTop(k)\=\frac{1}{N}\sum_{i\=1}^{N}[Hit(min(ik,M))>\=i],$ |  | (1) | where $N$ is the number of ground truth (GT) boxes, $M$ is the number of predicted boxes, and Hit(X) represents the number of boxes with top-X scores that can match GT boxes (with IoU below a specific threshold).

## Citation

```bibtex
@misc{lyu2024mmscan,
  title={MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations},
  author={Lyu et al. (2024)},
  year={2024},
  note={arXiv:2406.09401}
}
```

- arXiv: 2406.09401