Openscan Eval

---
name: openscan-eval
description: Evaluates 3D vision models' ability to perform open-vocabulary scene understanding by querying for fine-grained object attributes (e.g., material, affordance, synonym) rather than standard object classes. It measures both 3D instance segmentation and 3D semantic segmentation capabilities on attribute-based queries across eight linguistic aspects. Use when the user wants to benchmark on OpenScan, or asks about evaluating this task. Reports AP, mIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.11030
  bibtex_key: zhao2024openscan
  confidence: high
---

# openscan-eval

> OpenScan: A Benchmark for Generalized Open-Vocabulary 3D Scene Understanding — Zhao et al. (2024) (arXiv:2408.11030, 2024)

## What this evaluates

Evaluates 3D vision models' ability to perform open-vocabulary scene understanding by querying for fine-grained object attributes (e.g., material, affordance, synonym) rather than standard object classes. It measures both 3D instance segmentation and 3D semantic segmentation capabilities on attribute-based queries across eight linguistic aspects.

## Datasets

- **OpenScan** — total ?; splits: validation (-1)

## Metrics

- `AP` **(primary)** — range: percent
  - Average Precision computed across multiple IoU thresholds (typically 0.25 to 0.95), averaged over all queried attributes.
- `AP50` — range: percent
  - Average Precision calculated at a single IoU threshold of 0.50.
- `AP25` — range: percent
  - Average Precision calculated at a single IoU threshold of 0.25.
- `mIoU` **(primary)** — range: percent
  - Mean Intersection over Union across all queried attributes, computed as the average of IoU scores for each attribute.
- `mAcc` — range: percent
  - Mean Accuracy across all queried attributes, representing the average proportion of correctly classified points/voxels per attribute.

## Input / output format

**Input**: 3D scene data (point cloud or mesh) with aligned 2D RGB images, paired with a natural language query specifying an object attribute (e.g., 'made of wood', 'keep food cold').

**Output**: 3D instance masks or per-point/voxel semantic labels identifying objects in the scene that match the queried attribute.

## Scoring recipe

```python
def compute_ap(preds, gts, iou_thresh):
    matches = []
    for gt in gts:
        best_iou = max(iou(p, gt) for p in preds)
        matches.append(1.0 if best_iou >= iou_thresh else 0.0)
    return average_precision(matches)

def compute_miou(preds, gts):
    return sum(iou(p, gt) for p, gt in zip(preds, gts)) / len(gts)
```

## Common pitfalls

- Models are evaluated strictly zero-shot; fine-tuning on OpenScan is prohibited.
- Query formulation drastically affects results: full sentence templates yield significantly higher scores than single keywords due to VLM limitations.
- Performance varies widely across the eight linguistic aspects; reporting only the mean masks critical capability gaps.

## Evidence (verbatim from paper)

> We evaluate OpenScene(Peng et al. [2023]), PLA(Ding et al. [2023]), and RegionPLC(Yang et al. [2024]), reporting the average score of all attributes in our OpenScan and that of all object classes in ScanNet(Dai et al. [2017]). Table[4] shows that although these OV-3D models perform well in recognizing object classes, they exhibit poor performances on linguistic aspects with low mIoU and mAcc metrics.

## Citation

```bibtex
@misc{zhao2024openscan,
  title={OpenScan: A Benchmark for Generalized Open-Vocabulary 3D Scene Understanding},
  author={Zhao et al. (2024)},
  year={2024},
  note={arXiv:2408.11030}
}
```

- arXiv: 2408.11030