Stanford 2d 3d S Eval

---
name: stanford-2d-3d-s-eval
description: Evaluates 2D-to-3D semantic transfer pipelines for indoor scene understanding, focusing on per-point labeling accuracy for structural and furniture classes, and detection sensitivity for novel safety-critical objects in public safety contexts. Use when the user wants to benchmark on Stanford 2D-3D-S*, or asks about evaluating this task. Reports per-point accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.23095
  bibtex_key: dimopoulos2026insight
  confidence: high
---

# stanford-2d-3d-s-eval

> INSIGHT: Indoor Scene Intelligence from Geometric-Semantic Hierarchy Transfer for Public~Safety — Dimopoulos et al. (2026) (arXiv:2604.23095, 2026)

## What this evaluates

Evaluates 2D-to-3D semantic transfer pipelines for indoor scene understanding, focusing on per-point labeling accuracy for structural and furniture classes, and detection sensitivity for novel safety-critical objects in public safety contexts.

## Datasets

- **Stanford 2D-3D-S*** — total ?; splits: test (-1)

## Metrics

- `per-point accuracy` **(primary)** — range: [0, 1]
  - For each pipeline point whose semantic label maps to one of the 7 overlapping classes, the nearest Stanford GT point (by Euclidean distance in 3D) supplies the reference label. A point is correct if the two mapped labels agree. Per-area accuracy is a_k = n_k^correct / n_k^total, and overall accuracy is the area-weighted mean A = sum_k w_k a_k, where w_k = n_k^total / sum_j n_j^total.
- `detection sensitivity` — range: other
  - Raw count of detected instances per novel safety-critical class across all evaluated areas. No ground truth exists; reported as detection volume to compare pipeline performance.
- `fragmentation ratio` — range: other
  - Pipeline instances divided by reference count (GT instances for overlapping classes, code-derived plausibility cap K for novel classes) at a merge distance of 0.5 m.

## Input / output format

**Input**: RGB+OpenEXR images and corresponding 3D point clouds from indoor scenes; pipeline receives 2D images to generate semantic labels and 3D points, which are then evaluated against Stanford 2D-3D-S ground truth point clouds.

**Output**: Per-point semantic labels mapped to a unified taxonomy (7 overlapping classes + 15 novel safety classes); detection counts for novel classes; instance coordinates for overlap and fragmentation analysis.

## Scoring recipe

```python
def compute_per_point_accuracy(pred_points, gt_points, valid_classes):
    correct = 0
    total = 0
    for p in pred_points:
        if p.label not in valid_classes:
            continue
        gt_label = find_nearest_gt_label(p.coords, gt_points, metric='euclidean')
        if p.label == gt_label:
            correct += 1
        total += 1
    area_acc = correct / total if total > 0 else 0.0
    return area_acc
```

## Common pitfalls

- Structural classes (ceiling, floor, wall) show artificially low per-point accuracy due to single-instance-per-area compression and boundary overlaps with adjacent GT labels, not spatial misalignment.
- Novel safety-class detection counts are raw volumes without ground truth verification; they serve as proxy signals rather than strict precision/recall metrics.
- Inter-pipeline agreement is measured at a fixed 1 m centroid radius, which may over- or under-estimate true overlap for elongated or fragmented objects.

## Evidence (verbatim from paper)

> Per-point accuracy is computed over the 7 classes shared between the pipeline taxonomy and Stanford 2D-3D-S after class mapping (table+chair+sofa+bookcase → furniture; beam → column; six classes map directly). For each pipeline point whose semantic label maps to one of these 7 classes, the nearest Stanford GT point (by Euclidean distance in 3D) supplies the reference label. A point is correct if the two mapped labels agree. Points carrying pipeline-only labels (the 15 novel safety classes) are excluded; Stanford-only classes (board, clutter) have no pipeline equivalent and are excluded from the denominator. Per-area accuracy is $a_{k}\=n_{k}^{\text{correct}}/n_{k}^{\text{total}}$ and the reported overall accuracy is the area-weighted mean $A\=\sum_{k}w_{k}\,a_{k}$, where $w_{k}\=n_{k}^{\text{total}}\big/\sum_{j}n_{j}^{\text{total}}$.

## Citation

```bibtex
@misc{dimopoulos2026insight,
  title={INSIGHT: Indoor Scene Intelligence from Geometric-Semantic Hierarchy Transfer for Public~Safety},
  author={Dimopoulos et al. (2026)},
  year={2026},
  note={arXiv:2604.23095}
}
```

- arXiv: 2604.23095