Structured3d Layout Eval

---
name: structured3d-layout-eval
description: Evaluates a model's ability to predict architectural elements (walls, doors, windows) and room layouts within indoor 3D scenes. It tests the model's capacity for structured scene understanding and spatial reasoning by comparing predicted layouts against ground-truth annotations. Use when the user wants to benchmark on Structured3D, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.07491
  bibtex_key: mao2025spatiallm
  confidence: high
---

# structured3d-layout-eval

> SpatialLM: Training Large Language Models for Structured Indoor Modeling — Mao et al. (2025) (arXiv:2506.07491, 2025)

## What this evaluates

Evaluates a model's ability to predict architectural elements (walls, doors, windows) and room layouts within indoor 3D scenes. It tests the model's capacity for structured scene understanding and spatial reasoning by comparing predicted layouts against ground-truth annotations.

## Datasets

- **Structured3D** — total 3500; splits: train (3000), val (250), test (250)

## Metrics

- `F1` **(primary)** — range: percent
  - F1 score computed using 2D Intersection over Union (IoU) between predicted and ground truth room layouts at a threshold of 0.25.

## Input / output format

**Input**: 3D point clouds of indoor scenes.

**Output**: Structured 3D layouts describing walls, doors, and windows (polygons and corners).

## Scoring recipe

```python
def compute_f1(preds, golds, iou_thresh=0.25):
    tp, fp, fn = 0, 0, 0
    for pred, gold in zip(preds, golds):
        iou = compute_iou_2d(pred, gold)
        if iou >= iou_thresh:
            tp += 1
        else:
            fp += 1
    fn = len(golds) - tp
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
```

## Common pitfalls

- RoomFormer converts 2D layouts to 3D by simple extrusion, which may not reflect true 3D geometry.
- Auto-regressive models do not output confidence scores, so standard mAP cannot be used; F1 is reported instead.

## Evidence (verbatim from paper)

> Following the setting of RoomFormer, we perform evaluation on the Structured3D benchmark, which contains 3,500 residential houses with diverse floorplans. We use the original data split of 3000/250/250 for training/validation/testing, respectively. Since auto-regressive models (i.e., SpatialLM and SceneScript) do not produce confidence scores, we report F1 scores instead of mean Average Precision (mAP), as suggested in[SceneScript].

## Citation

```bibtex
@misc{mao2025spatiallm,
  title={SpatialLM: Training Large Language Models for Structured Indoor Modeling},
  author={Mao et al. (2025)},
  year={2025},
  note={arXiv:2506.07491}
}
```

- arXiv: 2506.07491