Scene Smith Eval

---
name: scene-smith-eval
description: Evaluates text-to-3D indoor scene generation systems on their ability to produce dense, physically plausible, and prompt-faithful environments. It probes both visual realism and simulation-readiness, measuring collision-free layouts and stable physics properties required for robotics policy testing. Use when the user wants to benchmark on SceneSmith Prompt Corpus, or asks about evaluating this task. Reports Realism Win%.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.09153
  bibtex_key: pfaff2026scenexmith
  confidence: high
---

# scene-smith-eval

> SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes — Pfaff et al. (2026) (arXiv:2602.09153, 2026)

## What this evaluates

Evaluates text-to-3D indoor scene generation systems on their ability to produce dense, physically plausible, and prompt-faithful environments. It probes both visual realism and simulation-readiness, measuring collision-free layouts and stable physics properties required for robotics policy testing.

## Datasets

- **SceneSmith Prompt Corpus** — total 210; splits: test (210)

## Metrics

- `Realism Win%` **(primary)** — range: percent
  - Percentage of pairwise human comparisons where the generated scene is preferred over the baseline for visual realism. Ties are excluded from the denominator.
- `Faithfulness Win%` — range: percent
  - Percentage of pairwise human comparisons where the generated scene better follows the text prompt requirements compared to the baseline.
- `CNT` — range: percent
  - VLM-based score measuring object count accuracy against the prompt.
- `ATR` — range: percent
  - VLM-based score measuring object attribute correctness.
- `OOR` — range: percent
  - VLM-based score measuring object-object relationship accuracy.
- `OAR` — range: percent
  - VLM-based score measuring object-architecture relationship accuracy.
- `ACC` — range: percent
  - VLM-based score measuring object accessibility.
- `NAV` — range: percent
  - VLM-based score measuring scene navigability.
- `OOB` — range: percent
  - VLM-based score measuring out-of-bounds violations (lower is better).
- `COL` — range: percent
  - Physics simulation collision rate measured using Drake (lower is better).
- `STB` — range: percent
  - Static equilibrium stability score measured using Drake (higher is better).

## Input / output format

**Input**: Natural language text prompts describing indoor scenes (room-level or house-level), optionally paired with baseline scene assets for direct comparison.

**Output**: 3D scene representations including asset placements, layouts, collision geometry, and physical properties, formatted for direct import into physics simulators (e.g., Drake).

## Scoring recipe

```python
def evaluate_scene(scene, prompt, baseline_scene=None):
    # Human pairwise comparison (if applicable)
    realism_win = human_prefers(scene, baseline_scene, criterion="realism")
    faith_win = human_prefers(scene, baseline_scene, criterion="prompt_following")

    # Automated VLM metrics
    metrics = {
        "CNT": vlm_score(scene, "object_count"),
        "ATR": vlm_score(scene, "attributes"),
        "OOR": vlm_score(scene, "object_relations"),
        "OAR": vlm_score(scene, "arch_relations"),
        "ACC": vlm_score(scene, "accessibility"),
        "NAV": vlm_score(scene, "navigability"),
        "OOB": vlm_score(scene, "out_of_bounds")
    }

    # Physics metrics (requires augmented collision/physics data)
    metrics["COL"] = drake_collision_rate(scene)
    metrics["STB"] = drake_static_equilibrium(scene)

    return {"human": {"realism": realism_win, "faithfulness": faith_win}, "auto": metrics}
```

## Common pitfalls

- VLM-based metrics (CNT, ATR, etc.) are prone to false positives/negatives and require careful validation or manual spot-checking.
- Baseline methods lack native simulation-ready outputs, so collision geometry and physical properties must be artificially augmented to compute COL and STB fairly.
- Human study statistical power is limited; ablations like NoCritic and NoAgentMemory showed non-significant effects that would require 6-18x more comparisons to detect reliably.

## Evidence (verbatim from paper)

> We use SceneEval*(Tam et al., [2025])* with the following metrics: CNT (object count), ATR (object attributes), OOR (object-object relationships), OAR (object-architecture relationships), ACC (accessibility), NAV (navigability), and OOB (out-of-bounds). We note that these VLM-based metrics have limitations including false positives and negatives (Appendix[L.2]). We add two physics metrics using Drake*([Tedrake et al., 2019])* to evaluate simulation-readiness: COL (collision rate) and STB (static equilibrium).

## Citation

```bibtex
@misc{pfaff2026scenexmith,
  title={SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes},
  author={Pfaff et al. (2026)},
  year={2026},
  note={arXiv:2602.09153}
}
```

- arXiv: 2602.09153