Xr Scene Eval

---
name: xr-scene-eval
description: Probes large-scale 3D scene understanding by evaluating a model's ability to reason across multiple rooms, locate specific objects, generate embodied task plans, and produce detailed captions in complex, high-density environments. It specifically tests spatial awareness, contextual inference, and fine-grained detail retention beyond single-room benchmarks. Use when the user wants to benchmark on XR-Scene, or asks about evaluating this task. Reports CIDEr.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.01292
  bibtex_key: zhi2024lscenellm
  confidence: high
---

# xr-scene-eval

> LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences — Zhi et al. (2024) (arXiv:2412.01292, 2024)

## What this evaluates

Probes large-scale 3D scene understanding by evaluating a model's ability to reason across multiple rooms, locate specific objects, generate embodied task plans, and produce detailed captions in complex, high-density environments. It specifically tests spatial awareness, contextual inference, and fine-grained detail retention beyond single-room benchmarks.

## Datasets

- **XR-Scene** — total 1000; splits: test (-1); repo https://github.com/Hoyyyaard/LSceneLLM

## Metrics

- `ROUGE` — range: [0, 1]
  - n-gram overlap between generated and reference text, typically computed as ROUGE-L or ROUGE-1/2/3. Measures lexical similarity.
- `METEOR` — range: [0, 1]
  - Alignment-based metric that considers synonyms, stemming, and word order to measure semantic similarity between generated and reference text.
- `CIDEr` **(primary)** — range: [0, 1]
  - Consensus-based Image Description Evaluation; computes TF-IDF weighted n-gram similarity between generated and multiple reference captions to penalize common words and reward consensus.

## Input / output format

**Input**: 3D point cloud data of multi-room scenes, top-down spatial views, and room-level annotations. Inputs are paired with task-specific prompts: questions for XR-QA, high-level goals for XR-EmbodiedPlanning, and room identifiers for XR-SceneCaption.

**Output**: Natural language text: direct answers to spatial queries, step-by-step subtask plans for navigation/planning, or descriptive captions for specific rooms.

## Scoring recipe

```python
def compute_metrics(predictions, references):
    # predictions and references are lists of strings
    rouge_scores = rouge_score(predictions, references, use_stemmer=True)
    meteor_scores = [meteor_score([ref], pred) for pred, ref in zip(predictions, references)]
    cider_scores = compute_cider(predictions, references)
    return {
        "ROUGE": np.mean(rouge_scores),
        "METEOR": np.mean(meteor_scores),
        "CIDEr": np.mean(cider_scores)
    }
```

## Common pitfalls

- Down-sampling large point clouds to fit model context windows often discards fine-grained details, causing severe performance drops on small objects (addressed by the XR-QA-S subset with a 0.05 m² bounding box threshold).
- Single-room benchmarks heavily bias models toward common furniture (chairs/tables); cross-room evaluation requires balanced object diversity and explicit spatial reasoning across room boundaries.
- Models may rely on shortcut heuristics or room-level priors rather than true cross-room attention, failing to localize objects accurately in dense environments.

## Evidence (verbatim from paper)

> XR-SceneCaption challenges 3D-VLM to generate comprehensive scene descriptions and captions for specific rooms while inferring attributes based on present objects. Table 5: Ablation studies. ATR: the activate token ratio of sparse vision tokens. #: do not use the scene magnifier module.

| Parameter | | ROUGE | METEOR | CIDEr |

## Citation

```bibtex
@misc{zhi2024lscenellm,
  title={LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences},
  author={Zhi et al. (2024)},
  year={2024},
  note={arXiv:2412.01292}
}
```

- arXiv: 2412.01292