Symsearch Omnigibson Eval

---
name: symsearch-omnigibson-eval
description: Evaluates an agent's ability to perform open-vocabulary interactive object search in indoor environments using relational semantic reasoning over 3D scene graphs. It probes exploration efficiency, reasoning accuracy, and computational cost compared to embedding-based and LLM-based planners. Use when the user wants to benchmark on SymSearch, OmniGibson, or asks about evaluating this task. Reports Success Rate (SR), Success weighted by Path Length (SPL).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.05642
  bibtex_key: mahdi2026scout
  confidence: high
---

# symsearch-omnigibson-eval

> Relational Semantic Reasoning on 3D Scene Graphs for Open World Interactive Object Search — Mahdi et al. (2026) (arXiv:2603.05642, 2026)

## What this evaluates

Evaluates an agent's ability to perform open-vocabulary interactive object search in indoor environments using relational semantic reasoning over 3D scene graphs. It probes exploration efficiency, reasoning accuracy, and computational cost compared to embedding-based and LLM-based planners.

## Datasets

- **SymSearch** — total 200; splits: test (200)
- **OmniGibson** — total 50; splits: test (50)

## Metrics

- `Success Rate (SR)` **(primary)** — range: percent
  - Percentage of episodes where the agent successfully locates the target object within the maximum step limit.
- `Success weighted by Path Length (SPL)` **(primary)** — range: [0, 1]
  - Standard robotics metric: SR × (optimal_steps / actual_steps). Rewards finding the target efficiently.
- `N Steps` — range: other
  - Average number of high-level actions taken per episode.
- `Inference Time` — range: other
  - Average time in seconds spent computing the next action per step.

## Input / output format

**Input**: Textual object query, current 3D scene graph state, and agent observations (RGB-D images or symbolic node features).

**Output**: Discrete high-level action selection (e.g., navigate to a specific room, frontier, or object node; or interact with a container).

## Scoring recipe

```python
def compute_metrics(episodes, optimal_steps):
    sr_sum = 0
    spl_sum = 0
    steps_sum = 0
    inf_time_sum = 0
    for ep in episodes:
        success = 1 if ep.found_target else 0
        sr_sum += success
        spl_sum += success * (optimal_steps / ep.actual_steps) if ep.actual_steps > 0 else 0
        steps_sum += ep.actual_steps
        inf_time_sum += ep.inference_time
    n = len(episodes)
    return {
        'SR': (sr_sum / n) * 100,
        'SPL': spl_sum / n,
        'N_Steps': steps_sum / n,
        'Inference_Time': inf_time_sum / n
    }
```

## Common pitfalls

- Simulation uses oracle 'magic open' actions and ground truth segmentation, which overestimates real-world performance.
- Real-world failures are dominated by perception errors (segmentation/localization) rather than reasoning flaws.
- LLM baselines incur high inference latency and non-determinism, making direct SR comparisons sensitive to sampling variance.
- Inference time is reported separately from total execution time, which includes navigation and manipulation latency.

## Evidence (verbatim from paper)

> As metrics, we compute the Success Rate (SR) and Success weighted by Path Length (SPL). Additionally, we record the number of high-level steps taken per episode and the average inference time to quantify exploration efficiency.

## Citation

```bibtex
@misc{mahdi2026scout,
  title={Relational Semantic Reasoning on 3D Scene Graphs for Open World Interactive Object Search},
  author={Mahdi et al. (2026)},
  year={2026},
  note={arXiv:2603.05642}
}
```

- arXiv: 2603.05642