Sage Bench Eval

---
name: sage-bench-eval
description: Evaluates embodied vision-and-language navigation (VLN) capabilities within physically executable 3D Gaussian Splatting environments. It probes a model's ability to follow natural language instructions (high- and low-level), navigate to goals without collisions, and exhibit smooth, natural motion continuity rather than mechanical or wall-hugging behaviors. Use when the user wants to benchmark on SAGE-Bench, VLN-CE (R2R Val-Unseen), or asks about evaluating this task. Reports SR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.21307
  bibtex_key: miao2025sage3d
  confidence: high
---

# sage-bench-eval

> Towards Physically Executable 3D Gaussian for Embodied Navigation — Miao et al. (2025) (arXiv:2510.21307, 2025)

## What this evaluates

Evaluates embodied vision-and-language navigation (VLN) capabilities within physically executable 3D Gaussian Splatting environments. It probes a model's ability to follow natural language instructions (high- and low-level), navigate to goals without collisions, and exhibit smooth, natural motion continuity rather than mechanical or wall-hugging behaviors.

## Datasets

- **SAGE-Bench** — total 2000000; splits: train (500000), test (-1)
- **VLN-CE (R2R Val-Unseen)** — total ?; splits: val-unseen (-1)

## Metrics

- `SR` **(primary)** — range: [0, 1]
  - Success Rate: binary metric indicating whether the agent reaches within a predefined threshold distance of the goal location at the end of the episode.
- `OSR` — range: [0, 1]
  - Oracle Success Rate: binary metric indicating whether the agent visits within the threshold distance of the goal at any point during the episode.
- `SPL` — range: [0, 1]
  - Success weighted by Path Length: SR multiplied by the ratio of the shortest possible path length to the actual path length taken by the agent.
- `CR` — range: other
  - Collision Rate: frequency or count of collisions with environment geometry during navigation.
- `CSR` — range: [0, 1]
  - Continuity Success Rate: novel metric that measures inclusive success without requiring exact ground-truth trajectory fitting (formulas detailed in Sec. 3.3).
- `ICP` — range: [0, 1]
  - Inter-Collision Points: measures sustained collisions during navigation, capturing micro-collisions or wall-hugging that standard CR misses (formulas detailed in Sec. 3.3).
- `PS` — range: [0, 1]
  - Path Smoothness: evaluates motion continuity by penalizing large, mechanical turning angles in favor of smooth, natural motion (formulas detailed in Sec. 3.3).
- `Episode Time` — range: seconds
  - Duration in seconds until the episode terminates (goal reached, collision occurs, or max time of 120s is hit).
- `Explored Areas` — range: other
  - Total area covered by the agent during the episode, used for the No-goalNav task.

## Input / output format

**Input**: RGB-D or 3DGS scene observations paired with natural language navigation instructions (categorized as high-level or low-level).

**Output**: Sequential navigation actions/trajectories (e.g., move forward, turn left/right) until the goal is reached or the episode terminates.

## Scoring recipe

```python
def compute_vln_metrics(traj, goal, shortest_path, threshold=0.5):
    sr = 1.0 if distance(traj[-1], goal) < threshold else 0.0
    osr = 1.0 if any(distance(p, goal) < threshold for p in traj) else 0.0
    spl = sr * (shortest_path / max(len(traj), 1))
    cr = collision_count(traj) / len(traj)
    # CSR, ICP, PS computed per Sec 3.3 protocol
    return {'SR': sr, 'OSR': osr, 'SPL': spl, 'CR': cr}
```

## Common pitfalls

- Conventional metrics like SR and CR fail to capture unnatural navigation behaviors (e.g., prolonged wall-hugging or sustained micro-collisions); CSR, ICP, and PS are required to detect these.
- 3DGS-based scenes require ~33% more training iterations (160 vs 120) to reach the same success rate compared to scanned mesh data, despite offering faster per-frame rendering.
- Models perform significantly worse on high-level instructions compared to low-level step-by-step instructions, revealing a gap in natural language grounding and planning.

## Evidence (verbatim from paper)

> In addition to the three novel metrics we proposed in Section[3.3] for evaluating the natural continuity of model navigation — CSR, ICP, and PS — we also adopt common metrics used in VLN tasks, including success rate (SR), oracle success rate (OSR), and success weighted by path length (SPL) and Collision Rate (CR).

## Citation

```bibtex
@misc{miao2025sage3d,
  title={Towards Physically Executable 3D Gaussian for Embodied Navigation},
  author={Miao et al. (2025)},
  year={2025},
  note={arXiv:2510.21307}
}
```

- arXiv: 2510.21307