Msnn Eval

---
name: msnn-eval
description: Evaluates a model's ability to predict the immediate next navigation step in a 3D scene given a multi-modal situation description and a textual goal. Use when the user wants to benchmark on MSNN, or asks about evaluating this task. Reports Next-step Action Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.02389
  bibtex_key: linghu2024multimodal
  confidence: high
---

# msnn-eval

> Multi-modal Situated Reasoning in 3D Scenes — Linghu et al. (2024) (arXiv:2409.02389, 2024)

## What this evaluates

Evaluates a model's ability to predict the immediate next navigation step in a 3D scene given a multi-modal situation description and a textual goal.

## Datasets

- **MSNN** — total 34000; splits: test (-1)

## Metrics

- `Next-step Action Accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted immediate next-step actions compared to the ground truth action derived from an A* optimal trajectory.

## Input / output format

**Input**: Interleaved multi-modal description of the agent's current situation (location, orientation, text), a textual goal description, and the overall scene.

**Output**: Textual description of the immediate next navigation action.

## Scoring recipe

```python
correct = 0
for situation, goal, gt_action in test_set:
    pred_action = model.predict(situation, goal)
    if pred_action == gt_action:
        correct += 1
accuracy = correct / len(test_set) * 100
```

## Common pitfalls

- Ground truth actions are derived from A* shortest-path planning on floor plans, which may not reflect real-world navigation constraints or agent capabilities.
- Evaluation focuses only on the immediate next step, potentially ignoring long-horizon planning failures.

## Evidence (verbatim from paper)

> we instruct models to answer the immediate next action for navigating to the goal in a textual form. For evaluation, we generate MSNN... The optimal trajectory is sampled by running an A* algorithm planning the shortest path from the starting location to the goal on the floor plan and the immediate next-step action is determined by following the direction of optimal trajectory relative to the starting situation.

## Citation

```bibtex
@misc{linghu2024multimodal,
  title={Multi-modal Situated Reasoning in 3D Scenes},
  author={Linghu et al. (2024)},
  year={2024},
  note={arXiv:2409.02389}
}
```

- arXiv: 2409.02389