Saw Bench Eval

---
name: saw-bench-eval
description: Evaluates multimodal foundation models' ability to perform observer-centric spatial reasoning, path integration, and camera geometry inference using egocentric videos recorded from smart glasses. It probes sustained tracking of intermediate movements and the distinction between physical translation and camera rotation. Use when the user wants to benchmark on SAW-BENCH, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.16682
  bibtex_key: li2026learning
  confidence: high
---

# saw-bench-eval

> Learning Situated Awareness in the Real World — Li et al. (2026) (arXiv:2602.16682, 2026)

## What this evaluates

Evaluates multimodal foundation models' ability to perform observer-centric spatial reasoning, path integration, and camera geometry inference using egocentric videos recorded from smart glasses. It probes sustained tracking of intermediate movements and the distinction between physical translation and camera rotation.

## Datasets

- **SAW-BENCH** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly answered multiple-choice questions after extracting the predicted answer via a regular-expression parser, with a GPT-4o-mini fallback for failed extractions.

## Input / output format

**Input**: Egocentric video clips from smart glasses paired with a multiple-choice question about spatial reasoning, trajectory shape, or memory.

**Output**: Free-form text response containing the selected option, which is then parsed via regex or GPT-4o-mini to extract the final answer.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred_raw, gold in zip(predictions, golds):
        pred = extract_answer(pred_raw) # regex parser, fallback to GPT-4o-mini
        if pred == gold:
            correct += 1
    return correct / len(golds)
```

## Common pitfalls

- Models frequently conflate camera rotation/panning with the observer's physical translation.
- Open-source models often rely on shortcut cues from key frames (e.g., first and last) instead of tracking full trajectory dynamics.
- Static video captions (Socratic baseline) discard critical observer-centric cues like viewpoint changes and temporal structure.

## Evidence (verbatim from paper)

> We use accuracy as the primary metric to evaluate model performance on SAW-BENCH. Following recent benchmarks for foundation model evaluation (Wang et al., 2024b, Shangguan et al., 2025, Zhao et al., 2025), we first apply a regular-expression-based parser to extract the predicted answer from each model's raw response. If the parser fails, we additionally use GPT-4o-mini to extract the answer from the raw output. Prompt used for answer extraction is provided in §B.3.

## Citation

```bibtex
@misc{li2026learning,
  title={Learning Situated Awareness in the Real World},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2602.16682}
}
```

- arXiv: 2602.16682