Spamem Eval

---
name: spamem-eval
description: This benchmark evaluates dynamic spatial reasoning and perception-memory integration in embodied environments. It probes models across three levels: static spatial perception, text-conditioned temporal memory, and visual-conditioned temporal memory, testing capabilities like object recognition, visual grounding, depth estimation, trajectory tracking, and long-horizon state reconstruction. Use when the user wants to benchmark on SpaMEM, or asks about evaluating this task. Reports mIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.22409
  bibtex_key: liao2026spamem
  confidence: high
---

# spamem-eval

> SpaMEM: Benchmarking Dynamic Spatial Reasoning via Perception-Memory Integration in Embodied Environments — Liao et al. (2026) (arXiv:2604.22409, 2026)

## What this evaluates

This benchmark evaluates dynamic spatial reasoning and perception-memory integration in embodied environments. It probes models across three levels: static spatial perception, text-conditioned temporal memory, and visual-conditioned temporal memory, testing capabilities like object recognition, visual grounding, depth estimation, trajectory tracking, and long-horizon state reconstruction.

## Datasets

- **SpaMEM** — total ?; splits: test (-1)

## Metrics

- `mIoU` **(primary)** — range: [0, 1]
  - Mean Intersection over Union between predicted and ground-truth bounding boxes for visual grounding.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall for semantic object recognition.
- `AbsRel` — range: other
  - Mean absolute relative error for depth and proximity estimation.
- `Acc@1` — range: [0, 1]
  - Accuracy of predicting the exact object count.
- `TL` — range: [0, 1]
  - Accuracy of temporal localization (identifying correct frames or timesteps).
- `STT` — range: [0, 1]
  - Accuracy of spatio-temporal tracking (reconstructing object trajectories across containers).
- `SED` — range: [0, 1]
  - Accuracy of short-term event detection for add and remove operations.
- `CSR` — range: [0, 1]
  - Accuracy of cumulative state reconstruction (global inventory update at episode end).

## Input / output format

**Input**: Egocentric RGB frames, optionally paired with aligned depth maps (RGB-D). For Level 2, explicit text-based state histories are provided. Queries are issued either step-by-step (short-term) or at the end of an episode (long-term).

**Output**: Model must produce task-specific predictions: bounding box coordinates (grounding), object class labels (recognition), depth values (estimation), relative directions (left/right/behind), integer counts, temporal frame indices (localization), trajectory paths (tracking), and updated state/inventory lists (event detection & reconstruction).

## Scoring recipe

```python
def score(predictions, gold):
    scores = {}
    scores['mIoU'] = mean([iou(p, g) for p, g in zip(predictions['boxes'], gold['boxes'])])
    tp = sum(1 for p, g in zip(predictions['labels'], gold['labels']) if p == g)
    scores['F1'] = 2 * tp / (len(predictions['labels']) + len(gold['labels']) + 1e-8)
    scores['AbsRel'] = mean([abs(p - g) / (g + 1e-8) for p, g in zip(predictions['depth'], gold['depth'])])
    scores['Acc@1'] = mean([1 if p == g else 0 for p, g in zip(predictions['count'], gold['count'])])
    scores['TL'] = mean([1 if p == g else 0 for p, g in zip(predictions['timestep'], gold['timestep'])])
    scores['STT'] = mean([1 if p == g else 0 for p, g in zip(predictions['trajectory'], gold['trajectory'])])
    scores['SED'] = mean([1 if p == g else 0 for p, g in zip(predictions['event'], gold['event'])])
    scores['CSR'] = mean([1 if p == g else 0 for p, g in zip(predictions['state'], gold['state'])])
    return scores
```

## Common pitfalls

- Text-conditioned (L2) performance overestimates true visual memory; models excel at bookkeeping with symbolic history but collapse without it.
- Semantic recognition accuracy does not transfer to spatial grounding; high F1 often coexists with near-zero mIoU.
- Short-term event detection success does not compose into long-term state reconstruction; local change signals fail to update global inventory.

## Evidence (verbatim from paper)

> Visual grounding (VGL), measured by mean IoU (mIoU), is nearly non-functional: scores remain at 0.00–0.01 for almost all models. This indicates that while models may recognize object presence, they struggle to map semantics to coordinate-consistent localization in cluttered indoor scenes.

## Citation

```bibtex
@misc{liao2026spamem,
  title={SpaMEM: Benchmarking Dynamic Spatial Reasoning via Perception-Memory Integration in Embodied Environments},
  author={Liao et al. (2026)},
  year={2026},
  note={arXiv:2604.22409}
}
```

- arXiv: 2604.22409