Physicalai Warehouse Eval

---
name: physicalai-warehouse-eval
description: Evaluates a model's capacity for metric spatial reasoning, object enumeration, relative spatial comparisons, and topological/directional relationship understanding within real-world warehouse environments. Use when the user wants to benchmark on PhysicalAI-Spatial-Intelligence-Warehouse, or asks about evaluating this task. Reports normalized exact-match accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.15490
  bibtex_key: traore2025smolrgpt
  confidence: medium
---

# physicalai-warehouse-eval

> SmolRGPT: Efficient Spatial Reasoning for Warehouse Environments with 600M Parameters — Traore et al. (2025) (arXiv:2509.15490, 2025)

## What this evaluates

Evaluates a model's capacity for metric spatial reasoning, object enumeration, relative spatial comparisons, and topological/directional relationship understanding within real-world warehouse environments.

## Datasets

- **PhysicalAI-Spatial-Intelligence-Warehouse** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `normalized exact-match accuracy` **(primary)** — range: [0, 1]
  - Accuracy computed by comparing model predictions against ground-truth answers after applying a normalization protocol that accommodates common variations in numeric and word representations.

## Input / output format

**Input**: A real-world warehouse scene image, a natural language question (distance, count, multiple-choice grounding, or spatial relation query), and region-indexed objects with a unified coordinate system.

**Output**: A normalized answer string (numeric value or word representation) corresponding to the question type.

## Scoring recipe

```python
def score(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        pred_norm = normalize_answer(pred)
        gold_norm = normalize_answer(gold)
        if pred_norm == gold_norm:
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Failing to apply the dataset's normalization protocol before comparison, leading to false negatives for semantically identical answers (e.g., '5m' vs '5 meters').
- Misinterpreting relative spatial references without leveraging the provided region coordinate system.
- Applying a single scoring threshold across all four question categories instead of using category-specific normalization rules.

## Evidence (verbatim from paper)

> To support consistent assessment, the dataset offers standardized training, validation, and test splits, with normalized answer formats. Each instance includes objects indexed by region within the warehouse, allowing for precise spatial referencing via a unified coordinate system. The normalization protocol accommodates common variations in response formats, supporting a range of numeric and word representations while maintaining consistency across different types of queries.

## Citation

```bibtex
@misc{traore2025smolrgpt,
  title={SmolRGPT: Efficient Spatial Reasoning for Warehouse Environments with 600M Parameters},
  author={Traore et al. (2025)},
  year={2025},
  note={arXiv:2509.15490}
}
```

- arXiv: 2509.15490