Osmabench Eval

---
name: osmabench-eval
description: Evaluates open semantic mapping models' robustness to dynamic indoor lighting and motion conditions. It measures semantic segmentation accuracy and frequency-weighted IoU, alongside LLM-generated visual question answering accuracy on scene graphs. Use when the user wants to benchmark on ReplicaCAD, HM3D, or asks about evaluating this task. Reports mAcc, f-mIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.10331
  bibtex_key: popov2025osmabench
  confidence: high
---

# osmabench-eval

> OSMa-Bench: Evaluating Open Semantic Mapping Under Varying Lighting Conditions — Popov et al. (2025) (arXiv:2503.10331, 2025)

## What this evaluates

Evaluates open semantic mapping models' robustness to dynamic indoor lighting and motion conditions. It measures semantic segmentation accuracy and frequency-weighted IoU, alongside LLM-generated visual question answering accuracy on scene graphs.

## Datasets

- **ReplicaCAD** — total 22; splits: test (-1)
- **HM3D** — total 8; splits: test (-1)

## Metrics

- `mAcc` **(primary)** — range: [0, 1]
  - Mean accuracy computed from an overall confusion matrix (ReplicaCAD) or averaged per scene (HM3D), giving equal weight to all classes regardless of frequency.
- `f-mIoU` **(primary)** — range: [0, 1]
  - Frequency-weighted mean Intersection over Union, where voluminous objects like walls and floors contribute more to the score based on class frequency.
- `VQA Accuracy` — range: percent
  - Percentage of correctly answered questions generated by Gemini across categories such as existence, attributes, and spatial relations.

## Input / output format

**Input**: RGB-D 3D scenes simulated in Habitat Sim with configurable lighting (baseline, nominal, camera, dynamic) and robot trajectories; scene graphs and LLM-generated questions for VQA.

**Output**: Semantic segmentation maps, scene graphs, and natural language answers to VQA questions.

## Scoring recipe

```python
# For ReplicaCAD (global confusion matrix)
conf_matrix = compute_confusion_matrix(predictions, ground_truth)
mAcc = mean(diag(conf_matrix) / conf_matrix.sum(axis=1))
f_mIoU = sum(class_freq * (diag(conf_matrix) / (conf_matrix.sum(axis=1) + conf_matrix.sum(axis=0) - diag(conf_matrix)))) / sum(class_freq)

# For HM3D (average per scene)
scene_mAcc = [compute_mAcc(scene_preds, scene_gt) for scene in scenes]
scene_f_mIoU = [compute_f_mIoU(scene_preds, scene_gt) for scene in scenes]
mAcc = mean(scene_mAcc)
f_mIoU = mean(scene_f_mIoU)

# VQA Accuracy
vqa_acc = sum(correct_answers) / total_questions * 100
```

## Common pitfalls

- HM3D metrics are averaged per scene due to label inconsistencies across scenes, unlike ReplicaCAD which uses a global confusion matrix.
- Functional relationship questions were excluded from VQA evaluation due to LLM ambiguity.
- Class name formatting (underscores vs spaces) significantly impacts segmentation scores.
- Object merging in scene graphs inflates instance counts, skewing measurement-based questions.

## Evidence (verbatim from paper)

> Collecting mAcc and f-mIoU metrics from testing ConceptGraphs, BBQ, and OpenScene on scenes in ReplicaCAD and HM3D reveals key trade-offs. f-mIoU is weighted by class frequency, so voluminous objects like walls and floors matter more, while mAcc gives equal weight to all objects.

## Citation

```bibtex
@misc{popov2025osmabench,
  title={OSMa-Bench: Evaluating Open Semantic Mapping Under Varying Lighting Conditions},
  author={Popov et al. (2025)},
  year={2025},
  note={arXiv:2503.10331}
}
```

- arXiv: 2503.10331