Hm Eqa Eval

---
name: hm-eqa-eval
description: This benchmark evaluates a robot's ability to autonomously explore an unseen indoor environment and answer multiple-choice questions requiring object identification, counting, spatial reasoning, and multi-goal navigation. It probes the agent's multimodal perception, iterative reasoning, and navigation efficiency in a dynamic, tool-invoking workflow. Use when the user wants to benchmark on HM-EQA, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.17462
  bibtex_key: [REDACTED]
  confidence: high
---

# hm-eqa-eval

> General-Purpose Robotic Navigation via LVLM-Orchestrated Perception, Reasoning, and Acting — Lange et al. (2025) (arXiv:2506.17462, 2025)

## What this evaluates

This benchmark evaluates a robot's ability to autonomously explore an unseen indoor environment and answer multiple-choice questions requiring object identification, counting, spatial reasoning, and multi-goal navigation. It probes the agent's multimodal perception, iterative reasoning, and navigation efficiency in a dynamic, tool-invoking workflow.

## Datasets

- **HM-EQA** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of multiple-choice questions answered correctly. Success, failure, and inconclusive outcomes are reported separately and always sum to 1.
- `Mean Path Length` — range: other
  - Average trajectory length in meters, measuring exploration efficiency.
- `Mean Token Usage` — range: other
  - Average number of tokens processed by the LVLM, serving as a proxy for computational cost and runtime.

## Input / output format

**Input**: Multimodal observations from the Habitat Lab simulator (e.g., RGB-D frames, scene graphs, occupancy grids) and a multiple-choice navigation query. The agent iteratively queries perception modules and reasons over these inputs.

**Output**: Sequential navigation actions (trajectory steps) and a final selected multiple-choice answer.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold) * 100

def compute_path_length(trajectories):
    return sum(len(t) for t in trajectories) / len(trajectories)

def compute_token_usage(token_counts):
    return sum(token_counts) / len(token_counts)
```

## Common pitfalls

- Inconclusive runs (exceeding 500 reasoning steps or $5 budget) are reported separately and do not sum to 100% with success/failure rates.
- Token usage and path length are highly variable; comparing them across methods requires accounting for different computational budgets and exploration strategies.
- Accuracy is calculated only on completed tasks, but inconclusive runs are excluded from the success/failure ratio, which can skew perceived performance.

## Evidence (verbatim from paper)

> We benchmark using three metrics: (1) Accuracy, the percentage of multiple-choice questions answered correctly; (2) Mean Path Length, the average trajectory length (in meters), measuring exploration efficiency; and (3) Mean Token Usage, the average number of tokens processed by the LVLM, which serves as a proxy for computational cost and runtime.

## Citation

```bibtex
@misc{lange2025arnanavigation,
  title={General-Purpose Robotic Navigation via LVLM-Orchestrated Perception, Reasoning, and Acting},
  author={Lange et al. (2025)},
  year={2025},
  note={arXiv:2506.17462}
}
```

- arXiv: 2506.17462