Momagraph Bench Eval

---
name: momagraph-bench-eval
description: Evaluates embodied task planning and visual correspondence capabilities of vision-language models. It probes spatial-functional reasoning, multi-step action planning, and cross-view consistency in indoor scenes. Use when the user wants to benchmark on MomaGraph-Bench, BLINK, or asks about evaluating this task. Reports accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.16909
  bibtex_key: ju2025momagraph
  confidence: high
---

# momagraph-bench-eval

> MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Model for Embodied Task Planning — Ju et al. (2025) (arXiv:2512.16909, 2025)

## What this evaluates

Evaluates embodied task planning and visual correspondence capabilities of vision-language models. It probes spatial-functional reasoning, multi-step action planning, and cross-view consistency in indoor scenes.

## Datasets

- **MomaGraph-Bench** — total ?; splits: test (-1)
- **BLINK** — total ?; splits: test (-1)

## Metrics

- `accuracy (%)` **(primary)** — range: percent
  - Calculated as the number of correctly predicted task plans or correspondence answers divided by the total number of instances, multiplied by 100.
- `success rate (%)` — range: percent
  - Calculated as the number of successful multi-step task trials divided by the total number of trials, multiplied by 100.

## Input / output format

**Input**: Multi-view RGB-D observations and natural language task instructions.

**Output**: Task-oriented scene graph G_T (nodes, spatial/functional edges, action types) followed by a structured action sequence, or direct task plan/answer.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100

def compute_success_rate(success_flags, total_trials):
    return (sum(success_flags) / total_trials) * 100
```

## Common pitfalls

- Confusing the 'Direct Plan (w/o Graph)' baseline with the 'Graph-then-Plan (w/ Graph)' setting, which significantly impacts reported scores.
- Assuming overall accuracy masks severe degradation on higher-complexity tiers (T3/T4) where open-source models drop sharply.
- Overlooking that real-robot success rates are measured per trial across 10 viewpoint changes, not per individual action step.

## Evidence (verbatim from paper)

> We report accuracy (%) across four tiers (T1–T4) and the overall score, with and without graph-based reasoning.

## Citation

```bibtex
@misc{ju2025momagraph,
  title={MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Model for Embodied Task Planning},
  author={Ju et al. (2025)},
  year={2025},
  note={arXiv:2512.16909}
}
```

- arXiv: 2512.16909