M Portal Eval

---
name: m-portal-eval
description: Evaluates multi-step spatial and physical reasoning in multimodal language models by requiring them to generate or validate chain-of-thought plans for solving Portal 2-inspired puzzle maps. The benchmark probes the model's ability to integrate visual map layouts with textual instructions to produce physically sound, multi-step traversal strategies. Use when the user wants to benchmark on M-Portal, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.22992
  bibtex_key: jiang2025marble
  confidence: high
---

# m-portal-eval

> MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning — Jiang et al. (2025) (arXiv:2506.22992, 2025)

## What this evaluates

Evaluates multi-step spatial and physical reasoning in multimodal language models by requiring them to generate or validate chain-of-thought plans for solving Portal 2-inspired puzzle maps. The benchmark probes the model's ability to integrate visual map layouts with textual instructions to produce physically sound, multi-step traversal strategies.

## Datasets

- **M-Portal** — total 1024; splits: test (1024)

## Metrics

- `F1 score` **(primary)** — range: percent
  - Binary classification F1-score for the plan correctness subtask, measuring the harmonic mean of precision and recall for identifying correct vs. incorrect candidate plans.
- `Accuracy` — range: percent
  - Exact match accuracy for the fill-the-blanks subtask, measuring the proportion of instances where the model correctly selects and orders the missing steps from a set of candidate options.

## Input / output format

**Input**: Multimodal inputs (screenshots of a Portal map and textual contextualization) plus a task instruction. For fill-the-blanks: a partial plan with masked steps and a set of candidate options.

**Output**: A Chain-of-Thought plan consisting of interpretable reasoning steps (s1, s2, ..., sn) for plan correctness; or a selection of correct missing steps in order for fill-the-blanks.

## Scoring recipe

```python
# Plan Correctness (512 instances)
correct = 0
for plan in model_plans:
    if plan == ground_truth_plan:
        correct += 1
precision = correct / len(model_plans)
recall = correct / len(ground_truth_plans)
f1 = 2 * precision * recall / (precision + recall)

# Fill-the-blanks (512 instances)
correct = 0
for selection in model_selections:
    if selection == ground_truth_steps:
        correct += 1
accuracy = correct / len(model_selections)
```

## Common pitfalls

- High class imbalance in plan correctness: each map yields 1 correct plan and 32 incorrect ones, skewing majority-class accuracy.
- Evaluation is closed-ended (binary/multiple-choice) rather than interactive; models are not tested in the actual game environment.
- Models frequently fail at the perception stage, misinterpreting visual map layouts, which cascades into reasoning failures even on simplified subtasks.

## Evidence (verbatim from paper)

> On the plan correctness task, all investigated models (MLLMs as well as text-only LLMs) performed very poorly with a minority class F1 score of around 6%, similar to the random baseline. In the easier fill-the-blanks task, 8 out of 12 models outperform the random baseline.

## Citation

```bibtex
@misc{jiang2025marble,
  title={MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning},
  author={Jiang et al. (2025)},
  year={2025},
  note={arXiv:2506.22992}
}
```

- arXiv: 2506.22992