Wmabench Eval

---
name: wmabench-eval
description: Evaluates Vision-Language Models on atomic world modeling capabilities across perception (spatial, temporal, motion) and prediction (mechanistic simulation, transitive/compositional inference) tasks. It probes whether VLMs possess internal representations of physical causality, dynamics, and multi-step reasoning comparable to human intuition. Use when the user wants to benchmark on WM-ABench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.21876
  bibtex_key: gao2025wmabench
  confidence: high
---

# wmabench-eval

> Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation — Gao et al. (2025) (arXiv:2506.21876, 2025)

## What this evaluates

Evaluates Vision-Language Models on atomic world modeling capabilities across perception (spatial, temporal, motion) and prediction (mechanistic simulation, transitive/compositional inference) tasks. It probes whether VLMs possess internal representations of physical causality, dynamics, and multi-step reasoning comparable to human intuition.

## Datasets

- **WM-ABench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy: the proportion of correctly predicted labels out of all evaluated instances. Calculated as (number of correct predictions) / (total number of parsed instances).

## Input / output format

**Input**: Image or video frames paired with a text question/prompt. All models receive a fixed system prompt and are evaluated using greedy decoding.

**Output**: Free-text response that is subsequently parsed into a categorical label using a predefined template. Instances that fail parsing are excluded from scoring.

## Scoring recipe

```python
correct = 0
total = 0
for instance in dataset:
    output = model.generate(instance.image, instance.prompt, greedy=True)
    label = parse_template(output)
    if label is not None:
        total += 1
        if label == instance.ground_truth:
            correct += 1
return correct / total if total > 0 else 0.0
```

## Common pitfalls

- Parsing failures are skipped rather than penalized, which may bias accuracy if certain models or tasks systematically fail to follow the output template.
- Greedy decoding is enforced across all models for consistency, potentially underestimating the performance of models that benefit from sampling or temperature tuning.
- Near-random performance on dynamic tasks (e.g., motion trajectory) reflects representation gaps rather than mere hallucination, requiring careful interpretation beyond raw scores.

## Evidence (verbatim from paper)

> We evaluate model performance by comparing the parsed labels from model outputs to the ground-truth labels, and skip the instances where model outputs failed to be parsed by our template. ... Qwen2-VL achieves the highest overall performance with an average accuracy of 67.7%.

## Citation

```bibtex
@misc{gao2025wmabench,
  title={Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation},
  author={Gao et al. (2025)},
  year={2025},
  note={arXiv:2506.21876}
}
```

- arXiv: 2506.21876