Morse 500 Eval

---
name: morse-500-eval
description: Evaluates multimodal reasoning capabilities of vision-language models across six categories: mathematical, abstract, spatial, temporal, physical, and planning. It uses programmatically generated video clips to test dynamic visual narrative comprehension while eliminating prompt-based shortcuts. Use when the user wants to benchmark on MORSE-500, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.05523
  bibtex_key: cai2025morse500
  confidence: high
---

# morse-500-eval

> MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoning — Cai et al. (2025) (arXiv:2506.05523, 2025)

## What this evaluates

Evaluates multimodal reasoning capabilities of vision-language models across six categories: mathematical, abstract, spatial, temporal, physical, and planning. It uses programmatically generated video clips to test dynamic visual narrative comprehension while eliminating prompt-based shortcuts.

## Datasets

- **MORSE-500** — total 500; splits: test (500); repo https://github.com/morse-benchmark/morse-500-code

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions over the benchmark. Model predictions are extracted using an external LLM (e.g., Qwen2.5 72B AWQ) and compared to ground truth via exact string matching.

## Input / output format

**Input**: Video clip (or sampled frames at 2fps, max 32 frames for image-only models, downsampled to max 512px side length) paired with the minimal instruction: 'Answer the question in this video.'

**Output**: Free-form text answer to the question embedded in the video.

## Scoring recipe

```python
def compute_accuracy(predictions, ground_truths):
    correct = 0
    for pred, gold in zip(predictions, ground_truths):
        extracted = extract_answer_with_llm(pred)
        if extracted.strip().lower() == gold.strip().lower():
            correct += 1
    return (correct / len(ground_truths)) * 100
```

## Common pitfalls

- Image-only models require frame sampling (2fps, max 32 frames) and downscaling to 512px, which may degrade temporal and spatial reasoning performance.
- Answer extraction relies on an external LLM for string matching, which can introduce parsing errors or bias independent of the evaluated model's actual capability.
- No few-shot examples or format-specific guidance are provided, forcing models to self-format answers and complicating automated evaluation.

## Evidence (verbatim from paper)

> We report accuracy as the primary evaluation metric—the percentage of correctly answered questions over the benchmark. We provided detailed instructions on the output formatting in the video, and we extract the answers from the model prediction using a LLM (e.g. Qwen2.5 72B AWQ) and perform string matching for accuracy calculation, following MathVista [Lu et al., 2024].

## Citation

```bibtex
@misc{cai2025morse500,
  title={MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoning},
  author={Cai et al. (2025)},
  year={2025},
  note={arXiv:2506.05523}
}
```

- arXiv: 2506.05523