Vaexbench Eval

---
name: vaexbench-eval
description: Evaluates multimodal large language models' ability to perform extractive and abstractive spatiotemporal reasoning on egocentric videos. It probes long-horizon memory, object tracking, spatial orientation, and metric distance estimation under both multiple-choice and free-form generation settings. Use when the user wants to benchmark on VAEX-Bench, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.13091
  bibtex_key: bang2026vaexbench
  confidence: high
---

# vaexbench-eval

> Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence — Bang et al. (2026) (arXiv:2603.13091, 2026)

## What this evaluates

Evaluates multimodal large language models' ability to perform extractive and abstractive spatiotemporal reasoning on egocentric videos. It probes long-horizon memory, object tracking, spatial orientation, and metric distance estimation under both multiple-choice and free-form generation settings.

## Datasets

- **VAEX-Bench** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correct predictions out of total queries. For stochastic decoding stability, Accuracy@k is reported as the average accuracy across k independent generations per query (k=5 in this work). Free-form responses are scored via exact match or LLM-as-a-judge.

## Input / output format

**Input**: Egocentric video (uniformly sampled to 32 frames for most models) paired with a fixed instruction header prompt. MCQ tasks include answer options; free-form tasks provide open-ended queries without options.

**Output**: Single selected option for MCQ tasks, or free-form text generation for open-ended queries.

## Scoring recipe

```python
def compute_accuracy(predictions, golds, task_type='mcq'):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if task_type == 'mcq':
            if pred.strip() == gold.strip():
                correct += 1
        elif task_type == 'freeform':
            if gold == 'Memory-Action':
                correct += llm_judge_score(pred, gold)
            else:
                if pred.strip() == gold.strip():
                    correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- MCQ evaluation may overestimate model capabilities due to option elimination or chance selection; free-form generation reveals a significant performance drop.
- Frame sampling is not uniformly controllable across all model APIs (e.g., Gemini), potentially introducing input distribution shifts that affect temporal reasoning.
- Averaging Accuracy@5 across 5 generations may smooth out catastrophic failures in long-horizon reasoning tasks.

## Evidence (verbatim from paper)

> We evaluate in a zero-shot setting using each MLLM’s default interface. Following prior work in MLLM evaluation, we adopt a fixed decoding configuration across MLLMs with temperature 0.7, top-p =1.0, and top-k =40. We evaluate stochastic decoding stability by reporting Accuracy@k, defined as the average accuracy across k independent generations per query. In our experiments, we set k=5 and report Accuracy@5 for both MCQ and numerical-answer items.

## Citation

```bibtex
@misc{bang2026vaexbench,
  title={Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence},
  author={Bang et al. (2026)},
  year={2026},
  note={arXiv:2603.13091}
}
```

- arXiv: 2603.13091