Vector Eval

---
name: vector-eval
description: Evaluates a model's ability to understand and reason about the temporal order of multiple events in long-form videos. It probes whether the model can correctly sequence events, identify relative ordering, and detect pattern anomalies across varying sequence lengths and difficulties. Use when the user wants to benchmark on VECTOR, or asks about evaluating this task. Reports EM (Exact Match).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.08979
  bibtex_key: ahn2025what
  confidence: high
---

# vector-eval

> What Happens When: Learning Temporal Orders of Events in Videos — Ahn et al. (2025) (arXiv:2512.08979, 2025)

## What this evaluates

Evaluates a model's ability to understand and reason about the temporal order of multiple events in long-form videos. It probes whether the model can correctly sequence events, identify relative ordering, and detect pattern anomalies across varying sequence lengths and difficulties.

## Datasets

- **VECTOR** — total ?; splits: test (-1)

## Metrics

- `EM (Exact Match)` **(primary)** — range: percent
  - Exact match accuracy: 1 if the predicted sequence of events exactly matches the ground-truth sequence, 0 otherwise. Averaged over all instances.

## Input / output format

**Input**: A video represented by 32 sampled frames, accompanied by a text prompt asking for the temporal order of events or pattern anomaly detection.

**Output**: A predicted ordered list of events or a textual answer specifying the sequence.

## Scoring recipe

```python
def compute_em(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Simply increasing the number of input frames beyond a threshold does not consistently improve performance and can even degrade it.
- Open-source VLMMs drop to chance-level accuracy on longer sequences (L2, 8 events), highlighting struggles with long-range temporal reasoning.
- Order-agnostic metrics (OM) remain stable across conditions, so relying solely on them masks failures in true temporal-order comprehension.

## Evidence (verbatim from paper)

> Unless specified, we use 32 input frames per video. Specifically, Table 2 reports EM scores for Tasks1-3, while Table 3 provides additional sequencing metrics. All models show notable performance declines from L1 to L2; for example, GPT-4o and Gemini drop from approximately 75–83% EM at L1 to 50–60% at L2.

## Citation

```bibtex
@misc{ahn2025what,
  title={What Happens When: Learning Temporal Orders of Events in Videos},
  author={Ahn et al. (2025)},
  year={2025},
  note={arXiv:2512.08979}
}
```

- arXiv: 2512.08979