Online Video Understanding Eval

---
name: online-video-understanding-eval
description: Evaluates a model's ability to perform real-time, streaming video understanding and long-horizon reasoning. It probes low-latency perception, temporal alignment, evidence-aligned response timing, and causal reasoning across short to hour-long video sequences. Use when the user wants to benchmark on StreamingBench, OVOBench, RTVBench, OVBench, VideoMME, MLVU, LongVideoBench, LVBench, or asks about evaluating this task. Reports QA accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.18459
  bibtex_key: zhang2026progressive
  confidence: high
---

# online-video-understanding-eval

> Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisions — Zhang et al. (2026) (arXiv:2604.18459, 2026)

## What this evaluates

Evaluates a model's ability to perform real-time, streaming video understanding and long-horizon reasoning. It probes low-latency perception, temporal alignment, evidence-aligned response timing, and causal reasoning across short to hour-long video sequences.

## Datasets

- **StreamingBench** — total ?; splits: test (-1)
- **OVOBench** — total ?; splits: test (-1)
- **RTVBench** — total ?; splits: test (-1)
- **OVBench** — total ?; splits: test (-1)
- **VideoMME** — total ?; splits: test (-1)
- **MLVU** — total ?; splits: test (-1)
- **LongVideoBench** — total ?; splits: test (-1)
- **LVBench** — total ?; splits: test (-1)

## Metrics

- `QA accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. Computed as (number of correct predictions / total number of questions) * 100. Follows official per-suite scoring protocols, which typically use exact match or LLM-based grading depending on the benchmark.

## Input / output format

**Input**: Video frames sampled at a specified frame rate (e.g., 1 fps) or progressive clip sequences, combined with a natural language question/query. Input resolution is capped at 448×448.

**Output**: Textual answer to the question. For online benchmarks, responses may include timestamped evidence or deferred answers until sufficient future frames are observed.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        # Follows official benchmark scoring (exact match or LLM-as-judge)
        if is_correct(pred, gold): 
            correct += 1
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Models must adhere to strict latency constraints and 'answer-when-ready' timing rules in online benchmarks, which many offline models fail to satisfy.
- Token budgets are tightly constrained (e.g., 32 frames, hierarchical aggregation tokens), so naive frame sampling or full-context processing will exceed limits and break the progressive evaluation protocol.

## Evidence (verbatim from paper)

> Our evaluation spans complementary online and offline long-video QA suites that jointly stress real-time perception, temporal alignment, and long-horizon reasoning. StreamingBench targets low-latency, timestamped queries under streaming constraints. OVOBench enforces answer-when-ready timing—models defer responses until sufficient future evidence (real-time perception, forward tracking, active responding). RTVBench and OVBench probe continuous perception and online spatio-temporal reasoning via multi-timestamp, hierarchical questions and Past/Current/Future anchoring. For offline long-form understanding, VideoMME, MLVU, LongVideoBench, and LVBench cover short clips to hour-long videos, emphasizing granular recall and cross-scale reasoning. We follow official scoring protocols (per-suite QA accuracy and aggregates); full task/metric definitions are in Appendix §A.5.

## Citation

```bibtex
@misc{zhang2026progressive,
  title={Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisions},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2604.18459}
}
```

- arXiv: 2604.18459