Svbench Eval

---
name: svbench-eval
description: Evaluates large vision-language models' ability to perform sustained temporal reasoning and context tracking across long-form streaming videos. It probes multi-turn dialogue continuity, temporal dependency handling, and complex reasoning skills like counterfactual analysis and spatio-temporal speculation. Use when the user wants to benchmark on SVBench, or asks about evaluating this task. Reports Overall Score (OS).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.10810
  bibtex_key: yang2025svbench
  confidence: high
---

# svbench-eval

> SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding — Yang et al. (2025) (arXiv:2502.10810, 2025)

## What this evaluates

Evaluates large vision-language models' ability to perform sustained temporal reasoning and context tracking across long-form streaming videos. It probes multi-turn dialogue continuity, temporal dependency handling, and complex reasoning skills like counterfactual analysis and spatio-temporal speculation.

## Datasets

- **SVBench** — total 49979; splits: test (-1)

## Metrics

- `Overall Score (OS)` **(primary)** — range: percent
  - Aggregates scores from Semantic Accuracy (SA), Contextual Coherence (CC), Logical Consistency (LC), Temporal Understanding (TU), and Informational Completeness (IC) via an LLM-based judge. The authors do not provide a symbolic formula; OS is computed by averaging or weighting the five criterion scores.
- `METEOR` — range: percent
  - Evaluates precision, recall, and alignment of words and phrases between the references and the ground truth by considering synonymy and stemming.
- `GPT4-Score` — range: percent
  - Assesses the accuracy of generated answers solely based on the semantic similarity between a single answer and the ground truth, evaluated by GPT-4.

## Input / output format

**Input**: Video clip/frames, current question, and optionally a history of preceding QA pairs up to the current timestamp.

**Output**: Natural language answer to the question.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    scores = []
    for pred, gold in zip(predictions, golds):
        sa = llm_judge(pred, gold, 'Semantic Accuracy')
        cc = llm_judge(pred, gold, 'Contextual Coherence')
        lc = llm_judge(pred, gold, 'Logical Consistency')
        tu = llm_judge(pred, gold, 'Temporal Understanding')
        ic = llm_judge(pred, gold, 'Informational Completeness')
        os = aggregate(sa, cc, lc, tu, ic)
        meteor = compute_meteor(pred, gold)
        gpt4 = compute_gpt4_score(pred, gold)
        scores.append({'OS': os, 'METEOR': meteor, 'GPT4-Score': gpt4})
    return mean_over_scores(scores)
```

## Common pitfalls

- Confusing single-instance QA evaluation with the paper's multi-turn dialogue setup, which requires feeding historical QA pairs as context.
- Overlooking the 80% probabilistic temporal jump rule in streaming evaluation, which tests handling of non-linear context transitions.
- Assuming standard static video QA metrics apply; streaming evaluation demands continuous context integration across evolving video segments.

## Evidence (verbatim from paper)

> Overall Score (OS) is derived by aggregating the scores from each aforementioned criterion. Semantic Accuracy (SA) evaluates the accuracy of the generated answers based on a holistic understanding. It considers not only the direct overlap with ground-truth answers but also the context, coherence, and overall relevance of the response to the question posed.

## Citation

```bibtex
@misc{yang2025svbench,
  title={SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2502.10810}
}
```

- arXiv: 2502.10810