Video Analytics Eval

---
name: video-analytics-eval
description: Evaluates long-video understanding and agentic retrieval capabilities by testing models on temporal grounding, summarization, reasoning, and event causality across ultra-long video streams. Use when the user wants to benchmark on LVBench, VideoMME-Long, Ava-100, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.00254
  bibtex_key: yan2025ava
  confidence: high
---

# video-analytics-eval

> AVA: Towards Agentic Video Analytics with Vision Language Models — Yan et al. (2025) (arXiv:2505.00254, 2025)

## What this evaluates

Evaluates long-video understanding and agentic retrieval capabilities by testing models on temporal grounding, summarization, reasoning, and event causality across ultra-long video streams.

## Datasets

- **LVBench** — total 1549; splits: test (1549)
- **VideoMME-Long** — total 900; splits: test (900)
- **Ava-100** — total 120; splits: test (120)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions out of the total number of questions.

## Input / output format

**Input**: A long video (duration varies from ~4100s to >10 hours) paired with a multiple-choice question.

**Output**: A single selected option from the provided multiple-choice answers.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold) * 100
```

## Common pitfalls

- Videos are extremely long (up to >10 hours), requiring specialized indexing or retrieval rather than naive frame sampling.
- Questions cover diverse task types (temporal grounding, reasoning, etc.), so overall accuracy may mask performance on specific capabilities.
- Ava-100 is a newly proposed benchmark with manually annotated questions and GPT-4o-generated distractors, so results may not be directly comparable to older benchmarks without careful protocol alignment.

## Evidence (verbatim from paper)

> The accuracy is evaluated by analyzing Ava’s responses to multiple-choice questions included in the benchmarks.

## Citation

```bibtex
@misc{yan2025ava,
  title={AVA: Towards Agentic Video Analytics with Vision Language Models},
  author={Yan et al. (2025)},
  year={2025},
  note={arXiv:2505.00254}
}
```

- arXiv: 2505.00254