Videoaesbench Eval

---
name: videoaesbench-eval
description: Evaluates large multimodal models' ability to perceive and judge video aesthetics across visual form, style, and affectiveness dimensions. It tests performance on diverse video sources (UGC, AIGC, RGC, compression, gaming) using multiple-choice, true/false, and open-ended questions. Use when the user wants to benchmark on VideoAesBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.21915
  bibtex_key: li2026videoaesbench
  confidence: high
---

# videoaesbench-eval

> VideoAesBench: Benchmarking the Video Aesthetics Perception Capabilities of Large Multimodal Models — Li et al. (2026) (arXiv:2601.21915, 2026)

## What this evaluates

Evaluates large multimodal models' ability to perceive and judge video aesthetics across visual form, style, and affectiveness dimensions. It tests performance on diverse video sources (UGC, AIGC, RGC, compression, gaming) using multiple-choice, true/false, and open-ended questions.

## Datasets

- **VideoAesBench** — total 1804; splits: test (1804)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. For closed-ended questions (True/False, single/multiple choice), a prediction is correct if it exactly matches the ground truth option(s). For open-ended questions, scores are typically derived via LLM-judge or human evaluation against reference captions, though the exact automated protocol is not detailed in the provided text.

## Input / output format

**Input**: A video clip paired with a question prompt. Questions are categorized as True/False, single choice (4 options), multiple choice (4 options, >1 correct), or open-ended.

**Output**: For True/False: "True" or "False". For single/multiple choice: the selected option letter(s) or text. For open-ended: free-text aesthetic description or suggestion.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred.strip().lower() == gold.strip().lower():
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- True/False questions must be balanced in ratio to avoid bias from unbalanced answer distributions.
- Multiple-choice questions require selecting *all* correct options; partial credit is not mentioned, so exact set matching is likely required.
- Open-ended questions lack predefined answer sets, making automated scoring unreliable without explicit LLM-judge prompts or human annotators.

## Evidence (verbatim from paper)

> The goal of LMMs is to select the most accurate answer from the set of options including distractors. This type of question can detailedly assess the capability of LMMs in understanding and distinguishing diverse aesthetics issues.

## Citation

```bibtex
@misc{li2026videoaesbench,
  title={VideoAesBench: Benchmarking the Video Aesthetics Perception Capabilities of Large Multimodal Models},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2601.21915}
}
```

- arXiv: 2601.21915