Veu Bench Eval

---
name: veu-bench-eval
description: VEU-Bench evaluates a model's ability to understand video editing by probing 19 fine-grained tasks across 10 dimensions (e.g., shot size, cut types, transitions) and three cognitive stages: recognition, reasoning, and judging. It tests whether models can identify editing components, infer their functions, and judge their effects in video content. Use when the user wants to benchmark on VEU-Bench, or asks about evaluating this task. Reports Scoreall.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.17828
  bibtex_key: li2025veubench
  confidence: high
---

# veu-bench-eval

> VEU-Bench: Towards Comprehensive Understanding of Video Editing — Li et al. (2025) (arXiv:2504.17828, 2025)

## What this evaluates

VEU-Bench evaluates a model's ability to understand video editing by probing 19 fine-grained tasks across 10 dimensions (e.g., shot size, cut types, transitions) and three cognitive stages: recognition, reasoning, and judging. It tests whether models can identify editing components, infer their functions, and judge their effects in video content.

## Datasets

- **VEU-Bench** — total 50000; splits: train (-1), val (-1), test (-1)

## Metrics

- `Scoremc` — range: [1, 5]
  - Average score on the recognition task dimension, rated on a 1-5 scale based on human or LLM evaluation of answer quality.
- `Scoreoe` — range: [1, 5]
  - Average score on the reasoning and judging task dimensions, rated on a 1-5 scale based on human or LLM evaluation of answer quality.
- `Scoreall` **(primary)** — range: [1, 5]
  - Overall average score across all 19 tasks and 10 dimensions, rated on a 1-5 scale.

## Input / output format

**Input**: Video frames sampled at 1 fps (maximum 64 frames) paired with a text prompt containing a question about a specific video editing element (e.g., shot size, cut type, transition function).

**Output**: Natural language text response answering the prompt, identifying or reasoning about the specified editing component.

## Scoring recipe

```python
scores = []
for answer in predictions:
    score = rater.evaluate(answer, gold_reference)  # Returns 1-5
    scores.append(score)
metric_value = sum(scores) / len(scores)
return metric_value
```

## Common pitfalls

- Uniform frame sampling (1 fps) limits models' ability to perceive motion speed and dynamic changes, sometimes causing performance worse than random guessing.
- Models frequently confuse intra-frame content with inter-frame editing effects, leading to artificially high recognition scores but poor reasoning/judging performance.
- LLM evaluators tend to bias scores toward visual factual content rather than correctly judging editing intent unless pattern-matching regularization is applied.

## Evidence (verbatim from paper)

> Meanwhile, Oscars gains 28.3% higher performance in Scoreall compared to SOTA open-source model LLaVA-OneVision*[[19]]*. In the more challenging dimensions cut and transition, Oscars demonstrates significantly better performance than the open-source Vid-LLM state-of-the-art, with improvements of 12.2% and 10.9% on Scoremc and Scoreoe respectively, showing exceeding video editing understanding ability.

## Citation

```bibtex
@misc{li2025veubench,
  title={VEU-Bench: Towards Comprehensive Understanding of Video Editing},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2504.17828}
}
```

- arXiv: 2504.17828