Openve Bench Eval

---
name: openve-bench-eval
description: Evaluates instruction-guided video editing models on spatially-aligned and non-spatially-aligned editing tasks, probing temporal consistency, spatial fidelity, and instruction following. Use when the user wants to benchmark on OpenVE-Bench, or asks about evaluating this task. Reports overall score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.07826
  bibtex_key: he2025openve3m
  confidence: high
---

# openve-bench-eval

> OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editing — He et al. (2025) (arXiv:2512.07826, 2025)

## What this evaluates

Evaluates instruction-guided video editing models on spatially-aligned and non-spatially-aligned editing tasks, probing temporal consistency, spatial fidelity, and instruction following.

## Datasets

- **OpenVE-Bench** — total ?; splits: test (-1)

## Metrics

- `overall score` **(primary)** — range: other
  - LLM-based evaluation using Seed-1.6VL and Gemini 2.5 Pro evaluators, averaged into a single overall score. Scores appear to be on a 1–5 scale based on ablation table values.

## Input / output format

**Input**: Original video, editing instruction text, and optionally control signals (e.g., Canny edge maps or depth maps).

**Output**: Edited video that applies the requested changes while preserving temporal and spatial consistency.

## Scoring recipe

```python
def compute_score(predictions, gold):
    scores_seed = run_llm_judge("Seed-1.6VL", predictions, gold)
    scores_gemini = run_llm_judge("Gemini 2.5 Pro", predictions, gold)
    return average(scores_seed + scores_gemini)
```

## Common pitfalls

- Baseline models have strict resolution/frame limits (e.g., OmniVideo capped at 640x352/17 frames, ICVE downsampled to 384x240 to avoid OOM).
- Closed-source Runway Aleph was only evaluated on 30 samples per category due to API cost constraints, which may not represent full benchmark performance.

## Evidence (verbatim from paper)

> Tab. 2 and 3 present the evaluation results of all instruction-following video editing models on OpenVE-Bench. The closed-source model, Runway Aleph, achieves the best performance on both the Seed-1.6VL [34] and Gemini 2.5 Pro [10] evaluators, significantly outperforming existing open-source models. The open-source models VACE [17], OmniVideo [33], and InsViE [44] exhibit limited performance due to constraints in their model size or training data. With 5B parameters, Lucy-Edit [35] achieves moderate performance. ICVE [25], with 13B parameters, obtains respectable results, but its high-resolution editing capability is limited to a smaller number of frames. Ditto [2] scores highly on the global style metric, as its training dataset primarily consists of this edit type. Our model, OpenVE-Edit, with only 5B parameters, achieves an overall score of 2.41, surpassing all existing open-source models with a smaller parameter count.

## Citation

```bibtex
@misc{he2025openve3m,
  title={OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editing},
  author={He et al. (2025)},
  year={2025},
  note={arXiv:2512.07826}
}
```

- arXiv: 2512.07826