Vefx Bench Eval

---
name: vefx-bench-eval
description: Evaluates video editing reward models and VLM judges on their ability to align with human preferences across three dimensions: instruction following, rendering quality, and edit exclusivity. It tests both global score correlation and local pairwise preference consistency within candidate sets. Use when the user wants to benchmark on VEFX-Bench, or asks about evaluating this task. Reports SRCC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.16272
  bibtex_key: gao2026vefxbench
  confidence: high
---

# vefx-bench-eval

> VEFX-Bench: A Holistic Benchmark for Generic Video Editing and Visual Effects — Gao et al. (2026) (arXiv:2604.16272, 2026)

## What this evaluates

Evaluates video editing reward models and VLM judges on their ability to align with human preferences across three dimensions: instruction following, rendering quality, and edit exclusivity. It tests both global score correlation and local pairwise preference consistency within candidate sets.

## Datasets

- **VEFX-Bench** — total 5049; splits: train (4200), test (849)

## Metrics

- `SRCC` **(primary)** — range: [0, 1]
  - Spearman Rank-Order Correlation Coefficient between raw model predictions and human scores.
- `KRCC` — range: [0, 1]
  - Kendall Rank-Order Correlation Coefficient between raw model predictions and human scores.
- `PLCC` — range: [0, 1]
  - Pearson Linear Correlation Coefficient computed after applying a standard 4-parameter logistic calibration to raw predictions.
- `RMSE` — range: other
  - Root Mean Squared Error between calibrated predictions and human scores.
- `Pairwise Accuracy` — range: [0, 1]
  - Fraction of correctly ordered candidate pairs within each group. Ties in ground truth count as correct; ties in prediction score 0.5.

## Input / output format

**Input**: Source video, editing instruction, and edited output video.

**Output**: Scores on a 1–4 rubric for Instruction Following (IF), Rendering Quality (RQ), and Edit Exclusivity (EE), or an overall score (mean of the three dimensions).

## Scoring recipe

```python
import scipy.stats as stats
import numpy as np

def compute_metrics(preds, golds):
    srcc = stats.spearmanr(preds, golds).correlation
    krcc = stats.kendalltau(preds, golds).correlation
    # PLCC/RMSE require 4-param logistic calibration per Sec 15.1
    rmse = np.sqrt(np.mean((preds - golds)**2))
    return srcc, krcc, rmse

def pairwise_accuracy(groups, preds, golds):
    correct, total = 0, 0
    for g in groups:
        for i, j in pairs_in(g):
            if golds[i] == golds[j]: correct += 1
            elif (preds[i] > preds[j]) == (golds[i] > golds[j]): correct += 1
            elif preds[i] == preds[j]: correct += 0.5
            total += 1
    return correct / total
```

## Common pitfalls

- PLCC and RMSE must be computed after 4-parameter logistic calibration, not on raw predictions.
- Pairwise Accuracy is strictly computed within groups sharing the same source video and instruction, not globally across all candidates.
- The overall human score is the arithmetic mean of IF, RQ, and EE scores, not a separately annotated dimension.

## Evidence (verbatim from paper)

> Our primary evaluation follows standard IQA/VQA protocol. We report Spearman Rank-Order Correlation Coefficient (SRCC), Kendall Rank-Order Correlation Coefficient (KRCC), Pearson Linear Correlation Coefficient (PLCC), and Root Mean Squared Error (RMSE) in [Section˜5.2]. SRCC and KRCC are computed on raw predictions, while PLCC and RMSE are computed after the standard four-parameter logistic calibration. We complement these global correlation metrics with a group-wise preference metric, Pairwise Accuracy, in [Section˜5.3].

## Citation

```bibtex
@misc{gao2026vefxbench,
  title={VEFX-Bench: A Holistic Benchmark for Generic Video Editing and Visual Effects},
  author={Gao et al. (2026)},
  year={2026},
  note={arXiv:2604.16272}
}
```

- arXiv: 2604.16272