Videograin Eval

---
name: videograin-eval
description: Evaluates a video editing model's ability to perform multi-grained edits (class, instance, and part levels) on video-text pairs. It measures semantic alignment, temporal consistency, and pixel-level fidelity of the generated edits. Use when the user wants to benchmark on VideoGrain Evaluation Set, or asks about evaluating this task. Reports CLIP-T.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.17258
  bibtex_key: yang2025videograin
  confidence: high
---

# videograin-eval

> VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editing — Xiangpeng Yang et al. (arXiv:2502.17258, 2025)

## What this evaluates

Evaluates a video editing model's ability to perform multi-grained edits (class, instance, and part levels) on video-text pairs. It measures semantic alignment, temporal consistency, and pixel-level fidelity of the generated edits.

## Datasets

- **VideoGrain Evaluation Set** — total 76; splits: test (76)

## Metrics

- `CLIP-T` **(primary)** — range: [0, 100]
  - Average cosine similarity between the input text prompt and all video frames.
- `CLIP-F` — range: [0, 100]
  - Average cosine similarity between consecutive video frames.
- `Warp-Err` — range: [0, 100]
  - Pixel-level difference computed by warping edited frames using optical flow from the source video (extracted via RAFT-Large).
- `Q-edit` — range: [0, 100]
  - Ratio of CLIP-T to Warp-Err, used as a comprehensive measure of editing quality.

## Input / output format

**Input**: Source video frames, editing text prompt, and spatial region/mask information for target edits.

**Output**: Edited video frames matching the source duration and resolution.

## Scoring recipe

```python
def compute_metrics(frames, prompt, source_frames):
    clip_t = mean(cosine_similarity(prompt, frame) for frame in frames)
    clip_f = mean(cosine_similarity(frames[i], frames[i+1]) for i in range(len(frames)-1))
    flow = raft_large(source_frames)
    warped = warp(frames, flow)
    warp_err = mean_pixel_diff(frames, warped)
    q_edit = clip_t / warp_err
    return clip_t, clip_f, warp_err, q_edit
```

## Common pitfalls

- All automatic metrics are scaled by 100 in the paper, which may cause confusion if readers expect standard [0,1] ranges.
- Warp-Err relies on RAFT-Large optical flow; errors in flow estimation for fast motion or occlusions can artificially inflate the error score.
- Q-edit is a ratio metric; division by a near-zero Warp-Err can produce outlier values, though the paper scales results for clarity.

## Evidence (verbatim from paper)

> Four automatic metrics are employed for evaluation: CLIP-T, CLIP-F, Warp-Err, and Q-edit, following *(Wu et al., [2022]; Cong et al., [2023])*. All metrics are scaled by 100 for clarity. CLIP-T calculates the average cosine similarity between the input prompt and all video frames, while CLIP-F measures the average cosine similarity between consecutive frames. Additionally, Warp-Err captures pixel-level differences by warping the edited video frames according to the optical flow of the source video, extracted using RAFT-Large *(Teed \& Deng, [2020])*. To provide a more comprehensive measure of video editing quality, we follow *(Cong et al., [2023])* and use Q-edit, defined as CLIP-T/Warp-Err.

## Citation

```bibtex
@misc{yang2025videograin,
  title={VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editing},
  author={Xiangpeng Yang et al.},
  year={2025},
  note={arXiv:2502.17258}
}
```

- arXiv: 2502.17258