Vivd 10m Eval

---
name: vivd-10m-eval
description: Evaluates video editing models on local, entity-level modifications (addition, modification, deletion) by measuring background preservation, text alignment, temporal consistency, and visual quality. Use when the user wants to benchmark on VIVID-10M-Eval, or asks about evaluating this task. Reports Text Alignment (TA).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.15260
  bibtex_key: hu2024vivid10m
  confidence: high
---

# vivd-10m-eval

> VIVID-10M: A Dataset and Baseline for Versatile and Interactive Video Local Editing — Hu et al. (2024) (arXiv:2411.15260, 2024)

## What this evaluates

Evaluates video editing models on local, entity-level modifications (addition, modification, deletion) by measuring background preservation, text alignment, temporal consistency, and visual quality.

## Datasets

- **VIVID-10M-Eval** — total ?; splits: test (-1)

## Metrics

- `Background Preservation (BP)` — range: other
  - L1 distance between the original and edited videos in non-editing regions.
- `Text Alignment (TA)` **(primary)** — range: other
  - CLIP-score of the edited region.
- `Temporal Consistency (TC)` — range: [-1, 1]
  - Cosine similarity between consecutive frames in the CLIP-Image feature space.
- `Visual Quality (VQ)` — range: [0, 1]
  - Human-annotated win rate reflecting realness and aesthetics.

## Input / output format

**Input**: Input video frames, text caption/prompt, and a spatial mask defining the local editing region.

**Output**: Edited video frames matching the input resolution and frame count.

## Scoring recipe

```python
def compute_metrics(original_video, edited_video, mask, text_caption, clip_model):
    # Background Preservation (lower is better)
    bp = np.mean(np.abs(original_video - edited_video) * (1 - mask))
    # Text Alignment (higher is better)
    ta = clip_model.score(edited_region, text_caption)
    # Temporal Consistency (higher is better)
    tc = cosine_similarity(clip_model.encode(frame_t), clip_model.encode(frame_t+1))
    # Visual Quality (win rate from user study)
    vq = (wins + draws) / total_votes
    return bp, ta, tc, vq
```

## Common pitfalls

- BP is a distance metric where lower values indicate better performance, which is counterintuitive for typical accuracy metrics.
- TC measures semantic consistency via CLIP features rather than pixel-level stability, so it may overlook visual jitter or flicker.
- User study results are reported as 'win-or-draw' rates rather than strict win rates, affecting direct comparison with other benchmarks.

## Evidence (verbatim from paper)

> Background Preservation (BP): the L1 distance between the original and edited videos in non-editing regions. Text Alignment (TA): the CLIP-score*[[37], [10]]* of the edited region. Temporal Consistency (TC): the cosine similarity between consecutive frames in the CLIP-Image feature space*[[37]]*.

## Citation

```bibtex
@misc{hu2024vivid10m,
  title={VIVID-10M: A Dataset and Baseline for Versatile and Interactive Video Local Editing},
  author={Hu et al. (2024)},
  year={2024},
  note={arXiv:2411.15260}
}
```

- arXiv: 2411.15260