Text Video Alignment Eval

---
name: text-video-alignment-eval
description: This evaluation protocol assesses how well text-to-video generation models align generated content with textual prompts across fine-grained attributes like object counts, colors, actions, and spatial relationships. It measures both semantic alignment and visual/motion quality to determine if refinement techniques successfully correct misalignments without degrading fidelity. Use when the user wants to benchmark on EvalCrafter, T2V-CompBench, or asks about evaluating this task. Reports Text-Video Alignment.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.15115
  bibtex_key: lee2024videorepair
  confidence: high
---

# text-video-alignment-eval

> VideoRepair: Improving Text-to-Video Generation via Misalignment Evaluation and Localized Refinement — Daeun Lee et al. (2024) (arXiv:2411.15115, 2024)

## What this evaluates

This evaluation protocol assesses how well text-to-video generation models align generated content with textual prompts across fine-grained attributes like object counts, colors, actions, and spatial relationships. It measures both semantic alignment and visual/motion quality to determine if refinement techniques successfully correct misalignments without degrading fidelity.

## Datasets

- **EvalCrafter** — total ?; splits: count (-1), color (-1), action (-1), others (-1); repo https://github.com/evalcrafter/EvalCrafter/blob/master/metadata.json
- **T2V-CompBench** — total 300; splits: consistent_attribute_binding (100), spatial_relationship (100), numeracy (100)

## Metrics

- `Text-Video Alignment` **(primary)** — range: [0, 1]
  - Average of six component scores: CLIP-Score, SD-Score, BLIP-BLEU, Detection-Score, Count-Score, and Color-Score.
- `Video Quality` — range: [0, 1]
  - Average of Video Quality Assessment score and Inception Score.
- `Motion Quality` — range: [0, 1]
  - Weighted average of Action Recognition score (VideoMAE) and Average Flow score.
- `Temporal Consistency` — range: [0, 1]
  - Combination of Warping Error (optical flow) and CLIP-Temp.
- `T2V-CompBench Category Score` — range: [0, 1]
  - Evaluated per category using ImageGrid-LLaVA for attribute binding, and GroundingDINO for spatial relationships and numeracy.

## Input / output format

**Input**: Text prompt and an initial video generated by a text-to-video model.

**Output**: Refined video output after applying the localized refinement pipeline.

## Scoring recipe

```python
def compute_text_video_alignment(video, prompt):
    scores = [
        clip_score(video, prompt),
        sd_score(video, prompt),
        blip_bleu(video, prompt),
        detection_score(video, prompt),
        count_score(video, prompt),
        color_score(video, prompt)
    ]
    return sum(scores) / len(scores)
```

## Common pitfalls

- SLD's frame-level latent merging often breaks object count and spatial consistency across frames, artificially lowering alignment scores.
- OPT2I only optimizes in text space without spatial guidance, capping alignment improvements regardless of prompt paraphrasing.
- Composite metrics like Text-Video Alignment average multiple heterogeneous evaluators; reporting only the aggregate hides category-specific failures.

## Evidence (verbatim from paper)

> For evaluation metrics in EvalCrafter, we mainly adopt the overall text-video alignment, video quality scores, motion quality, and temporal consistency metrics. Here, the text-video alignment score is defined as an average of CLIP-Score, SD-Score, BLIP-BLEU, Detection-Score, Count-Score, and Color-Score.

## Citation

```bibtex
@misc{lee2024videorepair,
  title={VideoRepair: Improving Text-to-Video Generation via Misalignment Evaluation and Localized Refinement},
  author={Daeun Lee et al. (2024)},
  year={2024},
  note={arXiv:2411.15115}
}
```

- arXiv: 2411.15115