Opens2v Eval

---
name: opens2v-eval
description: Evaluates Subject-to-Video (S2V) generation models on their ability to maintain subject identity consistency, produce natural temporal dynamics, and align with text prompts. It covers open-domain, human-specific, and single-subject scenarios to expose common failure modes like copy-paste artifacts and fidelity degradation. Use when the user wants to benchmark on OpenS2V-Eval, or asks about evaluating this task. Reports NexusScore, NaturalScore, GmeScore.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.20292
  bibtex_key: yuan2025opens2vnexus
  confidence: high
---

# opens2v-eval

> OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation — Yuan et al. (2025) (arXiv:2505.20292, 2025)

## What this evaluates

Evaluates Subject-to-Video (S2V) generation models on their ability to maintain subject identity consistency, produce natural temporal dynamics, and align with text prompts. It covers open-domain, human-specific, and single-subject scenarios to expose common failure modes like copy-paste artifacts and fidelity degradation.

## Datasets

- **OpenS2V-Eval** — total 180; splits: test (180); repo https://github.com/PKU-YuanGroup/OpenS2V-Nexus

## Metrics

- `NexusScore` **(primary)** — range: percent
  - Measures subject consistency and identity fidelity between the reference image and the generated video.
- `NaturalScore` **(primary)** — range: percent
  - Evaluates temporal naturalness and detects copy-paste artifacts or unnatural pose/lighting replication.
- `GmeScore` **(primary)** — range: percent
  - Assesses text relevance and alignment between the input prompt and the generated video content.
- `Aesthetics` — range: percent
  - Measures overall visual quality using an improved aesthetic predictor.
- `Motion` — range: percent
  - Quantifies motion amplitude and dynamics in the generated video.
- `FaceSim` — range: percent
  - Computes face similarity between the reference subject and the generated video using FaceSim-Cur.
- `Total Score` — range: percent
  - A normalized weighted sum of the six dimension scores (Aesthetics, Motion, FaceSim, GmeScore, NexusScore, NaturalScore).

## Input / output format

**Input**: Reference subject image(s) and a text prompt specifying the desired video content, motion, and subject type.

**Output**: A generated video clip (typically 720p resolution, e.g., 81 frames × 720 × 1280 pixels).

## Scoring recipe

```python
def evaluate(generated_video, ref_image, prompt):
    aesthetics = run_aesthetic_predictor(generated_video)
    motion = run_motion_score(generated_video)
    facesim = run_facesim_cur(ref_image, generated_video)
    gme = run_gme_score(prompt, generated_video)
    nexus = run_nexus_score(ref_image, generated_video)
    natural = run_natural_score(generated_video)
    total = normalize_weighted_sum([aesthetics, motion, facesim, gme, nexus, natural])
    return {'Aesthetics': aesthetics, 'Motion': motion, 'FaceSim': facesim, 'GmeScore': gme, 'NexusScore': nexus, 'NaturalScore': natural, 'Total Score': total}
```

## Common pitfalls

- Models frequently exhibit 'copy-paste' issues, incorrectly replicating expressions, lighting, or poses from the reference image into the video.
- Subject fidelity tends to degrade over time or when multiple subjects are present in the prompt.
- Relying solely on coarse frame-level similarity metrics misses critical identity fidelity and temporal consistency.

## Evidence (verbatim from paper)

> Pika [pika] achieves the highest GmeScore, indicating that the generated videos are better aligned with the provided instructions. Kling [KeLing], on the other hand, produces videos with higher fidelity and realism, securing the highest NexusScore and NaturalScore.

## Citation

```bibtex
@misc{yuan2025opens2vnexus,
  title={OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation},
  author={Yuan et al. (2025)},
  year={2025},
  note={arXiv:2505.20292}
}
```

- arXiv: 2505.20292