Oscbench Eval

---
name: oscbench-eval
description: This benchmark probes a text-to-video model's ability to accurately render and maintain object state transformations (e.g., peeling, slicing) over time. It additionally measures semantic adherence to prompts, scene consistency, and overall perceptual quality to diagnose temporal coherence and physical realism in generated videos. Use when the user wants to benchmark on OSCBench, or asks about evaluating this task. Reports state-change accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.11698
  bibtex_key: han2026oscbench
  confidence: high
---

# oscbench-eval

> OSCBench: Benchmarking Object State Change in Text-to-Video Generation — Han et al. (2026) (arXiv:2603.11698, 2026)

## What this evaluates

This benchmark probes a text-to-video model's ability to accurately render and maintain object state transformations (e.g., peeling, slicing) over time. It additionally measures semantic adherence to prompts, scene consistency, and overall perceptual quality to diagnose temporal coherence and physical realism in generated videos.

## Datasets

- **OSCBench** — total 140; splits: test (140)

## Metrics

- `state-change accuracy` **(primary)** — range: [1, 5]
  - Average score on a 1–5 Likert scale indicating whether the object reaches the correct target state implied by the prompt.
- `state-change consistency` — range: [1, 5]
  - Average score on a 1–5 Likert scale assessing whether the transformation unfolds smoothly over time without abrupt jumps or unnatural appearances.
- `Semantic Adherence` — range: [1, 5]
  - Average score on a 1–5 Likert scale measuring alignment of subject, object, and action entities from the prompt in the generated video.
- `Scene Alignment` — range: [1, 5]
  - Average score on a 1–5 Likert scale evaluating whether the global environment and background context match the prompt description.
- `Perceptual Quality` — range: [1, 5]
  - Average score on a 1–5 Likert scale reflecting overall visual impression, including realism and aesthetic composition.

## Input / output format

**Input**: A text prompt specifying an object state change scenario, paired with the generated video to be evaluated.

**Output**: A discrete integer score from 1 to 5 for each evaluation dimension.

## Scoring recipe

```python
def compute_metric(scores_1_5):
    # scores_1_5: list of 1-5 ratings from 3 human evaluators or MLLM CoT output
    # For humans: average across 3 independent ratings
    # For MLLMs: score derived from CoT reasoning (criteria grounding -> evidence extraction -> score decision)
    return sum(scores_1_5) / len(scores_1_5)
```

## Common pitfalls

- Using coarse text-video similarity models (e.g., CLIP, ViCLIP) instead of reasoning-based evaluators, as they fail to capture fine-grained object state transitions.
- Evaluating all generated videos exhaustively with humans is prohibitively costly; the benchmark uses a representative sampling strategy (1 prompt per scenario) rather than full coverage.
- MLLM evaluators must be prompted with Chain-of-Thought reasoning; without it, they are easily distracted by salient but irrelevant visual details and miss temporal state changes.

## Evidence (verbatim from paper)

> state-change accuracy, which measures whether the object reaches the correct target state implied by the prompt (e.g., a whole apple becoming sliced), and state-change consistency, which assesses whether the transformation unfolds smoothly and coherently over time, without abrupt jumps or unnatural object appearances or unexplained appearance or disappearance of object parts.

## Citation

```bibtex
@misc{han2026oscbench,
  title={OSCBench: Benchmarking Object State Change in Text-to-Video Generation},
  author={Han et al. (2026)},
  year={2026},
  note={arXiv:2603.11698}
}
```

- arXiv: 2603.11698