Void Eval

---
name: void-eval
description: Evaluates a video generation model's ability to remove specified objects and their downstream physical interactions (e.g., collisions, shadows, reflections) while maintaining temporal consistency and visual quality. It probes counterfactual reasoning and intuitive physics simulation in dynamic scenes. Use when the user wants to benchmark on Real-world object removal dataset, Synthetic counterfactual dataset, or asks about evaluating this task. Reports Win %.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.02296
  bibtex_key: motamed2026void
  confidence: high
---

# void-eval

> VOID: Video Object and Interaction Deletion — Motamed et al. (2026) (arXiv:2604.02296, 2026)

## What this evaluates

Evaluates a video generation model's ability to remove specified objects and their downstream physical interactions (e.g., collisions, shadows, reflections) while maintaining temporal consistency and visual quality. It probes counterfactual reasoning and intuitive physics simulation in dynamic scenes.

## Datasets

- **Real-world object removal dataset** — total 75; splits: test (75)
- **Synthetic counterfactual dataset** — total 40; splits: test (40)

## Metrics

- `Win %` **(primary)** — range: percent
  - Percentage of times a model's output is selected as the best by human evaluators across randomized comparisons.
- `VLM-Judge Score` — range: [0, 30]
  - Sum of scores across six criteria (Interaction & Physics, Object Removal, Background & Artifacts, Temporal Consistency, Preservation, Sharpness), each rated 0-5 by a vision-language model.
- `PSNR` — range: other
  - Peak Signal-to-Noise Ratio computed between generated and ground-truth frames.
- `LPIPS` — range: [0, 1]
  - Learned Perceptual Image Patch Similarity, frame-wise perceptual distance.
- `DreamSim` — range: [0, 1]
  - Frame-wise perceptual metric capturing semantic similarity.
- `DINOv2` — range: [0, 1]
  - Frame-wise feature similarity metric.
- `FVD` — range: other
  - Fréchet Video Distance, measuring video-level distribution similarity.

## Input / output format

**Input**: Input video sequence, sparse user clicks converted to a binary object mask and then a quadmask, and optionally a natural language prompt specifying the object to remove and expected post-removal scene evolution.

**Output**: A counterfactual video sequence where the specified object and its physical effects are removed, and the background/other objects are inpainted consistently.

## Scoring recipe

```python
# Human Preference
wins = sum(1 for p in participants for s in sample(test_set, 5) if human_prefers(model_out, s))
win_pct = (wins / (len(participants) * 5)) * 100

# VLM Judge
score = 0
for crit in ["Interaction & Physics", "Object Removal", "Background & Artifacts", "Temporal Consistency", "Preservation", "Sharpness"]:
    score += vlm_judge.evaluate(output, input_or_gt, crit, scale=5)
```

## Common pitfalls

- LPIPS penalizes correct counterfactual effects if they are generated in slightly shifted regions due to local translation sensitivity.
- Human preference study only samples 5 out of 75 scenarios per participant, potentially missing rare interaction edge cases.
- VLM judges may exhibit scoring bias or inconsistency across different model families (e.g., Gemini vs GPT vs Qwen).

## Evidence (verbatim from paper)

> We follow previous work in reporting pixel-based metric PSNR and perceptual metric LPIPS [42]. However, with the new dataset introducing more diverse sets of effects, we also add in the more recent frame-wise perceptual metrics DreamSim [7] and DINOv2 [30], as well as video metric FVD [36].

## Citation

```bibtex
@misc{motamed2026void,
  title={VOID: Video Object and Interaction Deletion},
  author={Motamed et al. (2026)},
  year={2026},
  note={arXiv:2604.02296}
}
```

- arXiv: 2604.02296