Force Prompting Eval

---
name: force-prompting-eval
description: Evaluates a video generation model's ability to adhere to specified physics-based force signals (local point forces or global wind fields) and produce visually realistic, physically plausible dynamics. It probes generalization across diverse objects, materials, and motion categories using human preference judgments. Use when the user wants to benchmark on Local Point Force Benchmark, Global Force Benchmark, or asks about evaluating this task. Reports 2AFC win rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.19386
  bibtex_key: gillman2025forceprompting
  confidence: high
---

# force-prompting-eval

> Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signals — Gillman et al. (2025) (arXiv:2505.19386, 2025)

## What this evaluates

Evaluates a video generation model's ability to adhere to specified physics-based force signals (local point forces or global wind fields) and produce visually realistic, physically plausible dynamics. It probes generalization across diverse objects, materials, and motion categories using human preference judgments.

## Datasets

- **Local Point Force Benchmark** — total 63; splits: test (63)
- **Global Force Benchmark** — total 41; splits: test (41)

## Metrics

- `2AFC win rate` **(primary)** — range: percent
  - Percentage of times the Force Prompting model's output is preferred over a baseline model's output in a two-alternative forced-choice human study. Calculated as (number of wins for Force Prompting / total comparisons) × 100%. Evaluated across three criteria: force adherence, realistic physics, and visual quality.

## Input / output format

**Input**: Single image frame and a physics-based force signal (local point force coordinates/strength or global wind field vector). Baselines receive only text prompts describing the force.

**Output**: A generated video clip depicting the object's motion in response to the conditioning signal.

## Scoring recipe

```python
wins = 0
total = 0
for image, force_signal in benchmark:
    video_fp = model.generate(image, force_signal)
    video_bl = baseline.generate(image, text_prompt)
    annotator_choice = human_2afc(video_fp, video_bl)
    if annotator_choice == video_fp:
        wins += 1
    total += 1
win_rate = (wins / total) * 100
```

## Common pitfalls

- Human study uses a small sample size (N=10), which may limit statistical power and generalizability of preference scores.
- Baseline models like Motion Prompting are evaluated with out-of-domain inputs (3-frame trajectories) for force simulation, potentially skewing win rates against them.
- Win rates are reported per motion category (e.g., Linear, Oscillatory) rather than globally; aggregating across categories requires weighted averaging by category size.

## Evidence (verbatim from paper)

> We present % win rates of our method against baselines in 2AFC human study results (i.e. values above 50% indicate a preference for Force Prompting) for force adherence, realistic physics, and visual quality.

## Citation

```bibtex
@misc{gillman2025forceprompting,
  title={Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signals},
  author={Gillman et al. (2025)},
  year={2025},
  note={arXiv:2505.19386}
}
```

- arXiv: 2505.19386