Twiff Bench Eval

---
name: twiff-bench-eval
description: Evaluates a model's ability to perform dynamic visual reasoning by generating temporally grounded, physically plausible future frames and textual explanations. It probes both the quality of the step-by-step reasoning process and the correctness of the final answer in open-ended video scenarios. Use when the user wants to benchmark on TwiFF-Bench, Seed-Bench-R1, or asks about evaluating this task. Reports Answer score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.10675
  bibtex_key: liu2026twiff
  confidence: high
---

# twiff-bench-eval

> TwiFF (Think With Future Frames): A Large-Scale Dataset for Dynamic Visual Reasoning — Liu et al. (2026) (arXiv:2602.10675, 2026)

## What this evaluates

Evaluates a model's ability to perform dynamic visual reasoning by generating temporally grounded, physically plausible future frames and textual explanations. It probes both the quality of the step-by-step reasoning process and the correctness of the final answer in open-ended video scenarios.

## Datasets

- **TwiFF-Bench** — total ?; splits: test (-1); repo https://github.com/LiuJunhua02/TwiFF
- **Seed-Bench-R1** — total ?; splits: test (-1)

## Metrics

- `CoT score` — range: [0, 5]
  - LLM-judged score from 0 to 5 evaluating the plausibility, temporal alignment, and physical correctness of the generated reasoning chain (interleaved text and future frames).
- `Answer score` **(primary)** — range: [0, 5]
  - LLM-judged score from 0 to 5 evaluating the correctness and accuracy of the model's final answer to the prompt.

## Input / output format

**Input**: Video clips or sequential frames paired with a textual question or instruction requiring dynamic reasoning.

**Output**: Interleaved sequence of generated future frames and textual reasoning steps, terminated by a final answer.

## Scoring recipe

```python
def score_model_output(model_output, ground_truth_video, question):
    # LLM judge (GPT-5.1) evaluates CoT and Answer separately
    cot_score = llm_judge.score(
        prompt=scoring_prompt_cot,
        context={"question": question, "video": ground_truth_video, "reasoning": model_output.reasoning_chain}
    ) # Returns float 0-5
    ans_score = llm_judge.score(
        prompt=scoring_prompt_ans,
        context={"question": question, "video": ground_truth_video, "answer": model_output.final_answer}
    ) # Returns float 0-5
    return cot_score, ans_score
```

## Common pitfalls

- Models may enter infinite loops during visual Chain-of-Thought generation; the protocol explicitly caps tool invocations at 5 and image generations at 8, truncating any excess.
- Seed-Bench-R1 lacks reference reasoning chains, so only the answer score is reported for this benchmark, unlike TwiFF-Bench which reports both CoT and answer scores.
- CoT score measures reasoning quality/alignment with future events, not answer correctness; conflating the two leads to misinterpretation of model capabilities.

## Evidence (verbatim from paper)

> On TwiFF-Bench, we evaluate both the CoT score and the answer score of models. On Seed-Bench-R1, due to the absence of reference reasoning chains, we evaluate only the answer score. All scores are bounded between 0 and 5. The specific details of the scoring prompt are provided in the Section B.2.

## Citation

```bibtex
@misc{liu2026twiff,
  title={TwiFF (Think With Future Frames): A Large-Scale Dataset for Dynamic Visual Reasoning},
  author={Liu et al. (2026)},
  year={2026},
  note={arXiv:2602.10675}
}
```

- arXiv: 2602.10675