Cow Bench Eval

---
name: cow-bench-eval
description: Evaluates modal, spatial, and temporal consistency in general world models through 18 sub-tasks across six task categories. It uses human-designed checklists to verify fine-grained physical laws, causal reasoning, and cross-modal alignment, moving beyond perceptual metrics to hard verification. Use when the user wants to benchmark on CoW-Bench, or asks about evaluating this task. Reports checklist_score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.23152
  bibtex_key: wei2026trinity
  confidence: medium
---

# cow-bench-eval

> The Trinity of Consistency as a Defining Principle for General World Models — Wei et al. (2026) (arXiv:2602.23152, 2026)

## What this evaluates

Evaluates modal, spatial, and temporal consistency in general world models through 18 sub-tasks across six task categories. It uses human-designed checklists to verify fine-grained physical laws, causal reasoning, and cross-modal alignment, moving beyond perceptual metrics to hard verification.

## Datasets

- **CoW-Bench** — total ?; splits: test (-1)

## Metrics

- `checklist_score` **(primary)** — range: percent
  - Aggregated pass rate across five human-designed checklists per sub-task. Each checklist verifies a specific physical, logical, or cross-modal constraint, with scores normalized to a 0-100% scale.

## Input / output format

**Input**: Text prompts, images, and video sequences designed to test modal, spatial, and temporal consistency, including counterfactual interventions and multi-frame reasoning tasks.

**Output**: Generated images/videos or reasoning traces, evaluated against five checklist criteria per sub-task to verify physical laws, causal chains, and cross-modal alignment.

## Scoring recipe

```python
def evaluate_cow_bench(predictions, gold, checklists):
    total_checks = 0
    passed_checks = 0
    for task in predictions:
        for checklist_item in checklists[task.subtask]:
            total_checks += 1
            if verify_physical_logic(task.output, checklist_item):
                passed_checks += 1
    return (passed_checks / total_checks) * 100
```

## Common pitfalls

- Relies on human checklists rather than automated differentiable metrics, making large-scale evaluation costly and subjective.
- Current benchmarks it critiques (e.g., TiViBench, V-ReasonBench) over-rely on VLM-as-a-judge, which fails on fine-grained physical attributes like friction or momentum conservation.
- Long-range generation errors (butterfly effect) are masked by short-sequence testing protocols common in prior work.

## Evidence (verbatim from paper)

> Each sub-task is paired with five carefully designed human checklists, yielding a comprehensive, task-driven protocol with fine-grained criteria to pinpoint complementary failure modes and enable more precise, interpretable quantification.

## Citation

```bibtex
@misc{wei2026trinity,
  title={The Trinity of Consistency as a Defining Principle for General World Models},
  author={Wei et al. (2026)},
  year={2026},
  note={arXiv:2602.23152}
}
```

- arXiv: 2602.23152