Causal2needles Eval

---
name: causal2needles-eval
description: Evaluates Video-Language Models' ability to perform joint retrieval and causal reasoning over two causally separated video clips connected by a 'bridge entity'. It also probes causal world modeling by asking models to identify cause-effect relationships in human behaviors within long videos. Use when the user wants to benchmark on Causal2Needles, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.19853
  bibtex_key: li2025causal2needles
  confidence: high
---

# causal2needles-eval

> Two Causally Related Needles in a Video Haystack — Li et al. (2025) (arXiv:2505.19853, 2025)

## What this evaluates

Evaluates Video-Language Models' ability to perform joint retrieval and causal reasoning over two causally separated video clips connected by a 'bridge entity'. It also probes causal world modeling by asking models to identify cause-effect relationships in human behaviors within long videos.

## Datasets

- **Causal2Needles** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. For 2-needle visual grounding questions, both Part 1 (effect clip) and Part 2 (cause clip) must be correctly identified to count as correct. Results are averaged over forward and reversed clip orders to mitigate positional bias.

## Input / output format

**Input**: A vertically stacked image composed of 5 sampled frames per video clip, accompanied by the full narration text and a question. Clips are presented in a randomized span around the target cause and effect clips to prevent location shortcuts.

**Output**: A clip number (integer) indicating the location of the cause or effect event. Models may also output 'None' if they cannot locate the clip.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred == gold:
            correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Positional bias: models heavily favor earlier clips in the sequence, artificially inflating accuracy for cause events in forward order.
- Static output bias: some open-source models output the same clip number regardless of the question content, leading to zero scores on complex queries.
- Textual bias: models may answer correctly using only the narration text without actually processing the video frames.
- Knowledge leakage: models might rely on memorized movie details from pretraining rather than visual grounding.

## Evidence (verbatim from paper)

> We report the accuracy on each type of questions. For VG 2-needle questions, we separately compute the accuracy for Part 1, Part 2, and both parts answered correctly. In addition, we evaluate each model using both the original (forward) and reversed video clip order, and report the average of the two orders as the final result on VG 2-needle questions.

## Citation

```bibtex
@misc{li2025causal2needles,
  title={Two Causally Related Needles in a Video Haystack},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2505.19853}
}
```

- arXiv: 2505.19853