Cctvbench Eval

---
name: cctvbench-eval
description: Probes multimodal LLMs' ability to answer binary questions about traffic videos while maintaining logical consistency across counterfactual video-question pairs. It specifically diagnoses failure modes like positive omission, negative hallucination, and mutual-exclusivity violations by enforcing a strict quadruple-level decision rule. Use when the user wants to benchmark on CCTVBench, or asks about evaluating this task. Reports QuadAcc.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.20460
  bibtex_key: [REDACTED]
  confidence: high
---

# cctvbench-eval

> CCTVBench: Contrastive Consistency Traffic VideoQA Benchmark for Multimodal LLMs — Zhou et al. (2026) (arXiv:2604.20460, 2026)

## What this evaluates

Probes multimodal LLMs' ability to answer binary questions about traffic videos while maintaining logical consistency across counterfactual video-question pairs. It specifically diagnoses failure modes like positive omission, negative hallucination, and mutual-exclusivity violations by enforcing a strict quadruple-level decision rule.

## Datasets

- **CCTVBench** — total ?; splits: test (-1)

## Metrics

- `QuadAcc` **(primary)** — range: percent
  - Percentage of quadruples where the model correctly answers Yes to (positive video, positive question), No to (positive video, negative question), No to (counterfactual video, positive question), and Yes to (counterfactual video, negative question). Enforces strict logical consistency across paired counterfactuals.
- `BaAcc` — range: percent
  - Standard binary accuracy on the primary positive video-question pairs.
- `MCCScore` — range: percent
  - Matthews Correlation Coefficient for binary classification, measuring the quality of binary predictions across all quadruple instances.

## Input / output format

**Input**: A traffic video (or its counterfactual counterpart) paired with a binary question. The prompt is formatted as: 'You are given a traffic video and a question. Answer based only on the video with exactly one word: Yes or No. Question: {QUESTION}'.

**Output**: A single word: 'Yes' or 'No'.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    # predictions and gold are dicts with keys: v+q+, v+q-, v-q+, v-q-
    # Expected: v+q+=Yes, v+q-=No, v-q+=No, v-q-=Yes
    quad_correct = (
        predictions['v+q+'] == 'Yes' and
        predictions['v+q-'] == 'No' and
        predictions['v-q+'] == 'No' and
        predictions['v-q-'] == 'Yes'
    )
    ba_correct = (
        predictions['v+q+'] == gold['v+q+'] and
        predictions['v-q-'] == gold['v-q-']
    )
    return {'QuadAcc': quad_correct, 'BaAcc': ba_correct}
```

## Common pitfalls

- High binary accuracy (BaAcc) or MCCScore does not guarantee logical consistency; models often pass isolated instances but fail the quadruple constraint.
- High rejection rates on counterfactuals (Reject@v-) or positive questions (Reject@q-) can mask positive omissions, where models incorrectly answer 'No' to true events.
- Confidence intervals are computed via scene-level bootstrap resampling (2,000 replicates), not instance-level, which may underestimate per-instance variance.

## Evidence (verbatim from paper)

> Tab. 1 shows that most models reach moderate binary QA quality, with BaAcc commonly in the 50–70% range and non-trivial MCCScore, yet QuadAcc remains low across the board.

## Citation

```bibtex
@misc{zhou2026cctvbench,
  title={CCTVBench: Contrastive Consistency Traffic VideoQA Benchmark for Multimodal LLMs},
  author={Zhou et al. (2026)},
  year={2026},
  note={arXiv:2604.20460}
}
```

- arXiv: 2604.20460