Cdh Bench Eval

---
name: cdh-bench-eval
description: This benchmark probes a vision-language model's ability to maintain visual fidelity when explicit visual evidence conflicts with strong commonsense priors. It specifically measures whether models override entrenched prior-driven expectations with counterfactual image-grounded claims, isolating hallucination from generic perception errors. Use when the user wants to benchmark on CDH-Bench, or asks about evaluating this task. Reports CFAD.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.27982
  bibtex_key: chen2026cdhbench
  confidence: high
---

# cdh-bench-eval

> CDH-Bench: A Commonsense-Driven Hallucination Benchmark for Evaluating Visual Fidelity in Vision-Language Models — Chen et al. (2026) (arXiv:2603.27982, 2026)

## What this evaluates

This benchmark probes a vision-language model's ability to maintain visual fidelity when explicit visual evidence conflicts with strong commonsense priors. It specifically measures whether models override entrenched prior-driven expectations with counterfactual image-grounded claims, isolating hallucination from generic perception errors.

## Datasets

- **CDH-Bench** — total ?; splits: test (-1); repo https://github.com/MiLab-HITSZ/2026ChenCDH-Bench

## Metrics

- `CF-Acc` — range: percent
  - Counterfactual Accuracy: the proportion of counterfactual image instances where the model's prediction matches the ground truth.
- `CS-Acc` — range: percent
  - Commonsense Accuracy: the proportion of matched commonsense control image instances where the model's prediction matches the ground truth.
- `CFAD` **(primary)** — range: percent
  - Counterfactual Accuracy Drop: the difference between commonsense accuracy and counterfactual accuracy (CS-Acc - CF-Acc). Lower values indicate less prior-driven collapse.
- `CCR` — range: percent
  - Commonsense Collapse Rate: the proportion of counterfactual multiple-choice instances where the model outputs the commonsense-consistent distractor instead of the correct counterfactual answer. Reported only for MC tasks.

## Input / output format

**Input**: Paired image-question instances. Each instance provides an image (either a counterfactual variant or a matched commonsense control) along with a text prompt containing a question that embeds a strong commonsense prior and an explicit counterfactual claim. Tasks are formatted as binary QA or multiple-choice QA.

**Output**: A single predicted answer per instance. For binary QA, a yes/no or true/false response. For multiple-choice QA, a selected option from the provided choices. Answers are extracted deterministically.

## Scoring recipe

```python
def compute_metrics(predictions, golds, is_counterfactual):
    cf_acc = sum(1 for p, g, is_cf in zip(predictions, golds, is_counterfactual) if is_cf and p == g) / sum(is_counterfactual)
    cs_acc = sum(1 for p, g, is_cf in zip(predictions, golds, is_counterfactual) if not is_cf and p == g) / sum(not is_cf)
    cfad = cs_acc - cf_acc
    return {'CF-Acc': cf_acc, 'CS-Acc': cs_acc, 'CFAD': cfad}
```

## Common pitfalls

- Interpreting a low CFAD as strong counterfactual grounding without checking CS-Acc, as a model with poor baseline accuracy can mechanically produce a small drop.
- Assuming multiple-choice QA inherently tests superior reasoning, when the benchmark actually shows it amplifies prior-driven failures by pitting counterfactual evidence against plausible commonsense distractors.
- Treating CFAD as a measure of generic perception error; the paired counterfactual-commonsense design specifically isolates prior-driven normalization bias rather than random noise or basic vision failures.

## Evidence (verbatim from paper)

> Concretely, 7 out of 8 models show lower overall CF-Acc than CS-Acc in both settings; the only partial exception is gemini-3.1-pro-preview, which exhibits a slightly negative CFAD in QA but still shows a 7.33% drop in MC.

## Citation

```bibtex
@misc{chen2026cdhbench,
  title={CDH-Bench: A Commonsense-Driven Hallucination Benchmark for Evaluating Visual Fidelity in Vision-Language Models},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2603.27982}
}
```

- arXiv: 2603.27982