Do Bench Eval

---
name: do-bench-eval
description: DO-Bench probes object hallucination in vision-language models by disentangling it into two distinct failure mechanisms: prior-dominated (textual priors overriding visual evidence) and perception-limited (weak visual grounding causing false denials). It uses controlled within-image interventions to measure how models respond to strengthened contextual priors and concentrated visual evidence, revealing heterogeneous failure modes that aggregate accuracy masks. Use when the user wants to benchmark on DO-Bench, or asks about evaluating this task. Reports PerceptionAbility, PriorRobust.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.22822
  bibtex_key: wang2026dobench
  confidence: high
---

# do-bench-eval

> DO-Bench: An Attributable Benchmark for Diagnosing Object Hallucination in Vision-Language Models — Wang et al. (2026) (arXiv:2604.22822, 2026)

## What this evaluates

DO-Bench probes object hallucination in vision-language models by disentangling it into two distinct failure mechanisms: prior-dominated (textual priors overriding visual evidence) and perception-limited (weak visual grounding causing false denials). It uses controlled within-image interventions to measure how models respond to strengthened contextual priors and concentrated visual evidence, revealing heterogeneous failure modes that aggregate accuracy masks.

## Datasets

- **DO-Bench** — total 1240; splits: test (1240)

## Metrics

- `Acc` — range: percent
  - Standard accuracy over the unified yes/no answering protocol. Calculated as the fraction of instances where the normalized model prediction matches the ground truth label.
- `F1` — range: percent
  - F1 score computed over the yes/no predictions, balancing precision and recall across the dataset.
- `PerceptionAbility` **(primary)** — range: percent
  - Measures recognition recoverability under enhanced visual evidence at the baseline prompt (A0). Aggregates false-negative rates under Cluster and Crop views; higher values indicate stronger perceptual grounding and recoverability when localized evidence is provided.
- `PriorRobust` **(primary)** — range: percent
  - Aggregates robustness against prior-driven denial (A-series) and prior-driven completion (B-series). Computed via AUC aggregation over false-negative and false-positive rates across progressively strengthened contextual prior levels (A0–A3 and B0–B3). Higher values indicate greater resistance to textual prior override.

## Input / output format

**Input**: An image paired with a text prompt. The prompt varies by experimental condition: prior strength levels (A0–A3 for present objects, B0–B3 for absent objects) and visual evidence concentration (Full, Cluster, or Crop views). All instances follow a unified yes/no answering protocol.

**Output**: A deterministic yes/no answer (normalized to match ground truth labels).

## Scoring recipe

```python
def score(predictions, golds, conditions):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    acc = correct / len(golds) * 100
    
    # PerceptionAbility: A-series at baseline prompt (A0)
    fn_cluster = sum(1 for p, g in zip(predictions, golds) if p != g and g == 'Yes' and conditions == 'Cluster')
    fn_crop = sum(1 for p, g in zip(predictions, golds) if p != g and g == 'Yes' and conditions == 'Crop')
    perception_ability = 100 - (fn_cluster + fn_crop) / (2 * total_A0) * 100
    
    # PriorRobust: AUC of error rates across prior levels A0-A3
    fn_rates = [count_false_negatives(level) for level in ['A0','A1','A2','A3']]
    fp_rates = [count_false_positives(level) for level in ['B0','B1','B2','B3']]
    prior_robust = 100 - (auc(fn_rates) + auc(fp_rates)) / 2 * 100
    return acc, perception_ability, prior_robust
```

## Common pitfalls

- Relying solely on aggregate accuracy or F1 conflates qualitatively different failure modes; a model can have high accuracy but low PriorRobust, indicating severe susceptibility to textual priors.
- Assuming model responses must be strictly monotonic across prior strength or evidence concentration levels; the benchmark explicitly allows non-monotonicity and uses AUC aggregation to remain robust to discrete decoding effects.
- Treating perceptual recoverability and prior robustness as interchangeable; they are orthogonal dimensions, with evidence concentration primarily shifting prior-response curves vertically rather than altering their shape.

## Evidence (verbatim from paper)

> PerceptionAbility measures recognition under enhanced visual evidence under the baseline prompt (A0). PriorRobust aggregates robustness against prior-driven denial (A-series) and prior-driven completion (B-series). All values are in %.

## Citation

```bibtex
@misc{wang2026dobench,
  title={DO-Bench: An Attributable Benchmark for Diagnosing Object Hallucination in Vision-Language Models},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.22822}
}
```

- arXiv: 2604.22822