Vib Probe Hallucination Eval

---
name: vib-probe-hallucination-eval
description: Evaluates the ability of Vision-Language Models to avoid generating unfaithful or non-existent visual details in both closed-set discriminative QA and open-ended generative captioning. It also measures how well an auxiliary probing framework can detect these hallucinations via attention dynamics and mitigate them at inference time without retraining. Use when the user wants to benchmark on POPE, AMBER, M-HalDetect, COCO-Caption, or asks about evaluating this task. Reports AUPRC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.05547
  bibtex_key: zhang2026vibprobe
  confidence: high
---

# vib-probe-hallucination-eval

> VIB-Probe: Detecting and Mitigating Hallucinations in Vision-Language Models via Variational Information Bottleneck — Zhang et al. (2026) (arXiv:2601.05547, 2026)

## What this evaluates

Evaluates the ability of Vision-Language Models to avoid generating unfaithful or non-existent visual details in both closed-set discriminative QA and open-ended generative captioning. It also measures how well an auxiliary probing framework can detect these hallucinations via attention dynamics and mitigate them at inference time without retraining.

## Datasets

- **POPE** — total 9000; splits: test (9000)
- **AMBER** — total 5000; splits: test (5000)
- **M-HalDetect** — total 16000; splits: train (12800), val (3200)
- **COCO-Caption** — total 2000; splits: train (1600), val (400)

## Metrics

- `AUPRC` **(primary)** — range: [0, 1]
  - Area Under the Precision-Recall Curve computed across classification thresholds for hallucination probability scores.
- `AUROC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve computed across classification thresholds.
- `CHAIR (Ci, Cs)` — range: percent
  - Consistency (Ci) measures the fraction of hallucinated objects in the generated text relative to ground truth. Completeness (Cs) measures the fraction of ground-truth objects missing from the generated text.
- `Accuracy & F1` — range: percent
  - Standard classification metrics computed on POPE discriminative QA labels to evaluate mitigation performance.

## Input / output format

**Input**: Image paired with a text prompt (question for discriminative tasks, or instruction for generative captioning).

**Output**: Model generates a text response. VIB-Probe outputs a hallucination probability score derived from compressed attention latent states.

## Scoring recipe

```python
def compute_detection_metrics(scores, labels):
    prec, rec, _ = precision_recall_curve(labels, scores)
    auprc = auc(rec, prec)
    fpr, tpr, _ = roc_curve(labels, scores)
    auroc = auc(fpr, tpr)
    return auprc, auroc

def compute_mitigation_metrics(resp, gold):
    gen_objs = extract_entities(resp)
    gt_objs = gold.objects
    ci = len(gen_objs - gt_objs) / max(len(gen_objs), 1)
    cs = len(gt_objs - gen_objs) / max(len(gt_objs), 1)
    acc = accuracy_score(gold.labels, model.predict(image, prompt))
    f1 = f1_score(gold.labels, model.predict(image, prompt))
    return ci, cs, acc, f1
```

## Common pitfalls

- Using the full 14,216-query AMBER dataset instead of the 5,000-query subset actually used in experiments.
- Confusing detection metrics (AUPRC/AUROC) with generation quality metrics (CHAIR/F1), as they evaluate different pipeline stages.
- Calculating CHAIR metrics without normalizing by the number of generated or ground-truth objects, which fundamentally changes Ci and Cs values.
- Applying image perturbations (rotation, blur, brightness) during training rather than evaluation-only, which violates the robustness protocol.

## Evidence (verbatim from paper)

> To assess detection performance, we report AUPRC and AUROC Davis and Goadrich ([2006]). For generative evaluation, we utilized the CHAIR Rohrbach et al. ([2018]) metric, which quantifies object-level hallucinations by cross-referencing generated entities against ground-truth object lists. For POPE, we reported the Accuracy and F1 score metrics.

## Citation

```bibtex
@misc{zhang2026vibprobe,
  title={VIB-Probe: Detecting and Mitigating Hallucinations in Vision-Language Models via Variational Information Bottleneck},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2601.05547}
}
```

- arXiv: 2601.05547