Hallucination Eval

---
name: hallucination-eval
description: Probes a model's ability to avoid generating factually incorrect statements about visual content. It measures alignment between model outputs and ground-truth visual facts using binary detection and scoring metrics. Use when the user wants to benchmark on POPE, AMBER-d, HallusionBench, or asks about evaluating this task. Reports Accuracy (Acc).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.17115
  bibtex_key: [REDACTED]
  confidence: medium
---

# hallucination-eval

> Evaluating the Quality of Hallucination Benchmarks for Large Vision-Language Models — Yan et al. (2024) (arXiv:2406.17115, 2024)

## What this evaluates

Probes a model's ability to avoid generating factually incorrect statements about visual content. It measures alignment between model outputs and ground-truth visual facts using binary detection and scoring metrics.

## Datasets

- **POPE** — total ?; splits: test (-1)
- **AMBER-d** — total ?; splits: test (-1)
- **HallusionBench** — total ?; splits: test (-1)

## Metrics

- `Accuracy (Acc)` **(primary)** — range: [0, 1]
  - Proportion of correct binary predictions matching the ground truth label. Calculated as correct predictions divided by total instances.
- `CHAIR` — range: [0, 1]
  - Consistency-aware Hallucination Assessment Index; lower is better. Measures instance-level and sentence-level hallucination rates.
- `OCH` — range: [0, 1]
  - Open-Ended Consistency Hallucination metric; lower is better. Evaluates consistency in open-ended responses.
- `MMHal Score` — range: [0, 1]
  - Numerical score indicating hallucination quality; higher is better.
- `GAVIE A-Score / R-Score` — range: [0, 1]
  - Accuracy and Relevancy hallucination scores from GAVIE; higher is better.

## Input / output format

**Input**: An image paired with a question or statement requiring a visual fact check.

**Output**: Binary response ('yes'/'no') or a numerical score, depending on the specific benchmark's protocol.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

def compute_yes_rate(predictions):
    return sum(1 for p in predictions if p.lower() == 'yes') / len(predictions)
```

## Common pitfalls

- Response bias: models may over-respond 'yes', inflating accuracy on imbalanced datasets.
- Parallel test instability: scores can vary significantly when the same model is evaluated on parallel test sets.
- Metric misalignment: automated scores (e.g., CHAIR, OCH) may not correlate well with human judgment of hallucination severity.

## Evidence (verbatim from paper)

> Acc denotes the accuracy, A-Score and R-Score denotes the accuracy and relevancy hallucination score in GAVIE. The top-2 results are bolded and underlined, respectively.

## Citation

```bibtex
@misc{yan2024evaluating,
  title={Evaluating the Quality of Hallucination Benchmarks for Large Vision-Language Models},
  author={Yan et al. (2024)},
  year={2024},
  note={arXiv:2406.17115}
}
```

- arXiv: 2406.17115