Hallucination Mitigation Eval

---
name: hallucination-mitigation-eval
description: Evaluates the ability of Large Vision-Language Models to generate factually aligned outputs by measuring object hallucination rates in captions and yes/no answers, as well as logical reasoning and attribute consistency across diverse visual prompts. Use when the user wants to benchmark on POPE, CHAIR, MMHal-Bench, or asks about evaluating this task. Reports POPE Average Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.13707
  bibtex_key: jo2026attention
  confidence: high
---

# hallucination-mitigation-eval

> Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMs — Jo et al. (2026) (arXiv:2601.13707, 2026)

## What this evaluates

Evaluates the ability of Large Vision-Language Models to generate factually aligned outputs by measuring object hallucination rates in captions and yes/no answers, as well as logical reasoning and attribute consistency across diverse visual prompts.

## Datasets

- **POPE** — total ?; splits: random (-1), popular (-1), adversarial (-1), total (-1)
- **CHAIR** — total ?; splits: test (-1)
- **MMHal-Bench** — total 96; splits: test (96)

## Metrics

- `POPE Average Accuracy` **(primary)** — range: percent
  - Accuracy of binary yes/no predictions averaged across the random, popular, and adversarial splits.
- `CHAIRs` — range: percent
  - Percentage of generated sentences containing at least one hallucinated object.
- `CHAIRi` — range: percent
  - Percentage of images where the generated caption contains at least one hallucinated object.
- `CHAIR F1` — range: percent
  - Harmonic mean of recall and precision for detected objects in captions compared to ground truth.
- `MMHal-Bench Average Score` — range: other
  - GPT-4 judged alignment score averaged across eight categories (ATTR, ADV, COMP, COUNT, SPAT, ENV, HOL, OTHER).

## Input / output format

**Input**: Image and text prompt (question or instruction) for generation; for POPE, image and yes/no question.

**Output**: Text response: binary 'yes'/'no' for POPE; free-form caption or answer for CHAIR and MMHal-Bench.

## Scoring recipe

```python
def score_poPE(preds, golds):
    accs = [1.0 if p.strip().lower() == g.strip().lower() else 0.0 for p, g in zip(preds, golds)]
    return sum(accs) / len(accs)

def score_chair(preds, gold_objs):
    sents_halluc = 0; imgs_halluc = 0; tp=fp=fn=0
    for cap, gt in zip(preds, gold_objs):
        detected = parse_objects(cap)
        tp += len(detected & gt)
        fp += len(detected - gt)
        fn += len(gt - detected)
        if detected - gt:
            imgs_halluc += 1
            sents_halluc += count_sentences_with_halluc(cap, detected - gt)
    return (sents_halluc/len(preds), imgs_halluc/len(preds), 2*tp/(2*tp+fp+fn))
```

## Common pitfalls

- CHAIR metrics depend heavily on the object detection parser used to extract entities from generated text; inconsistent NLP pipelines yield different CHAIRs/CHAIRi values.
- POPE adversarial split contains semantically or statistically related negative samples, making it significantly harder than random/popular splits and requiring careful prompt formatting.
- MMHal-Bench relies on GPT-4 for scoring, which can introduce judge bias or variability; the paper notes that weighting parameter gamma must be tuned per model for stable behavior.

## Evidence (verbatim from paper)

> POPE measures binary yes/no object existence, while CHAIR evaluates object hallucinations in free-form captions. MMHal-Bench consists of 96 image–question pairs that probe object and attribute-level inconsistencies. Model responses’ alignment with ground-truth answers is evaluated by GPT-4. We report sentence-level hallucination rate (CHAIRs), instance-level hallucination rate (CHAIRi), and F1 score.

## Citation

```bibtex
@misc{jo2026attention,
  title={Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMs},
  author={Jo et al. (2026)},
  year={2026},
  note={arXiv:2601.13707}
}
```

- arXiv: 2601.13707