Mpd Hallucination Eval

---
name: mpd-hallucination-eval
description: Evaluates the ability of Large Vision-Language Models (LVLMs) to suppress object and semantic hallucinations while maintaining general perception, reasoning, and generative capabilities. It probes grounding fidelity across structured yes/no queries, open-ended captioning, and fine-grained visual diagnostics. Use when the user wants to benchmark on MSCOCO, MME, LLaVA-Bench, HallusionBench, or asks about evaluating this task. Reports CHAIR_S, CHAIR_I, POPE F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.20366
  bibtex_key: zhu2026mpd
  confidence: high
---

# mpd-hallucination-eval

> Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation — Zhu et al. (2026) (arXiv:2604.20366, 2026)

## What this evaluates

Evaluates the ability of Large Vision-Language Models (LVLMs) to suppress object and semantic hallucinations while maintaining general perception, reasoning, and generative capabilities. It probes grounding fidelity across structured yes/no queries, open-ended captioning, and fine-grained visual diagnostics.

## Datasets

- **MSCOCO** — total ?; splits: test (-1)
- **MME** — total ?; splits: test (-1)
- **LLaVA-Bench** — total ?; splits: test (-1)
- **HallusionBench** — total ?; splits: test (-1)

## Metrics

- `CHAIR_S` **(primary)** — range: percent
  - Sentence-level hallucination rate: proportion of generated sentences containing at least one hallucinated object. Lower is better.
- `CHAIR_I` **(primary)** — range: percent
  - Instance-level hallucination rate: proportion of generated captions containing at least one hallucinated object. Lower is better.
- `POPE F1` **(primary)** — range: [0, 1]
  - F1 score for yes/no object presence queries under random, popular, and adversarial negative sampling strategies.
- `BLEU` — range: [0, 1]
  - Standard n-gram overlap metric used to measure caption quality and fluency.
- `MME (Existence/Count/Position/Color)` — range: other
  - Scores on four hallucination-relevant subsets of the MME benchmark assessing perception and reasoning.
- `LLaVA-Bench Accuracy/Detailedness` — range: [1, 10]
  - GPT-4V evaluated scores on a 1-10 scale measuring factual alignment and response richness.
- `HallusionBench (fACC/qACC/easyA/hardA/aACC)` — range: percent
  - Fine-grained accuracy metrics for visual illusion vs language hallucination disentanglement, including easy/hard subsets and overall alignment.

## Input / output format

**Input**: Paired image and text prompt (e.g., open-ended description request or specific yes/no/factual questions).

**Output**: Free-form natural language response (captions, answers, or descriptions).

## Scoring recipe

```python
def score(predictions, gold):
    scores = {}
    scores['CHAIR_S'] = count_hallucinated_sentences(predictions) / len(predictions)
    scores['CHAIR_I'] = count_hallucinated_objects(predictions) / len(predictions)
    scores['POPE_F1'] = f1_score(gold, [pred == 'yes' for pred in predictions])
    scores['BLEU'] = compute_bleu(predictions, gold)
    scores['LLaVA_Acc'] = gpt4v_evaluate(predictions, gold, 'accuracy')
    scores['Hallusion_aACC'] = gpt4v_evaluate(predictions, gold, 'alignment')
    return scores
```

## Common pitfalls

- CHAIR scores represent hallucination rates, so lower values indicate better performance, contrary to most accuracy metrics.
- LLaVA-Bench and HallusionBench rely on GPT-4V for scoring, which can introduce evaluator bias or inconsistency compared to rule-based metrics like POPE.
- MME results only report four hallucination-relevant subsets (Existence, Count, Position, Color), not the full benchmark score.

## Evidence (verbatim from paper)

> We evaluate OH mitigation performance using the following three categories of benchmarks. MSCOCO-based metrics. We adopt CHAIR and POPE on the MSCOCO dataset to evaluate hallucination in image descriptions. CHAIR includes sentence-level (CHAIR_S) and instance-level (CHAIR_I) hallucination metrics, with lower scores indicating better grounding. POPE measures LVLMs’ ability to correctly answer yes/no object presence queries under different negative sampling strategies.

## Citation

```bibtex
@misc{zhu2026mpd,
  title={Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation},
  author={Zhu et al. (2026)},
  year={2026},
  note={arXiv:2604.20366}
}
```

- arXiv: 2604.20366