Sentinel Hallucination Eval

---
name: sentinel-hallucination-eval
description: Evaluates a sentence-level early intervention framework for reducing object hallucinations in multimodal large language models (MLLMs) while preserving or enhancing general vision-language capabilities across multiple standard benchmarks. Use when the user wants to benchmark on Object HalBench, AMBER, HallusionBench, VQAv2, TextVQA, ScienceQA, MM-Vet, or asks about evaluating this task. Reports response-level hallucination rate (Resp.).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.12455
  bibtex_key: peng2025sentinel
  confidence: high
---

# sentinel-hallucination-eval

> Mitigating Object Hallucinations via Sentence-Level Early Intervention — Peng et al. (2025) (arXiv:2507.12455, 2025)

## What this evaluates

Evaluates a sentence-level early intervention framework for reducing object hallucinations in multimodal large language models (MLLMs) while preserving or enhancing general vision-language capabilities across multiple standard benchmarks.

## Datasets

- **Object HalBench** — total ?; splits: (unstated)
- **AMBER** — total ?; splits: (unstated)
- **HallusionBench** — total ?; splits: (unstated)
- **VQAv2** — total ?; splits: (unstated)
- **TextVQA** — total ?; splits: (unstated)
- **ScienceQA** — total ?; splits: (unstated)
- **MM-Vet** — total ?; splits: (unstated)

## Metrics

- `response-level hallucination rate (Resp.)` **(primary)** — range: percent
  - Percentage of generated responses that contain at least one hallucinated object. Calculated as (number of responses with hallucinations / total responses) * 100.
- `mention-level hallucination rate (Ment.)` — range: percent
  - Percentage of hallucinated object mentions relative to the total number of object mentions in the responses.
- `AMBER F1` — range: other
  - F1 score computed on the discriminative part of the AMBER benchmark across six hallucination types.
- `CHAIR` — range: percent
  - Sentence-level hallucination rate measuring the proportion of sentences containing hallucinated objects.
- `MM-Vet Overall` — range: other
  - Composite score evaluating general vision-language capabilities, including reasoning, translation, OCR, and more.

## Input / output format

**Input**: Image paired with a text prompt or question.

**Output**: Natural language response describing the image or answering the question.

## Scoring recipe

```python
def compute_metrics(predictions, golds, benchmark):
    if benchmark == 'Object_HalBench':
        resp_hall = sum(1 for p in predictions if contains_hallucination(p)) / len(predictions)
        ment_hall = count_hallucinated_mentions(predictions) / count_total_mentions(predictions)
        return resp_hall * 100, ment_hall * 100
    elif benchmark == 'AMBER':
        return f1_score(predictions, golds), accuracy(predictions, golds)
    elif benchmark == 'MM-Vet':
        return mmvet_scoring(predictions, golds)
    return standard_vqa_accuracy(predictions, golds)
```

## Common pitfalls

- Confusing response-level vs. mention-level hallucination rates, which measure different granularities of error (per-response vs. per-mention).
- Assuming hallucination mitigation techniques inherently degrade general VQA performance; this method demonstrates concurrent improvement on benchmarks like ScienceQA and MM-Vet.
- Using out-of-domain or rewritten preference data instead of in-domain context-coherent samples, which significantly reduces effectiveness and increases hallucination rates.

## Evidence (verbatim from paper)

> Specifically, for the 7B model, our method achieves a 4.3 response-level (Resp.) and a 2.6 mention-level (Ment.) hallucination rate. Compared to the previous SOTA method, TPO, which achieves a 5.6 response-level and 3.2 mention-level hallucination rate, our proposed SENTINEL surpasses it by further reducing hallucinations by a total of 24% on Object Halbench.

## Citation

```bibtex
@misc{peng2025sentinel,
  title={Mitigating Object Hallucinations via Sentence-Level Early Intervention},
  author={Peng et al. (2025)},
  year={2025},
  note={arXiv:2507.12455}
}
```

- arXiv: 2507.12455