Halluscope Eval

---
name: halluscope-eval
description: Evaluates LVLMs' ability to resist prompt-induced hallucinations by disentangling perception failures from instruction-induced presuppositions. It probes whether models rely on visual evidence or textual priors when answering questions that imply the presence of non-existent objects. Use when the user wants to benchmark on HalluScope, or asks about evaluating this task. Reports AdP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.21911
  bibtex_key: khayatan2026halluscope
  confidence: high
---

# halluscope-eval

> When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMs — Khayatan et al. (2026) (arXiv:2604.21911, 2026)

## What this evaluates

Evaluates LVLMs' ability to resist prompt-induced hallucinations by disentangling perception failures from instruction-induced presuppositions. It probes whether models rely on visual evidence or textual priors when answering questions that imply the presence of non-existent objects.

## Datasets

- **HalluScope** — total ?; splits: test (-1)

## Metrics

- `AdP` **(primary)** — range: [0, 100]
  - Accuracy on adversarial presupposition tasks, measuring the model's ability to correctly reject false object presuppositions implied in the query. Reported as a percentage.
- `Rec_adv` — range: [0, 100]
  - Accuracy on adversarial object recognition tasks.
- `Rec_pos` — range: [0, 100]
  - Accuracy on positive recognition tasks.
- `Rec_rand` — range: [0, 100]
  - Accuracy on random recognition tasks.

## Input / output format

**Input**: An image paired with a text prompt containing either a direct question, an adversarial presupposition, or a description request.

**Output**: A natural language response, typically a yes/no answer or a descriptive caption.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for pred, g in zip(predictions, gold) if pred.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Metrics are reported as percentages (0-100), not probabilities (0-1).
- AdP specifically measures presupposition rejection, not general hallucination or perception accuracy.
- Baseline comparisons must use greedy decoding for fair comparison, except VCD which uses direct sampling.

## Evidence (verbatim from paper)

> They demonstrate substantial improvements for HalluVL-DPO in mitigating hallucinations (adversarial recognition and presupposition). For LLaVA, the model becomes remarkably more reliable at identifying objects that are not present when their presence is implied in the query: HalluScope AdP rises from 5.85 to over 80, and CP-Bench CPQ accuracy increases from 0.54 to above 70.

## Citation

```bibtex
@misc{khayatan2026halluscope,
  title={When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMs},
  author={Khayatan et al. (2026)},
  year={2026},
  note={arXiv:2604.21911}
}
```

- arXiv: 2604.21911