Hpope Eval

---
name: hpope-eval
description: Evaluates hallucinations in large vision-language models by probing their ability to correctly identify object existence and ground fine-grained attributes (color, material, shape) to specific objects in an image. Use when the user wants to benchmark on H-POPE, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.04077
  bibtex_key: pham2024hpope
  confidence: medium
---

# hpope-eval

> H-POPE: Hierarchical Polling-based Probing Evaluation of Hallucinations in Large Vision-Language Models — Pham et al. (2024) (arXiv:2411.04077, 2024)

## What this evaluates

Evaluates hallucinations in large vision-language models by probing their ability to correctly identify object existence and ground fine-grained attributes (color, material, shape) to specific objects in an image.

## Datasets

- **H-POPE** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Standard binary classification accuracy computed over model responses to yes/no questions about object existence and attributes.

## Input / output format

**Input**: An input image paired with a binary question (e.g., 'Does the [object] have [attribute]?') constructed from ground-truth annotations and negative sampling strategies.

**Output**: Binary response (Yes/No) from the vision-language model.

## Scoring recipe

```python
predictions = [model_answer == 'Yes' for model_answer in model_outputs]
gold = [1 if object/attribute_exists else 0 for _ in questions]
accuracy = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
```

## Common pitfalls

- Models often hallucinate attributes that are present in the image but belong to different objects (image-based adversarial sampling).
- Negative sampling strategies (random, popular, frequency-based, image-based) significantly impact difficulty and should be reported separately.
- Attribute evaluation requires mutual exclusion; models may confuse overlapping or non-mutually exclusive attributes if not restricted to color, material, and shape.

## Evidence (verbatim from paper)

> Our H-POPE benchmark formulates binary questions about object existence and object attributes. Negative attributes are sampled from the list of attributes across the dataset Ω, so we need to ensure mutual exclusion, i.e., if an object has a list A of attributes, it should not have any attributes from Ω∖A.

## Citation

```bibtex
@misc{pham2024hpope,
  title={H-POPE: Hierarchical Polling-based Probing Evaluation of Hallucinations in Large Vision-Language Models},
  author={Pham et al. (2024)},
  year={2024},
  note={arXiv:2411.04077}
}
```

- arXiv: 2411.04077