Pope Eval

---
name: pope-eval
description: Evaluates object perception and hallucination in LVLMs by prompting models to identify whether specific objects are present in an image. It measures how often models correctly affirm or deny object existence without generating false positives. Use when the user wants to benchmark on POPE, or asks about evaluating this task. Reports Acc.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.01958
  bibtex_key: jing2025visualobjecthallucination
  confidence: high
---

# pope-eval

> A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models — Liqiang Jing et al. (2025) (arXiv:2505.01958, 2025)

## What this evaluates

Evaluates object perception and hallucination in LVLMs by prompting models to identify whether specific objects are present in an image. It measures how often models correctly affirm or deny object existence without generating false positives.

## Datasets

- **POPE** — total ?; splits: test (-1)

## Metrics

- `Acc` **(primary)** — range: [0, 1]
  - Accuracy: proportion of correct predictions out of total instances.
- `F1` — range: [0, 1]
  - F1: harmonic mean of precision and recall for the positive class.

## Input / output format

**Input**: Image paired with a yes/no question about object presence.

**Output**: Yes/No prediction.

## Scoring recipe

```python
def compute_metrics(preds, golds):
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    tp = sum(1 for p, g in zip(preds, golds) if p == g == 'yes')
    fp = sum(1 for p, g in zip(preds, golds) if p == 'yes' and g != 'yes')
    fn = sum(1 for p, g in zip(preds, golds) if p != 'yes' and g == 'yes')
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return acc, f1
```

## Common pitfalls

- Hallucination is measured via binary yes/no accuracy, which may not capture partial matches or nuanced object descriptions.
- The benchmark relies on pre-generated questions, so model performance can be sensitive to question phrasing rather than pure visual understanding.

## Evidence (verbatim from paper)

> Table 7: Performance of different methods on QA-FB15K.

| Method | Entity | | Relation | |
| --- | | | | |
| | Acc | F1 | Acc | F1 |
| LLaVA-7B | 78.39 | 73.14 | 56.79 | 48.79 |
...
Contrastive alignment objective is beneficial for cognition-based knowledge, as evidenced by the performance boost on QA-FB15K.

## Citation

```bibtex
@misc{jing2025visualobjecthallucination,
  title={A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models},
  author={Liqiang Jing et al. (2025)},
  year={2025},
  note={arXiv:2505.01958}
}
```

- arXiv: 2505.01958