Pope Nocaps Eval

---
name: pope-nocaps-eval
description: Tests object perception and hallucination on images without captions, evaluating whether LVLMs can ground object detection purely from visual input without textual priors. Use when the user wants to benchmark on POPE-NoCaps, or asks about evaluating this task. Reports Acc.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.01958
  bibtex_key: jing2025visualobjecthallucination
  confidence: high
---

# pope-nocaps-eval

> A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models — Liqiang Jing et al. (2025) (arXiv:2505.01958, 2025)

## What this evaluates

Tests object perception and hallucination on images without captions, evaluating whether LVLMs can ground object detection purely from visual input without textual priors.

## Datasets

- **POPE-NoCaps** — total ?; splits: test (-1)

## Metrics

- `Acc` **(primary)** — range: [0, 1]
  - Accuracy: proportion of correct predictions out of total instances.
- `F1` — range: [0, 1]
  - F1: harmonic mean of precision and recall for the positive class.

## Input / output format

**Input**: Image paired with a yes/no question about object presence.

**Output**: Yes/No prediction.

## Scoring recipe

```python
def compute_metrics(preds, golds):
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    tp = sum(1 for p, g in zip(preds, golds) if p == g == 'yes')
    fp = sum(1 for p, g in zip(preds, golds) if p == 'yes' and g != 'yes')
    fn = sum(1 for p, g in zip(preds, golds) if p != 'yes' and g == 'yes')
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return acc, f1
```

## Common pitfalls

- Absence of captions removes textual grounding, making models more prone to language priors; evaluation must control for caption leakage.
- Binary yes/no scoring ignores confidence calibration, which is critical for hallucination detection.

## Evidence (verbatim from paper)

> Table 7: Performance of different methods on QA-FB15K.

| Method | Entity | | Relation | |
| --- | | | | |
| | Acc | F1 | Acc | F1 |
| LLaVA-7B | 78.39 | 73.14 | 56.79 | 48.79 |
...
Contrastive alignment objective is beneficial for cognition-based knowledge, as evidenced by the performance boost on QA-FB15K.

## Citation

```bibtex
@misc{jing2025visualobjecthallucination,
  title={A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models},
  author={Liqiang Jing et al. (2025)},
  year={2025},
  note={arXiv:2505.01958}
}
```

- arXiv: 2505.01958