Qa Visualgenome Eval

---
name: qa-visualgenome-eval
description: Evaluates attribute and relation hallucination by asking LVLMs to identify object properties and inter-object relationships in images, probing fine-grained visual understanding beyond basic object detection. Use when the user wants to benchmark on QA-VisualGenome, or asks about evaluating this task. Reports Acc.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.01958
  bibtex_key: jing2025visualobjecthallucination
  confidence: high
---

# qa-visualgenome-eval

> A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models — Liqiang Jing et al. (2025) (arXiv:2505.01958, 2025)

## What this evaluates

Evaluates attribute and relation hallucination by asking LVLMs to identify object properties and inter-object relationships in images, probing fine-grained visual understanding beyond basic object detection.

## Datasets

- **QA-VisualGenome** — total ?; splits: test (-1)

## Metrics

- `Acc` **(primary)** — range: [0, 1]
  - Accuracy: proportion of correct predictions out of total instances.
- `F1` — range: [0, 1]
  - F1: harmonic mean of precision and recall for the positive class.

## Input / output format

**Input**: Image paired with a question about object attributes or relations.

**Output**: Textual answer or predicted label (attribute/relation name).

## Scoring recipe

```python
def compute_metrics(preds, golds):
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    tp = sum(1 for p, g in zip(preds, golds) if p == g)
    fp = sum(1 for p, g in zip(preds, golds) if p != g)
    fn = sum(1 for p, g in zip(preds, golds) if p != g)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return acc, f1
```

## Common pitfalls

- Attribute and relation hallucinations are conflated with object hallucinations; the benchmark requires strict separation of perception vs. cognition errors.
- Metrics are reported per sub-split (Attribute vs. Relation), so aggregating them without weighting can mask performance drops on specific relation types.

## Evidence (verbatim from paper)

> Table 7: Performance of different methods on QA-FB15K.

| Method | Entity | | Relation | |
| --- | | | | |
| | Acc | F1 | Acc | F1 |
| LLaVA-7B | 78.39 | 73.14 | 56.79 | 48.79 |
...
Contrastive alignment objective is beneficial for cognition-based knowledge, as evidenced by the performance boost on QA-FB15K.

## Citation

```bibtex
@misc{jing2025visualobjecthallucination,
  title={A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models},
  author={Liqiang Jing et al. (2025)},
  year={2025},
  note={arXiv:2505.01958}
}
```

- arXiv: 2505.01958