Qa Fb15k Eval

---
name: qa-fb15k-eval
description: Evaluates cognition-based hallucination by testing whether LVLMs can leverage world knowledge stored in the LLM to answer entity and relation questions grounded in images. Use when the user wants to benchmark on QA-FB15K, or asks about evaluating this task. Reports Acc.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.01958
  bibtex_key: jing2025visualobjecthallucination
  confidence: high
---

# qa-fb15k-eval

> A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models — Liqiang Jing et al. (2025) (arXiv:2505.01958, 2025)

## What this evaluates

Evaluates cognition-based hallucination by testing whether LVLMs can leverage world knowledge stored in the LLM to answer entity and relation questions grounded in images.

## Datasets

- **QA-FB15K** — total ?; splits: test (-1)

## Metrics

- `Acc` **(primary)** — range: [0, 1]
  - Accuracy: proportion of correct predictions out of total instances.
- `F1` — range: [0, 1]
  - F1: harmonic mean of precision and recall for the positive class.

## Input / output format

**Input**: Image paired with a question requiring entity or relation identification based on world knowledge.

**Output**: Textual answer or predicted label (entity/relation name).

## Scoring recipe

```python
def compute_metrics(preds, golds):
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    tp = sum(1 for p, g in zip(preds, golds) if p == g)
    fp = sum(1 for p, g in zip(preds, golds) if p != g)
    fn = sum(1 for p, g in zip(preds, golds) if p != g)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return acc, f1
```

## Common pitfalls

- Cognition-based benchmarks require both visual grounding and external knowledge; improvements may stem from LLM pretraining rather than vision alignment.
- Entity and Relation splits are evaluated separately, and performance on one does not guarantee generalization to the other due to different knowledge retrieval paths.

## Evidence (verbatim from paper)

> Table 7: Performance of different methods on QA-FB15K.

| Method | Entity | | Relation | |
| --- | | | | |
| | Acc | F1 | Acc | F1 |
| LLaVA-7B | 78.39 | 73.14 | 56.79 | 48.79 |
| Int. Align. 🚀 | 84.28 | 83.03 | 59.16 | 58.07 |
| Int. Align. ❄️ | 84.05 | 81.76 | 59.16 | 56.97 |
| Sep. Ctrs. Align. | 83.94 | 81.65 | 59.39 | 57.41 |
| LLaVA-7B | 78.39 | 73.14 | 56.79 | 48.70 |
| w-ECLIP | 77.60 | 71.47 | 56.79 | 45.58 |
| w-FineIns | 76.47 | 69.86 | 55.45 | 49.10 |

To further investigate the influence of our method on other kinds of hallucination, we introduced the Cognition-based benchmark: necessitating world knowledge in LVLMs for problem solving. We construct a cognition-based benchmark QA-FB15k based on the knowledge graph FB15K Bordes et al. (2013). We show the results in Table [7].

Contrastive alignment objective is beneficial for cognition-based knowledge, as evidenced by the performance boost on QA-FB15K.

## Citation

```bibtex
@misc{jing2025visualobjecthallucination,
  title={A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models},
  author={Liqiang Jing et al. (2025)},
  year={2025},
  note={arXiv:2505.01958}
}
```

- arXiv: 2505.01958