Kg Vip Eval

---
name: kg-vip-eval
description: Evaluates multi-modal LLMs' ability to perform visual question answering by grounding visual inputs with external knowledge graphs. It probes the model's capacity for multi-hop reasoning, visual perception, and knowledge retrieval-augmented generation. Use when the user wants to benchmark on FVQA 2.0+, MVQA, or asks about evaluating this task. Reports LLM-J.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.11632
  bibtex_key: li2026kgvip
  confidence: high
---

# kg-vip-eval

> KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering — Li et al. (2026) (arXiv:2601.11632, 2026)

## What this evaluates

Evaluates multi-modal LLMs' ability to perform visual question answering by grounding visual inputs with external knowledge graphs. It probes the model's capacity for multi-hop reasoning, visual perception, and knowledge retrieval-augmented generation.

## Datasets

- **FVQA 2.0+** — total 2820; splits: test (2820)
- **MVQA** — total 1433; splits: test (1433)

## Metrics

- `LLM-J` **(primary)** — range: percent
  - LLM-as-a-Judge score where an LLM assigns a scalar score evaluating the relevance of the generated response with the ground-truth answer. Default judge is DeepSeek-V3.2.
- `METEOR` — range: percent
  - Harmonic mean of unigram precision and recall, capturing semantic variations while applying a fragmentation penalty for word order.
- `SAS` — range: percent
  - Semantic Answer Similarity score that utilizes a cross-encoder to assess the semantic equivalence between predicted and reference answers.

## Input / output format

**Input**: A query image and a natural language question requiring external knowledge.

**Output**: A natural language answer string.

## Scoring recipe

```python
def compute_metrics(predictions, gold_answers):
    meteor_scores = [meteor_score(pred, gold) for pred, gold in zip(predictions, gold_answers)]
    sas_scores = [cross_encoder_similarity(pred, gold) for pred, gold in zip(predictions, gold_answers)]
    llm_j_scores = [llm_judge_score(pred, gold) for pred, gold in zip(predictions, gold_answers)]
    return {
        'METEOR': mean(meteor_scores) * 100,
        'SAS': mean(sas_scores) * 100,
        'LLM-J': mean(llm_j_scores) * 100
    }
```

## Common pitfalls

- FVQA 2.0+ is a custom multimodal upgrade of the original FVQA dataset with a visual commonsense graph, not the standard textual FVQA benchmark.
- LLM-J scores are highly sensitive to the specific judge model (DeepSeek-V3.2 vs Qwen2.5-7B) and prompting strategy, which are not fully detailed in the main text.
- MVQA is a newly constructed, non-public dataset based on MovieBench, so external reproducibility requires following the Appendix A.2 pipeline.

## Evidence (verbatim from paper)

> Following prior work, we adopt three widely used metrics for evaluation. METEOR calculates the harmonic mean of unigram precision and recall, capturing semantic variations while applying a fragmentation penalty for word order. Semantic Answer Similarity (SAS) utilizes a cross-encoder to assess the semantic equivalence between predicted and reference answers. LLM-as-a-Judge leverages an LLM to assign a scalar score that evaluates the relevance of the generated response with the ground-truth answer. We denote this metric as LLM-J for short.

## Citation

```bibtex
@misc{li2026kgvip,
  title={KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2601.11632}
}
```

- arXiv: 2601.11632