Knowledge Based Vqa Eval

---
name: knowledge-based-vqa-eval
description: Probes a multimodal LLM's ability to answer visual questions that require external knowledge retrieval. It specifically tests the model's capacity to dynamically decide when to retrieve information and assess the relevance of retrieved documents using self-reflective tokens, without degrading performance on standard visual-only queries. Use when the user wants to benchmark on Encyclopedic-VQA, InfoSeek, or asks about evaluating this task. Reports BERT matching score (BEM), VQA accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.16863
  bibtex_key: [REDACTED]
  confidence: high
---

# knowledge-based-vqa-eval

> Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering — Cocchi et al. (2024) (arXiv:2411.16863, 2024)

## What this evaluates

Probes a multimodal LLM's ability to answer visual questions that require external knowledge retrieval. It specifically tests the model's capacity to dynamically decide when to retrieve information and assess the relevance of retrieved documents using self-reflective tokens, without degrading performance on standard visual-only queries.

## Datasets

- **Encyclopedic-VQA** — total ?; splits: train (1000000), val (13600), test (5800)
- **InfoSeek** — total ?; splits: train (934000), val (73000), test (348000)

## Metrics

- `BERT matching score (BEM)` **(primary)** — range: [0, 1]
  - Computes the cosine similarity between BERT embeddings of the predicted answer and the ground-truth answer.
- `VQA accuracy` **(primary)** — range: [0, 1]
  - Standard exact-match accuracy where the predicted answer must exactly match the ground-truth answer.
- `Relaxed accuracy` — range: [0, 1]
  - A question-type-dependent matching metric that allows for partial matches or synonyms based on the expected answer format.

## Input / output format

**Input**: Image, question, and optionally retrieved Wikipedia passages/documents.

**Output**: Textual answer string generated by the MLLM.

## Scoring recipe

```python
def evaluate(predictions, golds, dataset):
    if dataset == 'Encyclopedic-VQA':
        return bert_matching_score(predictions, golds)
    elif dataset == 'InfoSeek':
        return vqa_accuracy(predictions, golds)
    # Relaxed accuracy is applied per question type as specified by the dataset protocol
    return relaxed_accuracy(predictions, golds, question_types)
```

## Common pitfalls

- The paper reports contradictory dataset sizes for Encyclopedic-VQA (states 221k pairs but lists 1M training items).
- Evaluation splits differ between datasets: Encyclopedic-VQA uses the test set, while InfoSeek uses the validation set.
- Relaxed accuracy for InfoSeek is question-type dependent, requiring careful handling of evaluation scripts.

## Evidence (verbatim from paper)

> Generated answers for Encyclopedic-VQA are evaluated according to the BERT matching score (BEM) between predicted and ground-truth answers. Instead, when evaluating answers for image-question pairs from InfoSeek, we use VQA accuracy and relaxed accuracy depending on the question type.

## Citation

```bibtex
@misc{cocchi2024reflectiva,
  title={Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering},
  author={Cocchi et al. (2024)},
  year={2024},
  note={arXiv:2411.16863}
}
```

- arXiv: 2411.16863