Vqa Generalization Eval

---
name: vqa-generalization-eval
description: Evaluates a model's ability to answer visual questions by generalizing from synthetic template-based training data to complex, human-written questions. It probes both closed-form accuracy and open-form reasoning capabilities across 3D-rendered and medical imaging domains. Use when the user wants to benchmark on CLEVR-Human, VQA-RAD, SLAKE, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.06400
  bibtex_key: kim2024coqah
  confidence: high
---

# vqa-generalization-eval

> Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a Large Language Model — Taehee Kim et al. (2024) (arXiv:2401.06400, 2024)

## What this evaluates

Evaluates a model's ability to answer visual questions by generalizing from synthetic template-based training data to complex, human-written questions. It probes both closed-form accuracy and open-form reasoning capabilities across 3D-rendered and medical imaging domains.

## Datasets

- **CLEVR-Human** — total ?; splits: val (-1)
- **VQA-RAD** — total ?; splits: test (-1)
- **SLAKE** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered closed-form questions. Computed as the number of exact matches between predicted and ground-truth answers divided by the total number of questions.
- `LAVEGPT-4` — range: [0, 1]
  - A GPT-4-based evaluation metric used to score the quality and correctness of open-form answers. Scores are averaged across all open-form instances.

## Input / output format

**Input**: Image paired with a natural language question (closed-form or open-form).

**Output**: A final answer string (e.g., 'yes', 'no', or a descriptive text for open-form questions).

## Scoring recipe

```python
def compute_metrics(predictions, golds, is_open_form=False):
    if not is_open_form:
        correct = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower())
        return correct / len(golds)
    else:
        scores = [llm_evaluate(p, g) for p, g in zip(predictions, golds)]
        return sum(scores) / len(scores)
```

## Common pitfalls

- The model is trained exclusively on synthetic/template-based QA pairs, so evaluation tests zero-shot generalization to human-written questions.
- Open-form answers are evaluated using LAVEGPT-4 rather than exact match, requiring semantic similarity assessment.
- The LLM's intermediate questions must strictly follow template formats to be understood by the underlying VQA model.

## Evidence (verbatim from paper)

> Table 2 summarizes the performance of the medical foundation and template-based VQA models for VQA-RAD and SLAKE. Among all the models, CoQAH reported the highest accuracy in the closed-form questions (e.g., VQA-RAD: 67.5% for CoQAH vs. 59.5% for OFA-MIMIC, SLAKE: 73.9% for CoQAH vs. 69.4% for OFA-MIMIC), and also the highest LAVEGPT-4 in the open-form questions (e.g., VQA-RAD: 0.302 for CoQAH vs. 0.274 for MedVInT-TD, SLAKE: 0.425 for CoQAH vs. 0.396 for MedVInT-TD).

## Citation

```bibtex
@misc{kim2024coqah,
  title={Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a Large Language Model},
  author={Taehee Kim et al. (2024)},
  year={2024},
  note={arXiv:2401.06400}
}
```

- arXiv: 2401.06400