Kvasir Vqa X1 Eval

---
name: kvasir-vqa-x1-eval
description: Evaluates multimodal vision-language models on gastrointestinal endoscopy image understanding and clinical question answering. It probes factual recall, multi-step clinical reasoning across varying complexity levels, and robustness to realistic visual perturbations like motion blur and color shifts. Use when the user wants to benchmark on Kvasir-VQA-x1, or asks about evaluating this task. Reports BERT-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.09958
  bibtex_key: [REDACTED]
  confidence: high
---

# kvasir-vqa-x1-eval

> Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopy — Gautam et al. (2025) (arXiv:2506.09958, 2025)

## What this evaluates

Evaluates multimodal vision-language models on gastrointestinal endoscopy image understanding and clinical question answering. It probes factual recall, multi-step clinical reasoning across varying complexity levels, and robustness to realistic visual perturbations like motion blur and color shifts.

## Datasets

- **Kvasir-VQA-x1** — total 159549; splits: train (-1), test (-1); repo https://github.com/Simula/Kvasir-VQA-x1

## Metrics

- `BERT-F1` **(primary)** — range: [0, 1]
  - Embedding-based F1 score computed by comparing contextual BERT representations of the predicted answer and ground truth answer to assess semantic similarity.
- `ROUGE-1` — range: [0, 1]
  - Unigram overlap F-score between predicted and reference text, measuring surface-level content overlap.
- `LLM-Adjudicator Accuracy` — range: [0, 1]
  - Binary per-category scoring (1/0) by a Qwen3-30B-A3B adjudicator comparing model output to ground truth, aggregated as the fraction of correctly addressed clinical aspects per question category.

## Input / output format

**Input**: A single gastrointestinal endoscopy image (original or weakly augmented) paired with a natural language question string.

**Output**: A natural language answer string generated by the model.

## Scoring recipe

```python
def score(predictions, golds, questions, categories):
    # Standard n-gram & embedding metrics
    rouge1 = compute_rouge(predictions, golds, 'rouge1')
    bert_f1 = compute_bert_f1(predictions, golds)
    
    # LLM Adjudicator per-category accuracy
    correct = {cat: 0 for cat in categories}
    total = {cat: 0 for cat in categories}
    for pred, gold, q, cats in zip(predictions, golds, questions, categories):
        prompt = build_adjudicator_prompt(q, pred, gold, cats)
        resp = call_llm(prompt)  # Qwen3-30B-A3B
        scores = json.loads(resp)['eval_json']
        for cat in cats:
            total[cat] += 1
            if scores[cat]['score'] == 1:
                correct[cat] += 1
    cat_acc = {cat: correct[cat]/total[cat] for cat in categories}
    return {'ROUGE-1': rouge1, 'BERT-F1': bert_f1, 'Categorical_Accuracy': cat_acc}
```

## Common pitfalls

- Relying solely on n-gram metrics (BLEU/ROUGE) which fail to capture clinical semantic correctness and multi-step reasoning depth.
- Evaluating only on the original image split, thereby missing the dataset's core robustness benchmark against visual perturbations.
- Ignoring the complexity stratification (Levels 1–3), which masks model degradation on higher-order clinical inference tasks.

## Evidence (verbatim from paper)

> Models are assessed using a comprehensive suite of standard VQA and natural language processing (NLP) metrics, chosen to capture various facets of response quality: ROUGE-1, ROUGE-2, ROUGE-L: These metrics measure n-gram overlap and sequence similarity, providing insights into the content overlap between the model’s answer and the ground truth. BERT-F1: An embedding-based similarity metric with F1 aggregation using BERT. This metric assesses the semantic similarity between the generated answer and the ground truth by leveraging contextual embeddings from BERT.

## Citation

```bibtex
@misc{gautam2025kvasirvqax1,
  title={Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopy},
  author={Gautam et al. (2025)},
  year={2025},
  note={arXiv:2506.09958}
}
```

- arXiv: 2506.09958