Vindrcxrvqa Eval

---
name: vindrcxrvqa-eval
description: Evaluates a model's ability to answer clinical questions about chest X-rays and localize lesions via bounding boxes. It probes visual question answering, spatial grounding, and multi-task learning in a medical imaging context. Use when the user wants to benchmark on VinDr-CXR-VQA, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.00504
  bibtex_key: nguyen2025vindrcxrvqa
  confidence: high
---

# vindrcxrvqa-eval

> VinDr-CXR-VQA: A Visual Question Answering Dataset for Explainable Chest X-Ray Analysis with Multi-Task Learning — Nguyen et al. (2025) (arXiv:2511.00504, 2025)

## What this evaluates

Evaluates a model's ability to answer clinical questions about chest X-rays and localize lesions via bounding boxes. It probes visual question answering, spatial grounding, and multi-task learning in a medical imaging context.

## Datasets

- **VinDr-CXR-VQA** — total ?; splits: train (20880), val (659)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of Precision and Recall: 2 * (Prec * Rec) / (Prec + Rec). Computed on text-based lesion classification answers.
- `mIoU (TP)` — range: [0, 1]
  - Mean Intersection over Union computed only over true positive predictions (IoU ≥ 0.3).
- `IoU≥0.5 (%)` — range: percent
  - Percentage of validation images where at least one predicted bounding box achieves IoU ≥ 0.5 against ground truth.
- `IoU≥0.3 (%)` — range: percent
  - Percentage of validation images where at least one predicted bounding box achieves IoU ≥ 0.3 against ground truth.

## Input / output format

**Input**: Chest X-ray image paired with a clinical question.

**Output**: Textual answer to the question, optionally containing bounding box coordinates embedded as special tokens (<loc_x1_y1_x2_y2>).

## Scoring recipe

```python
# VQA Classification
tp = sum(1 for p, g in zip(preds, golds) if p == g == "positive")
fp = sum(1 for p, g in zip(preds, golds) if p == "positive" and g != "positive")
fn = sum(1 for p, g in zip(preds, golds) if p != "positive" and g == "positive")
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

# Bounding Box Localization
ious = [compute_iou(pred_box, best_gt_box) for pred_box in pred_boxes]
tp_ious = [iou for iou in ious if iou >= 0.3]
mIoU_TP = mean(tp_ious) if tp_ious else 0
good_loc_pct = sum(1 for iou in ious if iou >= 0.5) / len(ious) * 100
acceptable_loc_pct = sum(1 for iou in ious if iou >= 0.3) / len(ious) * 100
```

## Common pitfalls

- IoU threshold of 0.3 is used for detection metrics, but 0.5 is used for per-image 'good' localization; mixing these thresholds will skew results.
- mIoU is computed only over true positives (IoU ≥ 0.3), not over all predictions, which differs from standard object detection reporting.
- Bounding box coordinates are embedded as special tokens in the text output, requiring parsing before IoU calculation.

## Evidence (verbatim from paper)

> For VQA performance, we evaluate text-based lesion classification using Accuracy, Precision, Recall, and F1 score on the validation set (659 images, 5,471 ground-truth bounding boxes), comparing the pretrained baseline (zero-shot, no fine-tuning on our dataset) against our fine-tuned model. For bounding box localization, we adopt detection metrics with IoU threshold 0.3: a predicted box is a true positive (TP) if it overlaps any ground-truth box with IoU ≥ 0.3, otherwise a false positive (FP). Missed ground-truth boxes are false negatives (FN). We report Precision, Recall, F1, and mean IoU (mIoU) computed over TPs only.

## Citation

```bibtex
@misc{nguyen2025vindrcxrvqa,
  title={VinDr-CXR-VQA: A Visual Question Answering Dataset for Explainable Chest X-Ray Analysis with Multi-Task Learning},
  author={Nguyen et al. (2025)},
  year={2025},
  note={arXiv:2511.00504}
}
```

- arXiv: 2511.00504