Mimic Vqa Eval

---
name: mimic-vqa-eval
description: Evaluates a model's ability to answer clinical questions about chest X-rays, focusing on disease presence, type, location, and severity. It probes multi-modal reasoning by requiring the model to correlate image regions with structured medical knowledge and spatial/semantic relationships. Use when the user wants to benchmark on Mimic-VQA, or asks about evaluating this task. Reports AUC-micro.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2302.09636
  bibtex_key: hu2023interpretable
  confidence: high
---

# mimic-vqa-eval

> Interpretable Medical Image Visual Question Answering via Multi-Modal Relationship Graph Learning — Hu et al. (2023) (arXiv:2302.09636, 2023)

## What this evaluates

Evaluates a model's ability to answer clinical questions about chest X-rays, focusing on disease presence, type, location, and severity. It probes multi-modal reasoning by requiring the model to correlate image regions with structured medical knowledge and spatial/semantic relationships.

## Datasets

- **Mimic-VQA** — total 297723; splits: train (-1), val (-1), test (-1)

## Metrics

- `AUC-micro` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve. AUC-micro computes the final AUC by aggregating the contributions of each class across all samples before computing the curve.
- `AUC-macro` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve. AUC-macro treats all classes equally and computes the average AUC across the 169 answer categories.

## Input / output format

**Input**: A chest X-ray image and a natural language question (e.g., about abnormality, presence, location, type, or level).

**Output**: A set of predicted answer scores/probabilities for 169 possible answer classes, with top predictions selected if score > 0.04.

## Scoring recipe

```python
def compute_auc(y_true, y_pred, average='micro'):
    if average == 'micro':
        fpr, tpr, _ = roc_curve(y_true.ravel(), y_pred.ravel())
        return auc(fpr, tpr)
    else:
        fprs, tprs = [], []
        for i in range(y_true.shape[1]):
            fpr, tpr, _ = roc_curve(y_true[:, i], y_pred[:, i])
            fprs.append(fpr)
            tprs.append(tpr)
        mean_fpr = np.linspace(0, 1, 100)
        mean_tpr = np.interp(mean_fpr, fprs[0], tprs[0])
        for fpr, tpr in zip(fprs[1:], tprs[1:]):
            mean_tpr = np.maximum(mean_tpr, np.interp(mean_fpr, fpr, tpr))
        return auc(mean_fpr, mean_tpr)
```

## Common pitfalls

- Sequential train/val/test split (8:1:1) may cause data leakage or distribution shift if images from the same study appear across splits.
- AUC-micro vs AUC-macro handle class imbalance differently; micro favors majority classes while macro treats all 169 answer classes equally.
- The dataset filters rare answers to 169 classes, which may not reflect real-world long-tail disease distributions.

## Evidence (verbatim from paper)

> We used the AUC as the evaluation metric. AUC-micro computes the final AUC by aggregating the contributions of each class. AUC-macro treats all classes equally and computes the average AUC.

## Citation

```bibtex
@misc{hu2023interpretable,
  title={Interpretable Medical Image Visual Question Answering via Multi-Modal Relationship Graph Learning},
  author={Hu et al. (2023)},
  year={2023},
  note={arXiv:2302.09636}
}
```

- arXiv: 2302.09636