Medmax Eval

---
name: medmax-eval
description: Evaluates biomedical multimodal foundation models across visual question answering, image captioning, image generation, visual chat, and interleaved text-image generation. It probes the model's ability to understand medical images, reason over clinical reports, and generate clinically grounded multimodal responses. Use when the user wants to benchmark on VQA-RAD, SLAKE, PathVQA, QuiltVQA, PMC-VQA, PathMMU, ProbMed, OmniMedVQA, PMC-OA, MIMIC-CXR, Quilt-1M, LLaVA-Med, MedMax-Instruct, or asks about evaluating this task. Reports Accuracy (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.12661
  bibtex_key: bansal2024medmax
  confidence: high
---

# medmax-eval

> MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants — Bansal et al. (2024) (arXiv:2412.12661, 2024)

## What this evaluates

Evaluates biomedical multimodal foundation models across visual question answering, image captioning, image generation, visual chat, and interleaved text-image generation. It probes the model's ability to understand medical images, reason over clinical reports, and generate clinically grounded multimodal responses.

## Datasets

- **VQA-RAD** — total ?; splits: test (-1)
- **SLAKE** — total ?; splits: test (-1)
- **PathVQA** — total ?; splits: test (-1)
- **QuiltVQA** — total ?; splits: test (-1)
- **PMC-VQA** — total ?; splits: test (-1)
- **PathMMU** — total ?; splits: validation (-1)
- **ProbMed** — total ?; splits: test (-1)
- **OmniMedVQA** — total 1000; splits: hidden (1000)
- **PMC-OA** — total 800; splits: test (800)
- **MIMIC-CXR** — total 400; splits: test (400)
- **Quilt-1M** — total 400; splits: unseen (400)
- **LLaVA-Med** — total 193; splits: test (193)
- **MedMax-Instruct** — total 500; splits: hidden (500)

## Metrics

- `Accuracy (EM)` **(primary)** — range: [0, 1]
  - Exact match between the model's predicted answer and the ground-truth answer. Returns 1 if they match exactly, 0 otherwise. Averaged over all instances.
- `Accuracy (LLM)` — range: [0, 1]
  - An LLM (GPT-4o-mini) compares the predicted answer against the ground-truth answer and assigns a score of 0 or 1 based on reliability. Averaged over open-ended questions.
- `BioMedCLIPScore` — range: [0, 1]
  - Computes the similarity score from the BioMedCLIP model between the input image and the predicted caption (or generated image).
- `LLM score` — range: [0, 1]
  - An LLM scores the predicted answer out of 10 conditioned on the question and image context. The final metric is the average relative prediction score: score(predicted) / score(reference).
- `Image-Image BioMedCLIPScore` — range: [0, 1]
  - Computes the image-image similarity score from the BioMedCLIP model between the generated image and the reference image.

## Input / output format

**Input**: Multimodal input consisting of a medical image paired with a text prompt (question, captioning instruction, or generation query).

**Output**: Text response (answer, caption, or chatbot reply) or interleaved text-image response for generation tasks.

## Scoring recipe

```python
def score_vqa(preds, golds, mode):
    scores = []
    for p, g in zip(preds, golds):
        if mode == 'EM':
            scores.append(1.0 if p.strip() == g.strip() else 0.0)
        elif mode == 'LLM':
            scores.append(llm_compare(p, g)) # GPT-4o-mini returns 0 or 1
    return sum(scores) / len(scores)

def score_caption_gen(image, pred_caption):
    return biomedclip_similarity(image, pred_caption)

def score_chatbot(pred_ans, ref_ans, question, image):
    pred_score = llm_score(pred_ans, question, image) # out of 10
    ref_score = llm_score(ref_ans, question, image) # out of 10
    return pred_score / ref_score
```

## Common pitfalls

- Open-ended VQA answers are subjective; using exact match instead of LLM evaluation unfairly penalizes semantically correct but differently phrased answers.
- The LLM score for chatbot/generation is a relative ratio (predicted/reference), not an absolute score, which can amplify reference quality differences.
- Contamination must be explicitly checked: ensure no exact matches between instruction-tuning data and evaluation splits, as noted in the paper's contamination analysis.

## Evidence (verbatim from paper)

> Subsequently, we compute the BioMedCLIPScore to assess the closeness between the input image (caption) and predicted caption (image).

## Citation

```bibtex
@misc{bansal2024medmax,
  title={MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants},
  author={Bansal et al. (2024)},
  year={2024},
  note={arXiv:2412.12661}
}
```

- arXiv: 2412.12661