Bimedx2 Medical Eval

---
name: bimedx2-medical-eval
description: Evaluates a bilingual (Arabic-English) large multimodal model's ability to understand diverse medical imaging modalities, answer visual questions, and generate or summarize clinical reports. It probes factual accuracy, clinical relevance, and linguistic quality across text-only, visual-question-answering, and report-generation tasks. Use when the user wants to benchmark on BiMed-MBench, Rad-VQA, SLAKE, Path-VQA, MIMIC-CXR, MIMIC-III, or asks about evaluating this task. Reports accuracy, F1, F1-RadGraph, GPT-4o score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.07769
  bibtex_key: mullappilly2024bimedx2
  confidence: high
---

# bimedx2-medical-eval

> BiMediX2: Bio-Medical EXpert LMM for Diverse Medical Modalities — Mullappilly et al. (2024) (arXiv:2412.07769, 2024)

## What this evaluates

Evaluates a bilingual (Arabic-English) large multimodal model's ability to understand diverse medical imaging modalities, answer visual questions, and generate or summarize clinical reports. It probes factual accuracy, clinical relevance, and linguistic quality across text-only, visual-question-answering, and report-generation tasks.

## Datasets

- **BiMed-MBench** — total 386; splits: test (386)
- **Rad-VQA** — total ?; splits: (unstated)
- **SLAKE** — total ?; splits: (unstated)
- **Path-VQA** — total ?; splits: (unstated)
- **MIMIC-CXR** — total ?; splits: (unstated)
- **MIMIC-III** — total ?; splits: (unstated)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Exact match ratio for closed-ended VQA questions. Computed as correct predictions divided by total questions.
- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall for open-ended VQA questions. Measures overlap between predicted and gold token sets.
- `BLEU-1` — range: [0, 1]
  - Unigram precision score measuring lexical overlap between generated and reference text.
- `ROUGE-L` — range: [0, 1]
  - Longest common subsequence-based recall/precision score capturing sentence-level structural similarity.
- `F1-RadGraph` **(primary)** — range: [0, 1]
  - F1 score computed over extracted medical entities and relations using the RadGraph pipeline, measuring clinical entity preservation.
- `RadCliQ` — range: [0, 1]
  - Clinical quality metric scoring the diagnostic correctness and completeness of generated radiology reports.
- `CheXbert similarity` — range: [0, 1]
  - Vector similarity score between predicted and ground-truth clinical labels extracted via the CheXbert classifier.
- `GPT-4o score` **(primary)** — range: [1, 10]
  - LLM-as-a-judge rating on a 1-10 scale evaluating correctness, helpfulness, relevance, accuracy, and level of detail against a reference or upper-bound answer.

## Input / output format

**Input**: Medical images (CXR, MRI, histology, CT, gross pathology) paired with text prompts/questions. For report tasks: one or more radiology images + prompt 'Please caption this scan with findings and impressions.' or text findings + prompt 'Summarize the findings'.

**Output**: Natural language responses: direct answers to VQA prompts, or structured medical report sections (findings/impressions) for generation/summarization tasks.

## Scoring recipe

```python
def evaluate(predictions, gold, task_type):
    if task_type == 'vqa_closed':
        return exact_match(predictions, gold)
    elif task_type == 'vqa_open':
        return f1_score(predictions, gold)
    elif task_type in ['report_gen', 'report_summ']:
        lex = compute_bleu_rouge_meteor(predictions, gold)
        clin = compute_radgraph_f1(predictions, gold)
        clin += compute_chexbert_similarity(predictions, gold)
        clin += compute_radcliq(predictions, gold)
        return lex, clin
    elif task_type == 'bimem_bench':
        ref = gold if gold else generate_upper_bound(predictions)
        return gpt4o_evaluate(pred=predictions, ref=ref,
                              criteria=['correctness','helpfulness','relevance','accuracy','detail'])
    return None
```

## Common pitfalls

- Relying solely on lexical metrics (BLEU/ROUGE) for clinical reports fails to capture diagnostic intent and clinical accuracy.
- Using cascaded translation pipelines for Arabic evaluation yields inferior results compared to dedicated bilingual instruction tuning.
- GPT-4o evaluation requires generating a high-quality upper-bound reference first; poor references bias the 1-10 scoring scale.

## Evidence (verbatim from paper)

> In both report generation and summarization tasks, relying solely on lexical metrics such as BLEU, ROUGE-L, and METEOR is insufficient, as these do not guarantee clinical accuracy. Therefore, we complement them with clinically-informed metrics such as F1-RadGraph, CheXbert similarity, and RadCliQ.

## Citation

```bibtex
@misc{mullappilly2024bimedx2,
  title={BiMediX2: Bio-Medical EXpert LMM for Diverse Medical Modalities},
  author={Mullappilly et al. (2024)},
  year={2024},
  note={arXiv:2412.07769}
}
```

- arXiv: 2412.07769