Pmc Mi Bench Eval

---
name: pmc-mi-bench-eval
description: Evaluates multi-modal large language models on medical reasoning tasks involving compound figures, single images, and text-only prompts. It probes the model's ability to synthesize cross-modal information, perform clinical diagnosis, and generate accurate medical explanations across diverse imaging modalities and specialties. Use when the user wants to benchmark on PMC-MI-Bench, or asks about evaluating this task. Reports BLEU@4, Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.22232
  bibtex_key: chen2025m3llm
  confidence: high
---

# pmc-mi-bench-eval

> From Compound Figures to Composite Understanding: Developing a Multi-Modal LLM from Biomedical Literature with Medical Multiple-Image Benchmarking and Validation — Chen et al. (2025) (arXiv:2511.22232, 2025)

## What this evaluates

Evaluates multi-modal large language models on medical reasoning tasks involving compound figures, single images, and text-only prompts. It probes the model's ability to synthesize cross-modal information, perform clinical diagnosis, and generate accurate medical explanations across diverse imaging modalities and specialties.

## Datasets

- **PMC-MI-Bench** — total ?; splits: test (-1); repo https://github.com/franciszchen/M3LLM

## Metrics

- `BLEU@4` **(primary)** — range: [0, 100]
  - 4-gram precision between predicted and reference text, averaged over the dataset.
- `ROUGE-L` — range: [0, 100]
  - Longest common subsequence recall/precision between predicted and reference text.
- `BERTScore` — range: [0, 100]
  - Cosine similarity between contextual embeddings of predicted and reference tokens, aggregated via F1.
- `STS` — range: [0, 100]
  - Semantic Textual Similarity score measuring the degree of equivalence between two sentences.
- `Accuracy` **(primary)** — range: [0, 100]
  - Proportion of correctly predicted options in multi-choice VQA tasks.

## Input / output format

**Input**: Single or multiple medical images paired with a question or instruction; text-only QA receives only the text prompt.

**Output**: Free-text answer for open-ended VQA/QA tasks; selected option letter or text for multi-choice VQA.

## Scoring recipe

```python
def score(predictions, golds, task_type):
    if task_type == 'open_ended':
        bleu = nltk.translate.bleu_score.sentence_bleu([g], p, weights=(0.25,0.25,0.25,0.25)) * 100
        rouge = rouge_score(g, p, rouge_types=['rougeL']) * 100
        bert = bert_score.compute([p], [g], lang='en')
        sts = semantic_similarity(g, p) * 100
        return {'BLEU@4': bleu, 'ROUGE-L': rouge, 'BERTScore': bert, 'STS': sts}
    elif task_type == 'multi_choice':
        correct = sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g))
        return {'Accuracy': (correct / len(golds)) * 100}
```

## Common pitfalls

- BLEU@4 and ROUGE-L heavily penalize clinically correct but semantically equivalent paraphrases, potentially underestimating model capability.
- LLM-as-a-judge evaluations using GPT-4o may exhibit length bias, favoring verbose responses over concise correct answers.
- Modality-specific accuracy on public benchmarks like OmniMedVQA can be skewed if the test split is imbalanced across imaging types.

## Evidence (verbatim from paper)

> We employ a robust suite of evaluation metrics, ranging from accuracy for classification tasks to semantic metrics, including BLEU@4, ROUGE-L, BERTScore, and Semantic Textual Similarity (STS), and LLM-as-a-judge using GPT-4o for open-ended generation.

## Citation

```bibtex
@misc{chen2025m3llm,
  title={From Compound Figures to Composite Understanding: Developing a Multi-Modal LLM from Biomedical Literature with Medical Multiple-Image Benchmarking and Validation},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2511.22232}
}
```

- arXiv: 2511.22232