Hulu Med Eval

---
name: hulu-med-eval
description: Evaluates a unified vision-language model's capability to perform holistic medical understanding across text-only queries, 2D/3D medical images, and surgical videos. It probes visual question answering, radiology report generation, clinical reasoning, and multilingual medical dialogue. Use when the user wants to benchmark on MIMIC-CXR, CheXpert, IU X-ray, MedMNIST-2D, M3D, 3D-RAD, AMOS-MM, MedFrameQA, Cholec80-VQA, EndoVis18-VQA, PSI-AVA-VQA, SurgeryVideoQA, MMedBench, RareBench, HealthBench, MMLU-Pro-Med, MedXQA, Medbullets, SGPQA, MedMCQA, MedQA, PubMedQA, MedXpertQA, MMMU-Med, OmniMedVQA, PMC-VQA, VQA-RAD, SLAKE, PathVQA, or asks about evaluating this task. Reports RaTEScore.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.08668
  bibtex_key: jiang2025hulumed
  confidence: high
---

# hulu-med-eval

> Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding — Songtao Jiang et al. (arXiv:2510.08668, 2025)

## What this evaluates

Evaluates a unified vision-language model's capability to perform holistic medical understanding across text-only queries, 2D/3D medical images, and surgical videos. It probes visual question answering, radiology report generation, clinical reasoning, and multilingual medical dialogue.

## Datasets

- **MIMIC-CXR** — total ?; splits: test (-1)
- **CheXpert** — total ?; splits: test (-1)
- **IU X-ray** — total ?; splits: test (-1)
- **MedMNIST-2D** — total ?; splits: test (-1)
- **M3D** — total ?; splits: test (-1)
- **3D-RAD** — total ?; splits: test (-1)
- **AMOS-MM** — total ?; splits: test (-1)
- **MedFrameQA** — total ?; splits: test (-1)
- **Cholec80-VQA** — total ?; splits: test (-1)
- **EndoVis18-VQA** — total ?; splits: test (-1)
- **PSI-AVA-VQA** — total ?; splits: test (-1)
- **SurgeryVideoQA** — total ?; splits: test (-1)
- **MMedBench** — total ?; splits: test (-1)
- **RareBench** — total ?; splits: test (-1)
- **HealthBench** — total ?; splits: test (-1)
- **MMLU-Pro-Med** — total ?; splits: test (-1)
- **MedXQA** — total ?; splits: test (-1)
- **Medbullets** — total ?; splits: test (-1)
- **SGPQA** — total ?; splits: test (-1)
- **MedMCQA** — total ?; splits: test (-1)
- **MedQA** — total ?; splits: test (-1)
- **PubMedQA** — total ?; splits: test (-1)
- **MedXpertQA** — total ?; splits: test (-1)
- **MMMU-Med** — total ?; splits: test (-1)
- **OmniMedVQA** — total ?; splits: test (-1)
- **PMC-VQA** — total ?; splits: test (-1)
- **VQA-RAD** — total ?; splits: test (-1)
- **SLAKE** — total ?; splits: test (-1)
- **PathVQA** — total ?; splits: test (-1)

## Metrics

- `RaTEScore` **(primary)** — range: [0, 100]
  - A clinically oriented scoring metric that evaluates the clinical utility and accuracy of generated radiology reports, often validated against board-certified radiologist decisions.
- `BLEU` — range: [0, 1]
  - N-gram precision metric that measures the overlap between generated text and reference text, penalizing for brevity.
- `ROUGE-L` — range: [0, 1]
  - Recall-oriented metric based on the longest common subsequence between predicted and reference texts.
- `METEOR` — range: [0, 1]
  - Metric that aligns predicted and reference words using exact, stem, synonym, and paraphrase matches, then computes a weighted harmonic mean of precision and recall.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly predicted answers or classifications out of the total number of instances.

## Input / output format

**Input**: Textual instruction t concatenated with visual tokens from a 2D image, 3D volume, video sequence, or text-only input.

**Output**: Autoregressively generated textual response y.

## Scoring recipe

```python
def evaluate(predictions, gold):
    acc = sum(1 for p, g in zip(predictions, gold) if p.strip() == g.strip()) / len(gold)
    bleu = compute_bleu(predictions, gold)
    rouge = compute_rouge(predictions, gold)
    meteor = compute_meteor(predictions, gold)
    rat_score = compute_rat_score(predictions, gold)
    return {'accuracy': acc, 'bleu': bleu, 'rouge': rouge, 'meteor': meteor, 'rat_score': rat_score}
```

## Common pitfalls

- 3D volumetric data must be sliced into image sequences for baseline models, which can distort temporal/spatial context if not handled uniformly across all methods.
- Automated judges (e.g., ChatGPT-4o) were used for baselines lacking reported metrics, potentially introducing judge bias or inconsistency.
- Larger model sizes do not consistently yield better performance on medical report generation tasks; domain-specific pretraining is more critical than scale.

## Evidence (verbatim from paper)

> On MRG, we assessed Hulu-Med on three standard benchmarks—MIMIC-CXR, CheXpert, and IU X-ray—using both conventional natural language metrics (BLEU, ROUGE, METEOR) and the clinically oriented RaTEScore *[[11], [73], [22], [27]]* (Fig. [2]a-b).

## Citation

```bibtex
@misc{jiang2025hulumed,
  title={Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding},
  author={Songtao Jiang et al.},
  year={2025},
  note={arXiv:2510.08668}
}
```

- arXiv: 2510.08668