Vimed Pet Eval

---
name: vimed-pet-eval
description: Evaluates vision-language models on generating Vietnamese clinical reports from paired PET/CT images and answering medical questions about them. Probes the model's ability to align 3D medical imaging features with low-resource language text and produce clinically accurate descriptions. Use when the user wants to benchmark on ViMed-PET, or asks about evaluating this task. Reports BLEU-4.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.24739
  bibtex_key: nguyen2025vimedpet
  confidence: high
---

# vimed-pet-eval

> Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation — Nguyen et al. (2025) (arXiv:2509.24739, 2025)

## What this evaluates

Evaluates vision-language models on generating Vietnamese clinical reports from paired PET/CT images and answering medical questions about them. Probes the model's ability to align 3D medical imaging features with low-resource language text and produce clinically accurate descriptions.

## Datasets

- **ViMed-PET** — total ?; splits: test (-1)

## Metrics

- `BLEU-4` **(primary)** — range: [0, 1]
  - 4-gram precision with a brevity penalty to penalize overly short or fragmented generations.
- `ROUGE-1` — range: [0, 1]
  - Unigram recall/precision measuring word-level overlap between generated and reference text.
- `ROUGE-L` — range: [0, 1]
  - Longest common subsequence score capturing sentence-level fluency and structural alignment.
- `BERT score` — range: [0, 1]
  - Contextual embedding similarity (F1) between generated and reference texts using a pretrained BERT model.
- `F1-T` — range: [0, 1]
  - Clinical entity-level F1 score for general medical terms.
- `F1-TP` — range: [0, 1]
  - Clinical entity-level F1 score for tumor-related terms.
- `F1-TF` — range: [0, 1]
  - Clinical entity-level F1 score for treatment-related terms.
- `F1-TPF` — range: [0, 1]
  - Clinical entity-level F1 score for combined tumor, treatment, and follow-up terms.

## Input / output format

**Input**: Paired 3D PET/CT images, and for the VQA task, a natural language question about the scan.

**Output**: A Vietnamese clinical report (for report generation) or a short text answer (for VQA).

## Scoring recipe

```python
def compute_metrics(predictions, references):
    bleu4 = compute_bleu(predictions, references, n=4)
    rouge1 = compute_rouge(predictions, references, n=1)
    rouge_l = compute_rouge(predictions, references, n='l')
    bert = compute_bert_score(predictions, references)
    # Clinical F1 metrics require entity-level matching specific to Vietnamese medical terminology
    f1_t, f1_tp, f1_tf, f1_tpf = compute_clinical_f1(predictions, references)
    return {'BLEU-4': bleu4, 'ROUGE-1': rouge1, 'ROUGE-L': rouge_l, 'BERT score': bert,
            'F1-T': f1_t, 'F1-TP': f1_tp, 'F1-TF': f1_tf, 'F1-TPF': f1_tpf}
```

## Common pitfalls

- Metrics are reported as absolute scores, but the text heavily emphasizes relative percentage gains over baselines, which can cause confusion if not normalized correctly.
- Clinical F1 metrics (F1-T, F1-TP, etc.) are not standard NLP metrics and likely require entity-level matching specific to Vietnamese medical terminology, which is not detailed in the text.
- GPT-4o is evaluated via few-shot prompting while other models are fine-tuned, making direct comparison potentially unfair.

## Evidence (verbatim from paper)

> For instance, when LLaMA-2-7B is paired with either CT-ViT or our customized Cosmos Tokenizer and fine-tuned on ViMed-PET, it significantly outperforms the pretrained LLaMA-2-7B model used in M3D across all key metrics (i.e., BLEU-4, ROUGE-1, ROUGE-L, and BERT score).

## Citation

```bibtex
@misc{nguyen2025vimedpet,
  title={Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation},
  author={Nguyen et al. (2025)},
  year={2025},
  note={arXiv:2509.24739}
}
```

- arXiv: 2509.24739