Medvilam Medical Bench Eval

---
name: medvilam-medical-bench-eval
description: Evaluates a multimodal LLM's ability to perform disease classification, visual grounding, and zero-shot reasoning across diverse medical imaging modalities (X-ray, CT, MRI, ultrasound, endoscopy, video) and tasks. Use when the user wants to benchmark on Chest X-ray (10 test sets from 5 public + 5 private), ImageCAS, ASOCA, CCA200, LPPolypVideo/SUN-SEG/CVC-12k, EndoVis18, LDPolyVideo, ISIC16, HAM10000, TN3K, BUID, TBX11K, RSNA Pneumonia, Luna16, DeepLesion, ADNI, LGG, Object-CXR, or asks about evaluating this task. Reports Accuracy (ACC).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.19684
  bibtex_key: xu2024medvilam
  confidence: high
---

# medvilam-medical-bench-eval

> MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation — Lijian Xu et al. (2024) (arXiv:2409.19684, 2024)

## What this evaluates

Evaluates a multimodal LLM's ability to perform disease classification, visual grounding, and zero-shot reasoning across diverse medical imaging modalities (X-ray, CT, MRI, ultrasound, endoscopy, video) and tasks.

## Datasets

- **Chest X-ray (10 test sets from 5 public + 5 private)** — total ?; splits: test (-1)
- **ImageCAS, ASOCA, CCA200** — total ?; splits: test (-1)
- **LPPolypVideo/SUN-SEG/CVC-12k** — total ?; splits: test (-1)
- **EndoVis18, LDPolyVideo** — total ?; splits: test (-1)
- **ISIC16, HAM10000** — total ?; splits: test (-1)
- **TN3K, BUID** — total ?; splits: test (-1)
- **TBX11K, RSNA Pneumonia** — total ?; splits: test (-1)
- **Luna16, DeepLesion** — total ?; splits: test (-1)
- **ADNI, LGG** — total ?; splits: test (-1)
- **Object-CXR** — total ?; splits: test (-1)

## Metrics

- `Accuracy (ACC)` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted labels or bounding boxes out of total ground truth instances. Calculated as correct predictions divided by total samples.
- `AUC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve. Measures the model's ability to distinguish between classes across all classification thresholds.
- `Acc@0.5` — range: [0, 1]
  - Accuracy for referring bounding box detection where a predicted box is considered correct if its Intersection over Union (IoU) with the ground truth box is at least 0.5.

## Input / output format

**Input**: Medical image (2D/3D/video) or audio paired with a natural language instruction or question (e.g., 'Is there a plaque on LM?', 'Where is the plaque on LM?', 'Is there anything foreign in this x-ray...?').

**Output**: Text response containing classification labels, yes/no answers, descriptive explanations, or bounding box coordinates formatted as 'center at [x, y, z], box length is [a, b, c]'.

## Scoring recipe

```python
def compute_acc(pred_labels, gt_labels):
    return sum(p == g for p, g in zip(pred_labels, gt_labels)) / len(gt_labels)

def compute_auc_f1(pred_scores, gt_labels):
    auc = roc_auc_score(gt_labels, pred_scores)
    f1 = f1_score(gt_labels, (pred_scores > 0.5).astype(int))
    return auc, f1

def compute_acc_at_iou(pred_boxes, gt_boxes, iou_thresh=0.5):
    correct = 0
    for p, g in zip(pred_boxes, gt_boxes):
        if calculate_iou(p, g) >= iou_thresh:
            correct += 1
    return correct / len(gt_boxes)
```

## Common pitfalls

- Dataset splits and exact sizes are rarely disclosed; many are private or lack standard train/val/test splits, making reproducibility difficult.
- Acc@0.5 implies an IoU threshold of 0.5 for bounding box matching, but the exact matching logic (e.g., one-to-one vs. one-to-many assignment) is not specified.
- 3D grounding outputs use a specific coordinate format that requires precise parsing to evaluate correctly, and small lesion sizes in CT datasets cause near-zero accuracy regardless of model capability.

## Evidence (verbatim from paper)

> The overall accuracy (ACC) for plaque classification task was 30.1%/32.6%/34.5%, showing variations in performance across different coronary artery branches, likely due to differences in their morphological characteristics. For the visual grounding task of plaque localization, the overall ACC was 70.1%/73.2%/75.1%. Acc@0.5 is applied to evaluate methods.

## Citation

```bibtex
@misc{xu2024medvilam,
  title={MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation},
  author={Lijian Xu et al. (2024)},
  year={2024},
  note={arXiv:2409.19684}
}
```

- arXiv: 2409.19684