Medmnist Linear Probe Eval

---
name: medmnist-linear-probe-eval
description: Evaluates the generalization capability and scaling efficiency of self-supervised vision foundation models on a diverse suite of 12 biomedical image classification tasks. It probes how model capacity, data diversity, and pretraining objectives affect downstream diagnostic performance when using a frozen feature extractor. Use when the user wants to benchmark on MedMNIST (12 benchmarks), or asks about evaluating this task. Reports MCC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.09329
  bibtex_key: liu2025biovfm21m
  confidence: high
---

# medmnist-linear-probe-eval

> BioVFM-21M: Benchmarking and Scaling Self-Supervised Vision Foundation Models for Biomedical Image Analysis — Liu et al. (2025) (arXiv:2505.09329, 2025)

## What this evaluates

Evaluates the generalization capability and scaling efficiency of self-supervised vision foundation models on a diverse suite of 12 biomedical image classification tasks. It probes how model capacity, data diversity, and pretraining objectives affect downstream diagnostic performance when using a frozen feature extractor.

## Datasets

- **MedMNIST (12 benchmarks)** — total ?; splits: test (-1)

## Metrics

- `MCC` **(primary)** — range: [-1, 1]
  - Matthews Correlation Coefficient measures the quality of binary/multiclass classifications. Calculated as (TP*TN - FP*FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)). Ranges from -1 to 1, where 1 is perfect prediction.
- `BA` — range: [0, 1]
  - Balanced Accuracy is the average of recall obtained on each class. Computed as (TP/TP+FP + TN/TN+FN) / 2. It handles class imbalance by giving equal weight to each class.
- `F1` — range: [0, 1]
  - F1 score is the harmonic mean of precision and recall. Calculated as 2 * (Precision * Recall) / (Precision + Recall). Typically averaged macro across classes in multi-task settings.
- `AUC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve measures the model's ability to discriminate between classes across all classification thresholds. Computed by integrating the ROC curve.

## Input / output format

**Input**: Biomedical images across various modalities (X-ray, CT, ultrasound, histology, etc.) resized to the model's input resolution.

**Output**: Predicted class labels (binary or multi-class depending on the specific MedMNIST task).

## Scoring recipe

```python
def evaluate_linear_probe(model, images, labels):
    features = model.encode(images)  # Freeze backbone, extract features
    clf = LogisticRegression(max_iter=1000)
    clf.fit(features_train, labels_train)
    preds = clf.predict(features_test)
    mcc = matthews_corrcoef(labels_test, preds)
    ba = balanced_accuracy_score(labels_test, preds)
    f1 = f1_score(labels_test, preds, average='macro')
    auc = roc_auc_score(labels_test, clf.decision_function(features_test), multi_class='ovr')
    return {'MCC': mcc, 'BA': ba, 'F1': f1, 'AUC': auc}
```

## Common pitfalls

- Linear probing evaluates representation quality but may underperform compared to full fine-tuning, which is intentional for benchmarking foundation models.
- Averaging metrics across 12 heterogeneous benchmarks can obscure task-specific scaling plateaus or regressions.
- Results report 95% confidence intervals, but comparisons often rely on point estimates without formal statistical significance testing.

## Evidence (verbatim from paper)

> As shown in Table [1], BioVFM significantly outperforms existing medical foundation model BiomedGPT *[[27]]*, BiomedCLIP *[[29]]*, and RAD-DINO *[[20]]* across 12 medical benchmarks with linear-probing by at least 3.32% in MCC, 2.81% in BA, 2.14% in F1 score, and 0.94% in AUC.

## Citation

```bibtex
@misc{liu2025biovfm21m,
  title={BioVFM-21M: Benchmarking and Scaling Self-Supervised Vision Foundation Models for Biomedical Image Analysis},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2505.09329}
}
```

- arXiv: 2505.09329