Histo Vl Eval

---
name: histo-vl-eval
description: Evaluates vision-language foundation models on diverse histopathology clinical tasks (detection, subtyping, grading, mutation prediction) to assess their robustness to textual/visual perturbations, magnification changes, stain normalization, and model calibration. Use when the user wants to benchmark on CRC-100K, BreakHist, DataBiox, GasHisSDB, Breast IDC, LC25000-lung, or asks about evaluating this task. Reports balanced accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.12990
  bibtex_key: almajzoub2025histovl
  confidence: high
---

# histo-vl-eval

> How Good is my Histopathology Vision-Language Foundation Model? A Holistic Benchmark — Al Majzoub et al. (2025) (arXiv:2503.12990, 2025)

## What this evaluates

Evaluates vision-language foundation models on diverse histopathology clinical tasks (detection, subtyping, grading, mutation prediction) to assess their robustness to textual/visual perturbations, magnification changes, stain normalization, and model calibration.

## Datasets

- **CRC-100K** — total ?; splits: test (-1)
- **BreakHist** — total ?; splits: test (-1)
- **DataBiox** — total ?; splits: test (-1)
- **GasHisSDB** — total ?; splits: test (-1)
- **Breast IDC** — total ?; splits: test (-1)
- **LC25000-lung** — total ?; splits: test (-1)

## Metrics

- `balanced accuracy` **(primary)** — range: [0, 1]
  - Mean of recall scores across all classes. For binary tasks, it equals (TPR + TNR) / 2. It mitigates class imbalance by treating each class equally regardless of sample count.
- `F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).
- `precision` — range: [0, 1]
  - Ratio of true positive predictions to all positive predictions: TP / (TP + FP).
- `MCC` — range: [-1, 1]
  - Matthews Correlation Coefficient: (TP*TN - FP*FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)). Ranges from -1 to 1, with 1 representing perfect prediction.
- `ECE` — range: [0, 1]
  - Expected Calibration Error: weighted sum of absolute differences between predicted confidence and actual accuracy across probability bins. Lower values indicate better calibration.

## Input / output format

**Input**: Histopathology image patches extracted from whole slide images, paired with a single or ensemble textual caption/prompt describing the tissue, magnification, or class.

**Output**: Predicted class labels or probability distributions for tasks including binary detection, multi-class subtyping, grading, and mutation prediction.

## Scoring recipe

```python
def balanced_accuracy(y_true, y_pred):
    recalls = [np.mean(y_true[y==c] == c) for c in np.unique(y_true)]
    return np.mean(recalls)

def ece(y_true, y_prob, n_bins=15):
    confidences = np.max(y_prob, axis=1)
    predictions = np.argmax(y_prob, axis=1)
    accuracies = (predictions == y_true).astype(float)
    bin_boundaries = np.linspace(0, 1, n_bins + 1)
    ece = 0.0
    for i in range(n_bins):
        mask = (confidences > bin_boundaries[i]) & (confidences <= bin_boundaries[i+1])
        if np.sum(mask) > 0:
            bin_acc = np.mean(accuracies[mask])
            bin_conf = np.mean(confidences[mask])
            ece += np.sum(mask) / len(y_true) * abs(bin_acc - bin_conf)
    return ece
```

## Common pitfalls

- Models exhibit high sensitivity to prompt/caption variations, with balanced accuracy fluctuating by up to 26% across different textual descriptions of the same image.
- High balanced accuracy does not imply reliable uncertainty estimation; models consistently show high ECE and low confidence, indicating severe miscalibration for clinical deployment.
- Performance is heavily confounded by preprocessing choices, particularly stain normalization and magnification level, which vary across datasets and affect cellular vs. tissue-level information.

## Evidence (verbatim from paper)

> As per Figure 5(a), all models exhibit high ECE values across tasks. A general trend of the highest ECE values in tissue phenotyping, followed by TIL detection, MSI detection, and cancer grading is observed.

## Citation

```bibtex
@misc{almajzoub2025histovl,
  title={How Good is my Histopathology Vision-Language Foundation Model? A Holistic Benchmark},
  author={Al Majzoub et al. (2025)},
  year={2025},
  note={arXiv:2503.12990}
}
```

- arXiv: 2503.12990