Hibou Pathology Eval

---
name: hibou-pathology-eval
description: Evaluates the generalization and classification capabilities of foundational vision transformers on histopathology data across patch-level tissue classification, slide-level cancer subtyping, and nuclei segmentation tasks. Use when the user wants to benchmark on CRC-100K, MHIST, PCam, MSI-CRC, MSI-STAD, TIL-DET, BRCA, NSCLC, RCC, PanNuke, or asks about evaluating this task. Reports top-1 accuracy, AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.05074
  bibtex_key: nechaev2024hibou
  confidence: high
---

# hibou-pathology-eval

> Hibou: A Family of Foundational Vision Transformers for Pathology — Nechaev et al. (2024) (arXiv:2406.05074, 2024)

## What this evaluates

Evaluates the generalization and classification capabilities of foundational vision transformers on histopathology data across patch-level tissue classification, slide-level cancer subtyping, and nuclei segmentation tasks.

## Datasets

- **CRC-100K** — total 107180; splits: train (-1), val (-1), test (-1)
- **MHIST** — total 3152; splits: train (-1), val (-1), test (-1)
- **PCam** — total 327680; splits: train (-1), val (-1), test (-1)
- **MSI-CRC** — total 193312; splits: train (-1), val (-1), test (-1)
- **MSI-STAD** — total 218578; splits: train (-1), val (-1), test (-1)
- **TIL-DET** — total 304097; splits: train (-1), val (-1), test (-1)
- **BRCA** — total 963; splits: train (-1), val (-1), test (-1)
- **NSCLC** — total 973; splits: train (-1), val (-1), test (-1)
- **RCC** — total 927; splits: train (-1), val (-1), test (-1)
- **PanNuke** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `top-1 accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly classified patches out of total test patches.
- `AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve for slide-level binary or multi-class classification.
- `Average PQ` — range: [0, 1]
  - Average Panoptic Quality across three PanNuke splits for nuclei instance segmentation.
- `Precision, Recall, F1` — range: [0, 1]
  - Standard instance segmentation metrics averaged across three PanNuke splits for each nuclear category.

## Input / output format

**Input**: H&E-stained histopathology images (patches of 96x96 to 224x224 pixels) or whole slide images (WSIs) divided into non-overlapping foreground patches.

**Output**: Class labels for patches and slides; segmentation masks and nuclear class labels for segmentation.

## Scoring recipe

```python
def evaluate_patch_level(model, dataloader):
    features = model.extract_features(dataloader)
    linear_layer = train_linear_layer(features, labels, optimizer='SGD', lr_schedule='cosine', augmentations=None)
    best_checkpoint = select_best_val_checkpoint(linear_layer)
    test_preds = best_checkpoint.predict(dataloader.test)
    return accuracy(test_preds, dataloader.test.labels)

def evaluate_slide_level(model, wsi_patches):
    patch_features = model.extract_features(wsi_patches)
    pooling_model = train_attention_pooling(patch_features, wsi_labels, optimizer='AdamW', augmentations=None)
    best_checkpoint = select_best_val_checkpoint(pooling_model)
    test_preds = best_checkpoint.predict(wsi_patches.test)
    return auc(test_preds, wsi_patches.test.labels)
```

## Common pitfalls

- Linear probing training uses no data augmentations.
- Slide-level evaluation freezes the pretrained feature extractor and only updates the attention pooling layer parameters.
- Segmentation metrics are averaged over three different PanNuke splits following the CellViT protocol, not a single split.

## Evidence (verbatim from paper)

> To evaluate our models we use public datasets and perform evaluation on both patch-level and slide-level tasks. We use a linear probing protocol. We extract features from each image using the pretrained model and then train a linear layer to perform classification. Table 1: Linear probing benchmarks reporting top-1 accuracy. Table 2: AUC, WSI subtyping benchmarks, test subset Table 3: Average PQ across the three PanNuke splits for each nuclear category.

## Citation

```bibtex
@misc{nechaev2024hibou,
  title={Hibou: A Family of Foundational Vision Transformers for Pathology},
  author={Nechaev et al. (2024)},
  year={2024},
  note={arXiv:2406.05074}
}
```

- arXiv: 2406.05074