Champkit Eval

---
name: champkit-eval
description: Evaluates the transfer learning capability and generalization of deep learning models (CNNs and ViTs) on patch-level histopathology image classification tasks across multiple cancer-related benchmarks. Use when the user wants to benchmark on Various publicly available histopathology datasets, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2206.06862
  bibtex_key: kaczmarzyk2022champkit
  confidence: high
---

# champkit-eval

> Evaluating histopathology transfer learning with ChampKit — Kaczmarzyk et al. (2022) (arXiv:2206.06862, 2022)

## What this evaluates

Evaluates the transfer learning capability and generalization of deep learning models (CNNs and ViTs) on patch-level histopathology image classification tasks across multiple cancer-related benchmarks.

## Datasets

- **Various publicly available histopathology datasets** — total ?; splits: test (-1); repo https://github.com/kaczmarj/champkit

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area under the receiver operating characteristic curve, measuring the trade-off between true positive rate and false positive rate across all classification thresholds.
- `F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall, computed at a fixed decision threshold of 0.5.

## Input / output format

**Input**: 224x224 RGB histopathology image patches, normalized to ImageNet statistics or mean/std of 0.5 depending on the pretraining regime.

**Output**: Class probabilities or logits for patch-level classification, thresholded at 0.5 for F1 calculation.

## Scoring recipe

```python
def compute_metrics(predictions, labels):
    threshold = 0.5
    preds_binary = (predictions > threshold).astype(int)
    f1 = f1_score(labels, preds_binary, average='macro')
    auroc = roc_auc_score(labels, predictions)
    return auroc, f1
```

## Common pitfalls

- Using inconsistent hyperparameters across models or datasets invalidates the fair comparison the framework aims to provide.
- Evaluating generalization on a single dataset rather than across multiple tasks masks poor cross-task transfer performance.
- Mismatching image normalization (ImageNet vs. 0.5) between pretraining and fine-tuning can degrade performance.

## Evidence (verbatim from paper)

> Area under the receiver operating characteristic curve (AUROC) and F1-score (threshold=0.5) were calculated (using torchmetrics [66].

## Citation

```bibtex
@misc{kaczmarzyk2022champkit,
  title={Evaluating histopathology transfer learning with ChampKit},
  author={Kaczmarzyk et al. (2022)},
  year={2022},
  note={arXiv:2206.06862}
}
```

- arXiv: 2206.06862