Hest Benchmark Eval

---
name: hest-benchmark-eval
description: Evaluates the ability of histopathology foundation models to predict gene expression levels from H&E-stained whole-slide image patches. It probes the alignment between morphological features and transcriptomic profiles across diverse cancer types and organs. Use when the user wants to benchmark on HEST-Benchmark, or asks about evaluating this task. Reports Pearson correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.16192
  bibtex_key: jaume2024hest1k
  confidence: high
---

# hest-benchmark-eval

> HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysis — Jaume et al. (2024) (arXiv:2406.16192, 2024)

## What this evaluates

Evaluates the ability of histopathology foundation models to predict gene expression levels from H&E-stained whole-slide image patches. It probes the alignment between morphological features and transcriptomic profiles across diverse cancer types and organs.

## Datasets

- **HEST-Benchmark** — total ?; splits: train (-1), test (-1); repo https://github.com/mahmoodlab/hest

## Metrics

- `Pearson correlation` **(primary)** — range: [-1, 1]
  - Computes the Pearson correlation coefficient between predicted and measured log1p-normalized gene expression values for the top 50 highly variable genes. The final reported score is the mean correlation across all genes and cross-validation folds.

## Input / output format

**Input**: 112×112 μm H&E-stained image patches (224×224 pixels at 20× magnification) from whole-slide images.

**Output**: Predicted log1p-normalized expression values for the top 50 highly variable genes per patch.

## Scoring recipe

```python
def score(predictions, gold):
    # predictions, gold: arrays of shape (n_samples, 50)
    # Compute Pearson correlation for each gene
    corrs = [pearsonr(gold[:, i], predictions[:, i])[0] for i in range(50)]
    return np.mean(corrs)
```

## Common pitfalls

- Avoid patient-level data leakage by using patient-stratified splits instead of random patch-level splits.
- Directly applying Ridge regression to models with varying embedding dimensions unfairly penalizes larger embeddings; PCA reduction to 256 factors is recommended for fair comparison.
- Performance varies significantly across cancer types, with some tasks (e.g., READ, ccRCC) showing inherently low correlation due to biological or technical noise.

## Evidence (verbatim from paper)

> All tasks are evaluated using the Pearson correlation between the predicted and measured gene expression. We report mean and standard deviation across all folds (or patients). To avoid train/test patient-level data leakage, we use patient-stratified splits, resulting in a k-fold cross-validation, where k is the number of patients.

## Citation

```bibtex
@misc{jaume2024hest1k,
  title={HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysis},
  author={Jaume et al. (2024)},
  year={2024},
  note={arXiv:2406.16192}
}
```

- arXiv: 2406.16192