Pheno Ca Eval

---
name: pheno-ca-eval
description: Evaluates deep neural networks on phenotypic drug discovery tasks using high-content screening images. It probes the model's ability to deconvolve mechanisms of action, molecular targets, and compound identities from cellular phenotypes, as well as zero-shot compound retrieval for CRISPR perturbations. Use when the user wants to benchmark on Pheno-CA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.16773
  bibtex_key: linsley2023neural
  confidence: high
---

# pheno-ca-eval

> Neural scaling laws for phenotypic drug discovery — Linsley et al. (2023) (arXiv:2309.16773, 2023)

## What this evaluates

Evaluates deep neural networks on phenotypic drug discovery tasks using high-content screening images. It probes the model's ability to deconvolve mechanisms of action, molecular targets, and compound identities from cellular phenotypes, as well as zero-shot compound retrieval for CRISPR perturbations.

## Datasets

- **Pheno-CA** — total ?; splits: train (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted classes out of total predictions for MoA, target, and molecule deconvolution tasks.
- `CCE` — range: [0, ∞]
  - Standard categorical cross-entropy computed on molecule identity predictions.
- `rank-order` — range: other
  - Rank-order of representational distance between CRISPR perturbation phenotypes and molecule phenotypes to find matching targets.

## Input / output format

**Input**: High-content screening (iHCS) images of cells perturbed by compounds. For compound discovery, also includes images of CRISPR-perturbed targets.

**Output**: Class labels for MoA (1,282 classes), target (942 classes), or molecule identity (2,919 classes). For compound discovery, a ranked list of compounds based on representational distance.

## Scoring recipe

```python
def compute_accuracy(preds, gold):
    return sum(p == g for p, g in zip(preds, gold)) / len(gold)

def compute_cce(preds, gold):
    return -sum(g * log(p) for p, g in zip(preds, gold)) / len(gold)

def compute_rank_order(query_rep, cand_reps, gold_idx):
    dists = [cosine_dist(query_rep, c) for c in cand_reps]
    return sorted(range(len(dists)), key=lambda i: dists[i]).index(gold_idx)
```

## Common pitfalls

- IBP-pretrained models are evaluated with frozen weights and a separate 3-layer MLP probe, not fine-tuned end-to-end on the target task.
- Scaling laws are derived using out-of-distribution (OOD) molecules, not the in-distribution Pheno-CA training set.
- Compound discovery is a zero-shot retrieval task; models are not trained on CRISPR perturbation labels.

## Evidence (verbatim from paper)

> Each DNN in our zoo was given images of cells perturbed by different compounds, and trained to predict the MoA of a given compound out of 1,282 possibilities (Fig.1a). DNNs were either supervised directly for MoA deconvolution or pretrained with IBP (Fig.2a). Next, DNN weights were frozen and three-layer MLP probes were used to transform image representations from both models into MoA predictions (i.e. there was no direct task supervision for IBP models). Our DNN zoo yielded a wide range of performances on this task. At the low end was a 12.09% accurate 12-layer and 128-feature DNN trained with IBP on 100% of out-of-distribution molecules but only 0.01% of the replicates of each compound. At the high-end was a 52.62% accurate 9-layer and 1512-feature DNN trained with IBP on 100% of out-of-distribution molecules and 75% of the replicates of each compound.

## Citation

```bibtex
@misc{linsley2023neural,
  title={Neural scaling laws for phenotypic drug discovery},
  author={Linsley et al. (2023)},
  year={2023},
  note={arXiv:2309.16773}
}
```

- arXiv: 2309.16773