Gene Bench Eval

---
name: gene-bench-eval
description: This benchmark evaluates how well various deep learning models encode biological knowledge by predicting 312 ground-truth gene properties. It probes capabilities across five domains: genomic, regulatory, localization, biological processes, and protein features, using binary, multi-label, and multi-class classification tasks. Use when the user wants to benchmark on GeneBench, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.04075
  bibtex_key: kan-tor2024genebench
  confidence: high
---

# gene-bench-eval

> Does your model understand genes? A benchmark of gene properties for biological and text models — Kan-Tor et al. (2024) (arXiv:2412.04075, 2024)

## What this evaluates

This benchmark evaluates how well various deep learning models encode biological knowledge by predicting 312 ground-truth gene properties. It probes capabilities across five domains: genomic, regulatory, localization, biological processes, and protein features, using binary, multi-label, and multi-class classification tasks.

## Datasets

- **GeneBench** — total 312; splits: test (-1); repo http://github.com/BiomedSciAI/gene-benchmark

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve, measuring the probability that a randomly chosen positive instance ranks higher than a randomly chosen negative instance.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall, calculated as 2 * (precision * recall) / (precision + recall).

## Input / output format

**Input**: Pre-computed gene embeddings (vectors) from text models, scRNA foundation models, DNA/protein sequence models, or classical baselines.

**Output**: Predicted probabilities or class labels for each of the 312 gene property tasks (binary, multi-label, or multi-class).

## Scoring recipe

```python
# For each of the 312 gene property tasks:
clf = LogisticRegression()
clf.fit(X_train, y_train)
y_prob = clf.predict_proba(X_test)[:, 1]
auc = roc_auc_score(y_test, y_prob)
# F1 is computed similarly using thresholded predictions
f1 = f1_score(y_test, clf.predict(X_test), average='weighted')
```

## Common pitfalls

- Larger model size does not consistently yield better performance; smaller models often match or approach larger ones.
- Text-based models sometimes perform comparably to classical bag-of-words baselines, highlighting that raw embedding size isn't the sole driver of biological knowledge.
- Using non-linear models (e.g., MLPs) can introduce hyperparameter sensitivity; the authors recommend linear classifiers for robust cross-model comparison.

## Evidence (verbatim from paper)

> The performance of each model on the task families as measured by average area under the ROC curve. Parentheses show the corresponding standard deviation across all tasks of the same family. These trends are consistent when using other evaluation metrics such as F1 (see Figure[S1]).

## Citation

```bibtex
@misc{kan-tor2024genebench,
  title={Does your model understand genes? A benchmark of gene properties for biological and text models},
  author={Kan-Tor et al. (2024)},
  year={2024},
  note={arXiv:2412.04075}
}
```

- arXiv: 2412.04075