Deepprotein Benchmark Eval

---
name: deepprotein-benchmark-eval
description: Evaluates deep learning models on a comprehensive suite of protein sequence learning tasks, including function prediction, subcellular localization, protein-protein interaction, epitope/paratope prediction, antibody developability, CRISPR repair outcomes, and protein structure prediction. Use when the user wants to benchmark on Fluorescence, Stability, β-lactamase, Solubility, Subcellular, Binary, PPI Affinity, Yeast, Human PPI, IEDB, PDB-Jespersen, SAbDab-Liberis, TAP, SAbDab-Chen, CRISPR-Leenay, Fold, Secondary Structure, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.02023
  bibtex_key: xie2024deepprotein
  confidence: medium
---

# deepprotein-benchmark-eval

> DeepProtein: Deep Learning Library and Benchmark for Protein Sequence Learning — Jiaqing Xie et al. (2024) (arXiv:2410.02023, 2024)

## What this evaluates

Evaluates deep learning models on a comprehensive suite of protein sequence learning tasks, including function prediction, subcellular localization, protein-protein interaction, epitope/paratope prediction, antibody developability, CRISPR repair outcomes, and protein structure prediction.

## Datasets

- **Fluorescence** — total 54025; splits: train (-1), val (-1), test (-1)
- **Stability** — total 68934; splits: train (-1), val (-1), test (-1)
- **β-lactamase** — total 5198; splits: train (-1), val (-1), test (-1)
- **Solubility** — total 71419; splits: train (-1), val (-1), test (-1)
- **Subcellular** — total 13961; splits: train (-1), val (-1), test (-1)
- **Binary** — total 8634; splits: train (-1), val (-1), test (-1)
- **PPI Affinity** — total 2682; splits: train (-1), val (-1), test (-1)
- **Yeast** — total 2172; splits: train (-1), val (-1), test (-1)
- **Human PPI** — total 7348; splits: train (-1), val (-1), test (-1)
- **IEDB** — total 3159; splits: train (-1), val (-1), test (-1)
- **PDB-Jespersen** — total 447; splits: train (-1), val (-1), test (-1)
- **SAbDab-Liberis** — total 1023; splits: train (-1), val (-1), test (-1)
- **TAP** — total 242; splits: train (-1), val (-1), test (-1)
- **SAbDab-Chen** — total 2409; splits: train (-1), val (-1), test (-1)
- **CRISPR-Leenay** — total 1521; splits: train (-1), val (-1), test (-1)
- **Fold** — total 13766; splits: train (-1), val (-1), test (-1)
- **Secondary Structure** — total 11361; splits: train (-1), val (-1), test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted labels over total samples. Used for classification tasks (e.g., Solubility, Subcellular, PPI, Fold).
- `MSE` — range: [0, inf)
  - Mean Squared Error between predicted and ground-truth continuous values. Used for regression tasks (e.g., Fluorescence, Stability, β-lactamase, PPI Affinity, TAP, SAbDab-Chen).

## Input / output format

**Input**: Amino acid sequences (single or paired for PPI/developability), optionally augmented with structural features or edge information for graph-based models.

**Output**: Predicted class labels, binary indicators, or continuous values depending on the task.

## Scoring recipe

```python
def compute_metric(predictions, gold, task_type):
    if task_type == 'classification':
        return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif task_type == 'regression':
        return sum((p - g) ** 2 for p, g in zip(predictions, gold)) / len(gold)
    elif task_type == 'residue_level':
        return residue_level_aggregate(predictions, gold)
    return None
```

## Common pitfalls

- Splits strictly follow PEER benchmark and TDC conventions rather than random splits; using random splits causes data leakage and invalidates results.
- Residue-level tasks (epitope, paratope, secondary structure) require sequence-to-sequence alignment and residue-level aggregation, not sequence-level averaging.
- Paired-input tasks (PPI, developability) must maintain correct sequence pairing during batching and evaluation to avoid mismatched labels.

## Evidence (verbatim from paper)

> In this library, we follow the train-validation-test split in PEER benchmark (Xu et al., 2022) and TDC (Huang et al., 2022). Each individual split is reported from Table 2 to 7.

## Citation

```bibtex
@misc{xie2024deepprotein,
  title={DeepProtein: Deep Learning Library and Benchmark for Protein Sequence Learning},
  author={Jiaqing Xie et al. (2024)},
  year={2024},
  note={arXiv:2410.02023}
}
```

- arXiv: 2410.02023