Protein Function Benchmark Eval

---
name: protein-function-benchmark-eval
description: Evaluates protein language models on classification and regression tasks to assess their capability in predicting protein properties, sub-cellular localization, epitope regions, and mutational effects. Use when the user wants to benchmark on Sub-cellular Localization, Membrane Solubility, Epitope Region Prediction, GB1 Mutational Landscape, or asks about evaluating this task. Reports accuracy, Spearman's rank correlation coefficient.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.13519
  bibtex_key: [REDACTED]
  confidence: high
---

# protein-function-benchmark-eval

> Open-Source Protein Language Models for Function Prediction and Protein Design — Pandi et al. (2024) (arXiv:2412.13519, 2024)

## What this evaluates

Evaluates protein language models on classification and regression tasks to assess their capability in predicting protein properties, sub-cellular localization, epitope regions, and mutational effects.

## Datasets

- **Sub-cellular Localization** — total ?; splits: test (-1)
- **Membrane Solubility** — total ?; splits: test (-1)
- **Epitope Region Prediction** — total ?; splits: test (-1)
- **GB1 Mutational Landscape** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted labels out of total predictions. Calculated as the number of correct predictions divided by the total number of instances.
- `Spearman's rank correlation coefficient` **(primary)** — range: [-1, 1]
  - Rank-based measure of statistical dependence between predicted and actual continuous values. Computed as the Pearson correlation coefficient between the rank-ordered predictions and gold labels.

## Input / output format

**Input**: Raw protein amino acid sequences.

**Output**: For classification tasks: discrete predicted labels (e.g., cellular compartment, soluble/membrane, epitope region). For regression tasks: continuous activity scores.

## Scoring recipe

```python
if task_type == 'classification':
    metric = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
elif task_type == 'regression':
    metric = spearmanr(predictions, golds).correlation
```

## Common pitfalls

- Datasets are loaded via DeepChem's built-in utilities rather than explicit downloads, making exact train/val/test splits and preprocessing steps difficult to reproduce without matching library versions.
- The paper notes pretraining was limited to 1M sequences instead of the original 216M, which may cause performance gaps compared to the official ProtBERT baseline.
- Evaluation uses a one-layer MLP on top of embeddings, but exact fine-tuning hyperparameters (learning rate, batch size, epochs) are omitted, hindering exact replication.

## Evidence (verbatim from paper)

> For each classification task, we recorded the accuracy, while for the regression tasks, we reported Spearman's rank correlation coefficient  $(\rho)$ .

## Citation

```bibtex
@misc{pandi2024openprotein,
  title={Open-Source Protein Language Models for Function Prediction and Protein Design},
  author={Pandi et al. (2024)},
  year={2024},
  note={arXiv:2412.13519}
}
```

- arXiv: 2412.13519