Protein Mutational Effect Eval

---
name: protein-mutational-effect-eval
description: Evaluates a model's ability to predict the functional or stability impact of amino acid substitutions in proteins without prior experimental data for the specific variant. It probes zero-shot generalization across diverse protein families, taxonomic groups, and mutational depths (single-site vs. deep mutations). Use when the user wants to benchmark on DTm, DDG, ProteinGym, or asks about evaluating this task. Reports TPR@threshold.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.04899
  bibtex_key: tan2023p13lg
  confidence: high
---

# protein-mutational-effect-eval

> Multi-level Protein Representation Learning for Blind Mutational Effect Prediction — Yang et al. (2023) (arXiv:2306.04899, 2023)

## What this evaluates

Evaluates a model's ability to predict the functional or stability impact of amino acid substitutions in proteins without prior experimental data for the specific variant. It probes zero-shot generalization across diverse protein families, taxonomic groups, and mutational depths (single-site vs. deep mutations).

## Datasets

- **DTm** — total 2967; splits: test (-1)
- **DDG** — total 2967; splits: test (-1)
- **ProteinGym** — total ?; splits: test (-1)

## Metrics

- `TPR@threshold` **(primary)** — range: [0, 1]
  - True Positive Rate at specified percentile thresholds (5%, 25%, 50%). Measures the fraction of truly top-performing mutants correctly identified in the model's top-k predictions, where k is determined by the threshold percentage of the total dataset size.
- `Spearman's rho` — range: [-1, 1]
  - Rank-based correlation coefficient measuring the monotonic relationship between predicted mutational effect scores and experimentally measured values. Computed separately by mutational depth (single, double, all) and taxon (prokaryote, human, eukaryote, virus).

## Input / output format

**Input**: Protein domain represented as a kNN graph where nodes are residues with features extracted from a frozen ESM2-t33 prefix model, and edges encode spatial topology inferred by a 6-layer EGNN.

**Output**: Continuous predicted score representing the mutational effect (e.g., change in melting temperature ΔTm, change in Gibbs free energy ΔΔG, or relative fitness).

## Scoring recipe

```python
def compute_tpr(pred_scores, true_scores, pct):
    k = max(1, int(len(true_scores) * pct / 100))
    top_k_true = set(np.argsort(true_scores)[-k:])
    top_k_pred = set(np.argsort(pred_scores)[-k:])
    return len(top_k_true & top_k_pred) / len(top_k_true)

def compute_spearman_rho(pred_scores, true_scores):
    return scipy.stats.spearmanr(pred_scores, true_scores).correlation
```

## Common pitfalls

- TPR thresholds (5%, 25%, 50%) refer to percentile cutoffs for top-k selection, not standard precision/recall operating points.
- ProteinGym evaluation explicitly excludes the longest protein (A0A140D2T1_ZIKV_Sourisseau_growth_2019) due to AlphaFold2 folding failure, which must be accounted for in dataset construction.
- Node feature choice (ESM2 version) critically impacts performance; using larger models like t36 can degrade results compared to t33 due to over-parameterization or distribution shift.

## Evidence (verbatim from paper)

> Table 1 evaluates 100 protein assays using TPR at 5%, 25%, and 50%, wherein P13LG consistently outperforms competitors of varying model sizes. To further examine how our model efficiently achieves top performance relative to other large models, Figure 2 visualizes Spearman’s correlation from predictions of pre-trained models at different model scales.

## Citation

```bibtex
@misc{tan2023p13lg,
  title={Multi-level Protein Representation Learning for Blind Mutational Effect Prediction},
  author={Yang et al. (2023)},
  year={2023},
  note={arXiv:2306.04899}
}
```

- arXiv: 2306.04899