Protap Eval

---
name: protap-eval
description: Evaluates protein language models and geometric deep learning architectures on five realistic downstream biological tasks, including binding affinity prediction, functional annotation, mutation effects, cleavage site detection, and PROTAC interaction modeling. It probes how pretraining objectives, structural information integration, and domain-specific inductive biases affect generalization on limited biological data. Use when the user wants to benchmark on Protap Benchmark, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.02052
  bibtex_key: yan2025protap
  confidence: high
---

# protap-eval

> Protap: A Benchmark for Protein Modeling on Realistic Downstream Applications — Yan et al. (2025) (arXiv:2506.02052, 2025)

## What this evaluates

Evaluates protein language models and geometric deep learning architectures on five realistic downstream biological tasks, including binding affinity prediction, functional annotation, mutation effects, cleavage site detection, and PROTAC interaction modeling. It probes how pretraining objectives, structural information integration, and domain-specific inductive biases affect generalization on limited biological data.

## Datasets

- **Protap Benchmark** — total ?; splits: train (-1), test (-1); repo https://github.com/Trust-App-AI-Lab/protap

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve. Measures the probability that a classifier ranks a random positive instance higher than a random negative instance.
- `AUPR` — range: [0, 1]
  - Area Under the Precision-Recall Curve. Summarizes the trade-off between precision and recall across all classification thresholds.
- `Accuracy` — range: [0, 1]
  - Percentage of correctly classified instances out of the total number of instances.
- `MSE` — range: [0, ∞)
  - Mean Squared Error. Average of the squares of the errors between predicted and actual continuous values.
- `Pearson correlation` — range: [-1, 1]
  - Pearson correlation coefficient. Measures the linear correlation between predicted and actual values.
- `Fmax` — range: [0, 1]
  - Maximum F1 score across all classification thresholds. Represents the best harmonic mean of precision and recall achievable by threshold tuning.

## Input / output format

**Input**: Protein sequences and/or 3D atomic coordinates. For interaction tasks (PLI, PROTACs), molecular/ligand structures (drugs, warheads, linkers, E3-ligands) are also provided and encoded via GVP.

**Output**: Task-specific predictions: binary/multi-class labels for cleavage and PROTAC tasks, continuous binding affinity scores for PLI, functional ontology labels for PFA, and mutation effect scores for MTP.

## Scoring recipe

```python
def compute_metrics(preds, golds):
    auc = roc_auc_score(golds, preds)
    aupr = average_precision_score(golds, preds)
    acc = accuracy_score(golds, (preds > 0.5).astype(int))
    mse = mean_squared_error(golds, preds)
    pearson = pearsonr(golds, preds)[0]
    fmax = max_f1_score(golds, preds)
    return {'AUC': auc, 'AUPR': aupr, 'Accuracy': acc, 'MSE': mse, 'Pearson': pearson, 'Fmax': fmax}
```

## Common pitfalls

- Using different train/test splits across models instead of the fixed dataset per task specified in the paper.
- Incorrectly unfreezing pretrained encoders during fine-tuning; the protocol requires frozen encoders with only task-specific heads updated.
- Omitting molecular encoders (GVP) for ligand/warhead/linker components in interaction tasks, which breaks consistency with the benchmark setup.

## Evidence (verbatim from paper)

> Table 4: Performance comparison across model architectures under different training strategies. The first line for each model (e.g., EGNN) denotes a randomly initialized protein encoder trained purely with downstream task supervision. The subsequent lines (e.g., w/MLM) represent pretrained encoders with frozen weights, where only the task-specific head is fine-tuned. C14.005AUC(%)↑ AUPR(%)↑ M10.003AUC(%)↑ AUPR(%)↑ PROTACDBAcc(%)↑ AUPR(%)↑ DavisMSE↓ Pear(%)↑ MFFmax(%)↑ AUPR(%)↑ DMSPear(%)↑ AUPR(%)↑

## Citation

```bibtex
@misc{yan2025protap,
  title={Protap: A Benchmark for Protein Modeling on Realistic Downstream Applications},
  author={Yan et al. (2025)},
  year={2025},
  note={arXiv:2506.02052}
}
```

- arXiv: 2506.02052