Psbench Eval

---
name: psbench-eval
description: Evaluates the ability of Estimation of Model Accuracy (EMA) methods to predict the structural quality of protein complex models. It probes global and interface-level accuracy estimation using correlation, ranking, and classification metrics against reference structural scores. Use when the user wants to benchmark on CASP16_inhouse_TOP5_dataset, CASP16_community_dataset, or asks about evaluating this task. Reports Pearson’s correlation (CorrP).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.22674
  bibtex_key: neupane2025psbench
  confidence: high
---

# psbench-eval

> PSBench: a large-scale benchmark for estimating the accuracy of protein complex structural models — Neupane et al. (2025) (arXiv:2505.22674, 2025)

## What this evaluates

Evaluates the ability of Estimation of Model Accuracy (EMA) methods to predict the structural quality of protein complex models. It probes global and interface-level accuracy estimation using correlation, ranking, and classification metrics against reference structural scores.

## Datasets

- **CASP16_inhouse_TOP5_dataset** — total 32; splits: test (32); repo https://github.com/BioinfoMachineLearning/PSBench
- **CASP16_community_dataset** — total 37; splits: test (37); repo https://github.com/BioinfoMachineLearning/PSBench

## Metrics

- `Pearson’s correlation (CorrP)` **(primary)** — range: [-1, 1]
  - Measures the linear correlation between predicted quality scores and reference scores (e.g., TM-score or DockQ_wave). Ranges from -1 to 1, where 1 indicates perfect positive linear relationship.
- `Spearman’s correlation (CorrS)` — range: [-1, 1]
  - Measures the monotonic relationship between predicted and reference scores based on rank ordering. Ranges from -1 to 1.
- `Ranking loss (Loss)` — range: [0, 1]
  - Calculates the fraction of incorrectly ordered pairs of models based on predicted versus reference scores. Lower values indicate better ranking consistency.
- `AUROC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve, evaluating the ability to classify models as high or low quality based on a threshold applied to reference scores.

## Input / output format

**Input**: Protein complex structural models (typically top-ranked predictions from structure predictors) along with target sequence information.

**Output**: Predicted global quality score (e.g., TM-score) and/or interface quality score (e.g., DockQ_wave) for each model/target.

## Scoring recipe

```python
def evaluate(y_pred, y_true):
    pearson = pearsonr(y_pred, y_true).statistic
    spearman = spearmanr(y_pred, y_true).statistic
    n = len(y_pred)
    discordant = sum(1 for i in range(n) for j in range(i+1, n) if (y_pred[i]-y_pred[j])*(y_true[i]-y_true[j]) < 0)
    loss = discordant / (n*(n-1)/2)
    y_true_bin = (y_true > threshold).astype(int)
    auroc = roc_auc_score(y_true_bin, y_pred)
    return pearson, spearman, loss, auroc
```

## Common pitfalls

- Excluding targets with identical sequences but different conformations (e.g., T1249, T1294) to avoid data leakage.
- Excluding very large targets (e.g., H1217, H1227) when computational constraints prevent model generation.
- Evaluating only on the top 5 models per target rather than the full predictor output.

## Evidence (verbatim from paper)

> In terms of a global quality score - TM-score, GATE-AFM achieved the highest Spearman’s correlation (0.283), the lowest ranking loss (0.102), the best AUROC (0.658), and second highest Pearson’s correlation (0.372), indicating superior ranking consistency and classification reliability.

## Citation

```bibtex
@misc{neupane2025psbench,
  title={PSBench: a large-scale benchmark for estimating the accuracy of protein complex structural models},
  author={Neupane et al. (2025)},
  year={2025},
  note={arXiv:2505.22674}
}
```

- arXiv: 2505.22674