Psp Protein Structure Eval

---
name: psp-protein-structure-eval
description: Evaluates protein structure prediction models on their ability to infer accurate 3D atomic coordinates from amino acid sequences. It specifically probes topological backbone similarity and side-chain accuracy when trained on large-scale distilled protein datasets. Use when the user wants to benchmark on CASP14 test set, PSP dataset, or asks about evaluating this task. Reports TM-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2206.12240
  bibtex_key: liu2022psp
  confidence: high
---

# psp-protein-structure-eval

> PSP: Million-level Protein Sequence Dataset for Protein Structure Prediction — Liu et al. (2022) (arXiv:2206.12240, 2022)

## What this evaluates

Evaluates protein structure prediction models on their ability to infer accurate 3D atomic coordinates from amino acid sequences. It specifically probes topological backbone similarity and side-chain accuracy when trained on large-scale distilled protein datasets.

## Datasets

- **CASP14 test set** — total 87; splits: test (87)
- **PSP dataset** — total 1315000; splits: train (-1)

## Metrics

- `TM-score` **(primary)** — range: [0, 1]
  - Assesses topological similarity between the predicted and true protein structures. A score of 1 indicates an exact match, while scores below 0.17 indicate irrelevant structures.
- `lDDT` — range: [0, 1]
  - local Distance Difference Test measuring side-chain atom position accuracy. A score of 0 indicates totally different structures, and 1 indicates exactly the same.

## Input / output format

**Input**: Amino acid sequences accompanied by multiple sequence alignments (MSA) and template information.

**Output**: Predicted 3D protein structure represented as atomic coordinates (backbone and side-chain atoms).

## Scoring recipe

```python
def evaluate_structure(pred_coords, true_coords):
    # Calculate topological similarity for backbone
    tmscore = calculate_tmscore(pred_coords, true_coords)
    # Calculate side-chain atom position accuracy
    lddt = calculate_lddt(pred_coords, true_coords)
    return tmscore, lddt

# Protocol: Average TM-score over 87 CASP14 sequences
avg_tmscore = mean([evaluate_structure(p, t)[0] for p, t in casp14_pairs])
```

## Common pitfalls

- Data leakage must be prevented by strictly excluding CASP14 sequences with structure release dates after May 13, 2020.
- Dropout causes training instability and rapid TM-score drops during fine-tuning; it must be disabled.
- The violation loss clash term can cause training crashes due to high variance on distillation data; it requires clipping (e.g., to 5.0).

## Evidence (verbatim from paper)

> We took the TM-score on 87 CASP14 sequences as the monitor of accuracy. The TM-score (template modeling score) is used to assess the topological similarity between protein target structure and predicted structure. A value 1 means exactly the same, and <0.17 means totally irrelevant.

## Citation

```bibtex
@misc{liu2022psp,
  title={PSP: Million-level Protein Sequence Dataset for Protein Structure Prediction},
  author={Liu et al. (2022)},
  year={2022},
  note={arXiv:2206.12240}
}
```

- arXiv: 2206.12240