Protein Understanding Eval

---
name: protein-understanding-eval
description: Evaluates protein language models on sequence understanding across multiple downstream tasks (structure, function, interactions, developability) and 3D structure prediction from single amino acid sequences. Use when the user wants to benchmark on CAMEO, CASP15, OOD Protein Sequences (UniProt), or asks about evaluating this task. Reports TM-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.06199
  bibtex_key: [REDACTED]
  confidence: high
---

# protein-understanding-eval

> xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein — Chen et al. (2024) (arXiv:2401.06199, 2024)

## What this evaluates

Evaluates protein language models on sequence understanding across multiple downstream tasks (structure, function, interactions, developability) and 3D structure prediction from single amino acid sequences.

## Datasets

- **CAMEO** — total 194; splits: test (194)
- **CASP15** — total 56; splits: test (56)
- **OOD Protein Sequences (UniProt)** — total 20000; splits: test (10000), test (10000)

## Metrics

- `TM-score` **(primary)** — range: [0, 1]
  - A structural similarity metric that measures the global fold similarity between predicted and experimental protein structures. Values range from 0 to 1, with higher scores indicating better structural alignment. Computed using standard superposition algorithms like TM-align.
- `Perplexity` — range: other
  - Quantifies the uncertainty of a language model in predicting a sequence. Calculated as the exponential of the average negative log-likelihood of the tokens. Lower values indicate better predictive accuracy.

## Input / output format

**Input**: Single amino acid sequences (strings). For structure prediction, single and pair representations are extracted and fed into the structure module.

**Output**: For perplexity: token probability distributions. For structure prediction: 3D atomic coordinates. For downstream tasks: task-specific labels or regression values.

## Scoring recipe

```python
def compute_tm_score(pred_coords, true_coords):
    # Standard TM-align superposition and score calculation
    return tm_align_score(pred_coords, true_coords)

def compute_perplexity(logits, targets):
    log_probs = torch.nn.functional.log_softmax(logits, dim=-1)
    loss = -log_probs[range(len(targets)), targets].mean()
    return torch.exp(loss).item()
```

## Common pitfalls

- Comparing PLM-based structure predictors (xT-Fold) directly with MSA-based models (AlphaFold2) without accounting for the fundamental difference in input requirements (single sequence vs. multiple sequence alignment).
- Assuming full fine-tuning is feasible for 100B-parameter models; the protocol explicitly uses LoRA to reduce trainable parameters to <1%.
- Confusing the two OOD datasets filtered at 0.5 and 0.9 sequence identity thresholds, which yield different perplexity baselines.

## Evidence (verbatim from paper)

> xT-Fold achieved a TM-score of 0.86 on the CAMEO dataset and 0.70 on CASP15. The scores for ESMFold were 0.85 and 0.65, respectively, while OmegaFold scored 0.80 and 0.60 on these datasets.

## Citation

```bibtex
@misc{chen2024xtrimopglm,
  title={xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2401.06199}
}
```

- arXiv: 2401.06199