Protein Generative Ai Eval

---
name: protein-generative-ai-eval
description: This evaluation framework probes the physical validity, functional success, and generalization capability of generative protein models. It emphasizes leakage-aware dataset splits, structural and docking quality metrics, and experimental validation pipelines to ensure designs are biologically plausible and functionally active. Use when the user wants to benchmark on PLINDER, PoseBusters, PDFBench, PoseX, VenusX, FragBench, GeomMotif, or asks about evaluating this task. Reports RMSD.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.26378
  bibtex_key: wanasekara2026generative
  confidence: high
---

# protein-generative-ai-eval

> Generative Modeling in Protein Design: Neural Representations, Conditional Generation, and Evaluation Standards — Wanasekara et al. (2026) (arXiv:2603.26378, 2026)

## What this evaluates

This evaluation framework probes the physical validity, functional success, and generalization capability of generative protein models. It emphasizes leakage-aware dataset splits, structural and docking quality metrics, and experimental validation pipelines to ensure designs are biologically plausible and functionally active.

## Datasets

- **PLINDER** — total ?; splits: (unstated)
- **PoseBusters** — total ?; splits: (unstated)
- **PDFBench** — total ?; splits: (unstated)
- **PoseX** — total ?; splits: (unstated)
- **VenusX** — total ?; splits: (unstated)
- **FragBench** — total ?; splits: (unstated)
- **GeomMotif** — total ?; splits: (unstated)

## Metrics

- `RMSD` **(primary)** — range: other
  - Root mean square deviation between predicted and reference atomic coordinates. Used for structure prediction and docking pose validity.
- `scRMSD` — range: other
  - Folds the designed sequence with a structure predictor and compares the predicted structure to the intended backbone to assess de novo design self-consistency.
- `sequence recovery` — range: percent
  - Fraction of positions matching the native sequence on a fixed backbone, standard for inverse folding.
- `pairwise sequence dissimilarity` — range: other
  - Avg. Dissimilarity = (2/(k(k-1))) * sum_{i<j} d(s_i, s_j), where d is Hamming or Levenshtein distance over k generated sequences.
- `Shannon entropy` — range: other
  - H = -sum_{a in A} p_a log_2 p_a, measuring amino acid variability at each position to distinguish conserved cores from flexible surfaces.
- `DockQ` — range: [0, 1]
  - Interface-aware metric reflecting protein-protein interaction quality, preferred over global backbone RMSD for complexes.
- `PoseBusters validity` — range: percent
  - Checks physical plausibility constraints including steric clash rates, chirality, and valence consistency for docking poses.

## Input / output format

**Input**: Conditioned protein sequences, backbones, or complexes; reference structures/sequences for ground-truth comparison.

**Output**: Generated protein structures (backbones), sequences, or complexes; confidence scores (e.g., pLDDT, pTM); validity flags and metric scores.

## Scoring recipe

```python
def evaluate(predictions, gold, k=None):
    # Structural similarity
    rmsd = np.sqrt(np.mean((predictions.coords - gold.coords)**2))
    # De novo self-consistency
    sc_rmsd = fold_and_compare_rmsd(predictions.seq, gold.backbone)
    # Sequence recovery
    recovery = np.mean(predictions.seq == gold.seq)
    # Pairwise dissimilarity
    dissim = 0
    for i in range(k):
        for j in range(i+1, k):
            dissim += hamming_distance(seq[i], seq[j])
    dissim /= (k * (k - 1) / 2)
    # Physical validity
    validity = check_steric_clashes(predictions) and check_valence(predictions)
    return rmsd, sc_rmsd, recovery, dissim, validity
```

## Common pitfalls

- Random splits cause severe information leakage due to near-duplicates at sequence, structure, or ligand-scaffold levels.
- Relying solely on geometric similarity (RMSD) misrepresents functional success or physical plausibility.
- Computational metrics are only proxies; credible claims require wet-lab experimental validation.

## Evidence (verbatim from paper)

> For inverse folding, sequence recovery (fraction of positions matching the native sequence on a fixed backbone) is the standard benchmark. Pairwise sequence dissimilarity quantifies diversity among generated sequences: Avg. Dissimilarity = (2/(k(k-1))) sum_{i<j} d(s_i, s_j), where d is Hamming or Levenshtein distance over a set of k generated sequences.

## Citation

```bibtex
@misc{wanasekara2026generative,
  title={Generative Modeling in Protein Design: Neural Representations, Conditional Generation, and Evaluation Standards},
  author={Wanasekara et al. (2026)},
  year={2026},
  note={arXiv:2603.26378}
}
```

- arXiv: 2603.26378