Pseudo Ppl Eval

---
name: pseudo-ppl-eval
description: Evaluates the ability of masked language models to predict masked amino acids in short peptide sequences, measuring how well the model captures local sequence dependencies without autoregressive assumptions. It specifically probes the model's capacity to generalize to unseen short peptides that were excluded from the reference database during training. Use when the user wants to benchmark on UniRef100-excluded peptides, or asks about evaluating this task. Reports PseudoPPL.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.10531
  bibtex_key: [REDACTED]
  confidence: high
---

# pseudo-ppl-eval

> PepBenchmark: A Standardized Benchmark for Peptide Machine Learning — Zhang et al. (2026) (arXiv:2604.10531, 2026)

## What this evaluates

Evaluates the ability of masked language models to predict masked amino acids in short peptide sequences, measuring how well the model captures local sequence dependencies without autoregressive assumptions. It specifically probes the model's capacity to generalize to unseen short peptides that were excluded from the reference database during training.

## Datasets

- **UniRef100-excluded peptides** — total 85113; splits: test (85113)

## Metrics

- `PseudoPPL` **(primary)** — range: other
  - Computed as exp(-1/L * sum_{i=1}^L log p(x_i | x_{j!=i})), where L is the sequence length and p(x_i | x_{j!=i}) is the probability of the i-th amino acid given all other positions in the sequence. Lower values indicate better modeling of peptide sequences.

## Input / output format

**Input**: Amino acid sequence string (e.g., 'ACDEFGHIK') representing a short peptide.

**Output**: Scalar perplexity value (float) computed over the sequence.

## Scoring recipe

```python
def compute_pseudo_ppl(sequences, model):
    total_log_prob = 0.0
    total_len = 0
    for seq in sequences:
        L = len(seq)
        # Model predicts each token conditioned on all others (masked LM style)
        log_probs = model.predict_masked(seq)  # shape (L,)
        total_log_prob += sum(log_probs)
        total_len += L
    return math.exp(-total_log_prob / total_len)
```

## Common pitfalls

- Confusing PseudoPPL with standard autoregressive perplexity (PPL), which conditions on previous tokens rather than all other tokens in the sequence.
- Failing to exclude sequences present in the training/reference database (UniRef100) leads to data leakage and artificially low perplexity scores.
- Not stratifying or reporting results by sequence length, as the metric is known to deteriorate significantly for shorter peptides.

## Evidence (verbatim from paper)

> We employed pseudo-perplexity (PseudoPPL) to assess model performance on peptides. Unlike autoregressive models, where perplexity is computed as PPL(x)=exp(-1/L sum log p(x_i|x_<i)), masked language models require the following variant: PseudoPPL(x)=exp(-1/L sum log p(x_i|x_j!=i)). We used 85,113 unique peptide sequences absent from UniRef100 for evaluation.

## Citation

```bibtex
@misc{zhang2026pepbenchmark,
  title={PepBenchmark: A Standardized Benchmark for Peptide Machine Learning},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2604.10531}
}
```

- arXiv: 2604.10531