Peta Protein Eval

---
name: peta-protein-eval
description: Evaluates protein language models across 33 downstream tasks (fitness, localization, PPI, solubility, and structure prediction) to measure how sub-word tokenization and vocabulary size affect representation quality and task performance. Use when the user wants to benchmark on PETA Benchmark Suite, or asks about evaluating this task. Reports Spearman correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.17415
  bibtex_key: tan2023peta
  confidence: high
---

# peta-protein-eval

> PETA: Evaluating the Impact of Protein Transfer Learning with Sub-word Tokenization on Downstream Applications — Yang Tan et al. (2023) (arXiv:2310.17415, 2023)

## What this evaluates

Evaluates protein language models across 33 downstream tasks (fitness, localization, PPI, solubility, and structure prediction) to measure how sub-word tokenization and vocabulary size affect representation quality and task performance.

## Datasets

- **PETA Benchmark Suite** — total ?; splits: test (-1); repo https://github.com/ginnm/ProteinPretraining

## Metrics

- `Spearman correlation` **(primary)** — range: [-1, 1]
  - Measures the monotonic relationship between predicted and observed values. Computed as the rank correlation coefficient between model predictions and ground truth labels.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly classified instances out of the total number of instances.
- `MSE` — range: [0, inf)
  - Mean Squared Error between predicted and actual continuous values.

## Input / output format

**Input**: Protein sequences (amino acid strings) tokenized via per-amino-acid, BPE, or Unigram strategies, constrained to a maximum sequence length of 1024 tokens.

**Output**: Continuous regression values or discrete class labels, depending on the downstream task.

## Scoring recipe

```python
def compute_metric(predictions, gold, task_type):
    if task_type == 'correlation':
        return spearmanr(predictions, gold).correlation
    elif task_type == 'accuracy':
        preds_class = np.argmax(predictions, axis=1)
        return np.mean(preds_class == gold)
    elif task_type == 'mse':
        return np.mean((predictions - gold) ** 2)
```

## Common pitfalls

- Averaging results across multiple data splits, tokenization methods, classification heads, and random seeds can mask per-setting performance variations.
- Vocabulary sizes exceeding 800 tokens consistently degrade performance on structure prediction tasks, contrary to typical NLP scaling trends.
- Several datasets (e.g., GB1, AAV, Thermo, Solmut) contain multiple data partitioning methods that must be explicitly averaged before reporting final scores.

## Evidence (verbatim from paper)

> Fitness Prediction. Table 5 showcases results for five distinct tasks under the umbrella of Fitness Prediction and the evaluation metrics is Spearman correlation.

## Citation

```bibtex
@misc{tan2023peta,
  title={PETA: Evaluating the Impact of Protein Transfer Learning with Sub-word Tokenization on Downstream Applications},
  author={Yang Tan et al. (2023)},
  year={2023},
  note={arXiv:2310.17415}
}
```

- arXiv: 2310.17415