Global Epistasis Fitness Eval

---
name: global-epistasis-fitness-eval
description: This evaluation probes a model's ability to recover and predict sparse latent fitness functions from observed sequence-fitness data distorted by global epistasis. It compares data efficiency and predictive accuracy under complete versus incomplete (subsampled) data regimes, highlighting the robustness of contrastive losses over mean-squared error. Use when the user wants to benchmark on NK model synthetic data, FLIP benchmark, or asks about evaluating this task. Reports Spearman correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.03136
  bibtex_key: brookes2023contrastive
  confidence: high
---

# global-epistasis-fitness-eval

> Contrastive losses as generalized models of global epistasis — Brookes et al. (2023) (arXiv:2305.03136, 2023)

## What this evaluates

This evaluation probes a model's ability to recover and predict sparse latent fitness functions from observed sequence-fitness data distorted by global epistasis. It compares data efficiency and predictive accuracy under complete versus incomplete (subsampled) data regimes, highlighting the robustness of contrastive losses over mean-squared error.

## Datasets

- **NK model synthetic data** — total ?; splits: complete (-1), incomplete (-1)
- **FLIP benchmark** — total ?; splits: test (-1)

## Metrics

- `Spearman correlation` **(primary)** — range: [-1, 1]
  - Rank-based correlation coefficient measuring the monotonic relationship between predicted and true fitness values across all sequences in the space.
- `Pearson correlation` — range: [-1, 1]
  - Linear correlation coefficient measuring the strength of the linear relationship between predicted and true fitness values.

## Input / output format

**Input**: Sequence x_i and its observed fitness value y_i (transformed by a monotonic nonlinearity).

**Output**: Predicted fitness value f_hat(x_i).

## Scoring recipe

```python
def spearman_correlation(predictions, gold):
    pred_rank = rankdata(predictions)
    gold_rank = rankdata(gold)
    n = len(predictions)
    d_sq = sum((p - g) ** 2 for p, g in zip(pred_rank, gold_rank))
    return 1 - (6 * d_sq) / (n * (n**2 - 1))
```

## Common pitfalls

- Assuming the model must predict exact fitness magnitudes; recovery is evaluated up to an affine transformation due to monotonic nonlinearity invariance.
- Using MSE loss for incomplete data with strong global epistasis, which degrades performance because the epistatic representation becomes dense, violating sparsity assumptions.
- Ignoring the validation split used for early stopping, which can bias convergence metrics if not reported consistently.

## Evidence (verbatim from paper)

> After training, we measured the extent to which the models estimated the fitness function by calculating Spearman correlation between the model predictions and true fitness values on all sequences in the sequence space. Spearman correlation is commonly used to benchmark fitness prediction methods [[15], [27]].

## Citation

```bibtex
@misc{brookes2023contrastive,
  title={Contrastive losses as generalized models of global epistasis},
  author={Brookes et al. (2023)},
  year={2023},
  note={arXiv:2305.03136}
}
```

- arXiv: 2305.03136