Capsul Eval

---
name: capsul-eval
description: Evaluates the ability of protein sequence and structure models to predict the subcellular localization compartments of human proteins. It probes multi-label classification performance under severe class imbalance, testing whether models can leverage 3D structural motifs or sequence embeddings to identify fine-grained organelle targeting patterns. Use when the user wants to benchmark on CAPSUL, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.18571
  bibtex_key: hu2026capsul
  confidence: high
---

# capsul-eval

> CAPSUL: A Comprehensive Human Protein Benchmark for Subcellular Localization — Hu et al. (2026) (arXiv:2603.18571, 2026)

## What this evaluates

Evaluates the ability of protein sequence and structure models to predict the subcellular localization compartments of human proteins. It probes multi-label classification performance under severe class imbalance, testing whether models can leverage 3D structural motifs or sequence embeddings to identify fine-grained organelle targeting patterns.

## Datasets

- **CAPSUL** — total ?; splits: test (-1); repo https://github.com/getbetter-hyccc/CAPSUL

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (Precision * Recall) / (Precision + Recall). The paper reports both micro-averaged (aggregates TP/FP/FN across all classes before computing) and macro-averaged (unweighted mean of per-class F1) variants to handle multi-label classification and class imbalance.

## Input / output format

**Input**: Protein amino acid sequence and/or 3D structural graph where nodes represent residues (typically Cα atom positions) and edges represent spatial or sequential adjacency.

**Output**: Multi-label prediction vector y_hat in R^m (m = number of subcellular compartments) with values in (0,1) representing predicted probabilities for each compartment.

## Scoring recipe

```python
def compute_f1(y_true, y_pred, average='micro'):
    tp = sum((y_true == 1) & (y_pred == 1))
    fp = sum((y_true == 0) & (y_pred == 1))
    fn = sum((y_true == 1) & (y_pred == 0))
    precision = tp / (tp + fp + 1e-8)
    recall = tp / (tp + fn + 1e-8)
    return 2 * precision * recall / (precision + recall + 1e-8)
```

## Common pitfalls

- Standard BCE loss optimization neglects minority classes due to severe class imbalance, leading to poor performance on underrepresented compartments like lipid droplets or centrosomes.
- Some baseline tools (e.g., DeepLoc 2.1) do not support prediction for all 18 compartments, resulting in missing values ('/') in the evaluation tables rather than zero scores.
- Multi-label predictions require thresholding continuous probability outputs to binary values before computing precision/recall, a step not explicitly detailed in the paper.

## Evidence (verbatim from paper)

> Given the class imbalance in each location (i.e., the proportion of proteins localized to each subcellular compartment is often small), we consider the widely used evaluation metrics in this task: Precision, Recall, and F1-score (Jiang et al., 2021; Thumuluri et al., 2022). In addition, we utilize micro-averaged and macro-averaged F1-score to evaluate the overall performance across different categories.

## Citation

```bibtex
@misc{hu2026capsul,
  title={CAPSUL: A Comprehensive Human Protein Benchmark for Subcellular Localization},
  author={Hu et al. (2026)},
  year={2026},
  note={arXiv:2603.18571}
}
```

- arXiv: 2603.18571