Glycanml Eval

---
name: glycanml-eval
description: Evaluates machine learning models on glycan analysis tasks, including taxonomic classification, immunogenicity prediction, glycosylation type prediction, and protein-glycan binding affinity estimation. It probes the ability of sequence-based and graph-based encoders to capture multi-relational glycan structures and benefit from multi-task learning. Use when the user wants to benchmark on GlycanML, or asks about evaluating this task. Reports Macro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.16206
  bibtex_key: xu2024glycanml
  confidence: high
---

# glycanml-eval

> GlycanML: A Multi-Task and Multi-Structure Benchmark for Glycan Machine Learning — Xu et al. (2024) (arXiv:2405.16206, 2024)

## What this evaluates

Evaluates machine learning models on glycan analysis tasks, including taxonomic classification, immunogenicity prediction, glycosylation type prediction, and protein-glycan binding affinity estimation. It probes the ability of sequence-based and graph-based encoders to capture multi-relational glycan structures and benefit from multi-task learning.

## Datasets

- **GlycanML** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/GlycanML/GlycanML

## Metrics

- `Macro-F1` **(primary)** — range: [0, 1]
  - Macro-averaged F1 score computed across all classes for each taxonomy sub-task and the glycosylation type task.
- `AUPRC` — range: [0, 1]
  - Area Under the Precision-Recall Curve for binary immunogenicity prediction.
- `Spearman’s ρ` — range: [-1, 1]
  - Spearman rank correlation coefficient between predicted and true protein-glycan binding affinity values.
- `Weighted Mean Rank` — range: other
  - Comprehensive metric averaging the rank of each model across all 11 tasks, where each of the 8 taxonomy tasks weighs 1/8 and the other 3 tasks weigh 1.

## Input / output format

**Input**: Glycan structures represented as sequences or graphs (homogeneous/heterogeneous). For interaction prediction, protein sequences are encoded via ESM-1b and concatenated with glycan representations.

**Output**: Predicted class labels for taxonomy, immunogenicity, and glycosylation tasks; predicted continuous binding affinity score for interaction prediction.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred, y_prob=None, y_true_cont=None, y_pred_cont=None):
    macro_f1 = f1_score(y_true, y_pred, average='macro')
    auprc = average_precision_score(y_true, y_prob)
    spearman_rho = spearmanr(y_true_cont, y_pred_cont).correlation
    ranks = [rankdata(-scores[task]) for task in tasks]
    wmr = (sum(ranks[:8]) / 8) + sum(ranks[8:])
    return macro_f1, auprc, spearman_rho, wmr
```

## Common pitfalls

- Using homogeneous GNNs ignores glycosidic bond types, leading to suboptimal performance compared to heterogeneous/multi-relational GNNs.
- Applying small-molecule encoders (e.g., Graphormer) directly to glycans fails due to scale mismatch, as glycans are macromolecules with hundreds of atoms.
- Assuming multi-task learning (MTL) universally improves performance; the paper shows most MTL methods degrade results due to difficulty in balancing correlated taxonomy tasks.

## Evidence (verbatim from paper)

> We measure the comprehensive performance of a model with its weighted mean rank over all tasks, where each taxonomy prediction task weighs 1/8 and each of the other three tasks weighs 1, so as to balance between different types of tasks.

## Citation

```bibtex
@misc{xu2024glycanml,
  title={GlycanML: A Multi-Task and Multi-Structure Benchmark for Glycan Machine Learning},
  author={Xu et al. (2024)},
  year={2024},
  note={arXiv:2405.16206}
}
```

- arXiv: 2405.16206