Metabolite Retention Time Comparison

---
name: metabolite-retention-time-comparison
description: Use when after executing retention-order prediction on a test set of
  metabolites, you have both predicted retention orderings and experimentally measured
  ground-truth retention orderings for the same set of compounds, and you need to
  quantify prediction accuracy to validate the model's performance.
license: CC-BY-4.0
metadata:
  edam_operation: http://edamontology.org/operation_3927
  edam_topics:
  - http://edamontology.org/topic_3172
  - http://edamontology.org/topic_0091
  tools:
  - evaluation_scenarios_main.py
  - RankSVM
  - SVR (Support Vector Regression)
  - scipy, numpy, pandas
  - scikit-learn
  license_tier: open
  provenance_tier: literature
derived_from:
- doi: 10.1093/bioinformatics/bty590
  title: LC retention order prediction
evidence_spans: []
claims: []
provenance:
  collection: https://w3id.org/holobiomicslab/asb-skill/collection/metabolomics/v2
  assembled_by: scripts/collect_metabolomics_collection.py
  sources:
  - build: coll_lc_retention_order_prediction_cq
    doi: 10.1093/bioinformatics/bty590
    title: LC retention order prediction
  dedup_kept_from: coll_lc_retention_order_prediction_cq
schema_version: 0.2.0
attribution:
  generator: AgenticScienceBuilder
  original_doi: 10.1093/bioinformatics/bty590
  all_source_dois:
  - 10.1093/bioinformatics/bty590
  zenodo_doi: 10.5281/zenodo.20794027
  curators: []
  promoter: Louis-Félix Nothias
  sponsor: CNRS & Université Côte d'Azur
---

# metabolite-retention-time-comparison

## Summary

Compare predicted liquid-chromatography retention orders against experimentally determined ground-truth orderings using ranking accuracy and correlation metrics. This skill validates retention-order prediction models on metabolite datasets by quantifying how well predicted orderings match observed LC retention sequences.

## When to use

After executing retention-order prediction on a test set of metabolites, you have both predicted retention orderings and experimentally measured ground-truth retention orderings for the same set of compounds, and you need to quantify prediction accuracy to validate the model's performance across different LC systems or training scenarios.

## When NOT to use

- Ground-truth retention orderings are not available or are incomplete (missing pairwise comparisons for evaluation)
- Predicted orderings have not yet been generated by a model (comparison requires both predictions and observations)
- Input metabolite dataset is from a single LC system and no cross-system evaluation is intended (use only for single-system validation, not transfer-learning assessment)

## Inputs

- predicted retention orders (JSON or structured array per target system and scenario)
- ground-truth experimental retention orderings (metabolite compound pairs with known retention rank)
- configuration file specifying experiment scenario, molecular features, and kernel choice

## Outputs

- results summary table with predicted vs. ground-truth retention orders and performance scores
- ranking accuracy and Spearman correlation coefficient per target system
- structured results directory organized by scenario, feature representation, and kernel type

## How to apply

Load the predicted retention orders and ground-truth orderings from the experiment output (typically stored in structured result directories by scenario). Apply ranking-based evaluation metrics—specifically ranking accuracy (fraction of correctly predicted pairwise orderings) and Spearman rank correlation (monotonic association between predicted and observed ranks)—to each target system individually. Generate a results summary table listing predicted vs. ground-truth retention orders and corresponding performance scores for each system and scenario. Use these metrics to assess generalization: single-system predictions (baseline_single scenario) serve as a lower-bound reference, while cross-system predictions (all_on_one scenario, with and without leave-target-system-out flag) demonstrate transfer capability.

## Related tools

- **evaluation_scenarios_main.py** (Orchestrates experiment execution and produces predicted retention orders for specified scenario, estimator, and target system; outputs are stored hierarchically by feature representation and kernel, then compared against ground truth) — github.com/aalto-ics-kepaco/retention_order_prediction
- **RankSVM** (Order predictor estimator used to generate pairwise retention orderings; predictions are fed into comparison workflow) — github.com/aalto-ics-kepaco/retention_order_prediction
- **SVR (Support Vector Regression)** (Alternative order predictor estimator option; generates retention scores that can be converted to orderings for comparison) — github.com/aalto-ics-kepaco/retention_order_prediction
- **scipy, numpy, pandas** (Python packages for numerical computation, ranking calculations, and tabular results organization)
- **scikit-learn** (Provides ranking metrics and correlation functions (Spearman) used in evaluation)

## Examples

```
python src/evaluation_scenarios_main.py ranksvm baseline_single 10 -1 results/raw/PredRet/v2/config.json 2 False
```

## Evaluation signals

- Ranking accuracy is non-negative and ≤ 1.0 (fraction of correctly ordered pairs); baseline_single scenarios typically show 0.50–0.95 range depending on system and feature representation
- Spearman rank correlation coefficient ranges from −1 to +1; for well-performing models on same-system prediction, expect values > 0.7; cross-system transfer typically yields lower correlations (0.3–0.7)
- Results summary table is complete (no missing entries) for all target systems in the specified scenario; rows correspond to unique metabolite pairs, columns to predicted rank, ground-truth rank, and performance metrics
- Output directory structure matches expected hierarchy: `results/PredRet/v2/final/{estimator}_{params}/{feature_representation}/{kernel}/{scenario}`
- Performance comparison shows expected ranking: baseline_single ≥ all_on_one (LTSO=False) ≥ all_on_one (LTSO=True), reflecting increasing domain shift; deviation suggests data leakage or misconfigured split

## Limitations

- Evaluation is restricted to pairwise relative orderings; absolute retention-time prediction (e.g., minutes) is not assessed
- Performance depends heavily on choice of molecular features (MACCS fingerprints binary vs. counting) and kernel function (minmax, tanimoto); comparison results are not portable across feature/kernel combinations without rerunning the full pipeline
- Cross-system transfer (all_on_one scenario) assumes that training systems are sufficiently similar; highly dissimilar LC systems or unusual mobile phases may show poor generalization despite high baseline_single accuracy
- Ranking metrics assume a fixed set of metabolites per LC system; results are not comparable across datasets with different metabolite sets or different numbers of ground-truth orderings

## Evidence

- [other] Capture predicted retention orders and compare against ground-truth orderings using the repository's evaluation metrics (ranking accuracy, Spearman correlation, or similar).: "Capture predicted retention orders and compare against ground-truth orderings using the repository's evaluation metrics (ranking accuracy, Spearman correlation, or similar)."
- [readme] Single system used as training and target. Different percentage of data used for trainging.: "Single system used as training and target. Different percentage of data used for trainging."
- [readme] All systems used for training. Single system used as target. Target system in training (LTSO): True & False: "All systems used for training. Single system used as target. Target system in training (LTSO): True & False"
- [readme] The results will be stored into: results/PredRet/v2 └── final └── ranksvm_slacktype=on_pairs └── allow_overlap=True_d_lower=0_d_upper=16_ireverse=False_type=order_graph └── difference └── maccsCount_f2dcf0b3 └── minmax └── baseline_single: "The results will be stored into: results/PredRet/v2 └── final └── ranksvm_slacktype=on_pairs └── allow_overlap=True_d_lower=0_d_upper=16_ireverse=False_type=order_graph └── difference └──"
- [other] Generate a results summary table or report listing predicted vs. ground-truth retention orders and performance scores.: "Generate a results summary table or report listing predicted vs. ground-truth retention orders and performance scores."