Benchmarking And Comparative Performance Evaluation

---
name: benchmarking-and-comparative-performance-evaluation
description: Use when you have trained a new machine learning model for chemical formula or adduct assignment from MS/MS spectra and need to assess whether it offers genuine performance gains over established baselines. Use it specifically when you have access to ground-truth annotations (e.
license: CC-BY-4.0
metadata:
  edam_operation: http://edamontology.org/operation_3436
  edam_topics:
  - http://edamontology.org/topic_3372
  - http://edamontology.org/topic_0121
  - http://edamontology.org/topic_3053
  tools:
  - MIST
  - MIST-CF
  - SIRIUS
  - SCARF
  - FFN
  - Xformer
  techniques:
  - LC-MS
derived_from:
- doi: 10.1021/acs.jcim.3c01082
  title: mistcf
evidence_spans:
- an extension of MIST for annotating MS1 precursor masses from MS/MS data
- MIST-CF ranks chemical formula and adduct assignments for an unknown mass spectrum using an end-to-end energy based modeling approach
- Utilizing an internal chemical subformula assignment protocol (rather than SIRIUS fragmentation trees)
- Utilizing sinusoidal formula embeddings as developed in our previous work SCARF
claims: []
provenance:
  collection: https://w3id.org/holobiomicslab/asb-skill/collection/metabolomics/v2
  assembled_by: scripts/collect_metabolomics_collection.py
  sources:
  - build: coll_mistcf
    doi: 10.1021/acs.jcim.3c01082
    title: mistcf
  dedup_kept_from: coll_mistcf
schema_version: 0.2.0
attribution:
  generator: AgenticScienceBuilder
  original_doi: 10.1021/acs.jcim.3c01082
  all_source_dois:
  - 10.1021/acs.jcim.3c01082
  zenodo_doi: 10.5281/zenodo.20794027
  curators: []
  promoter: Louis-Félix Nothias
  sponsor: CNRS & Université Côte d'Azur
---

# Benchmarking and Comparative Performance Evaluation

## Summary

Systematically compare a novel method (MIST-CF) against established baseline approaches (SIRIUS, FFN, Xformer) on standardized benchmark datasets using ranking accuracy metrics (top-1, top-3, top-k). This skill enables quantitative assessment of whether a new chemical formula inference model outperforms existing tools on retrospective and prospective MS/MS data.

## When to use

Apply this skill when you have trained a new machine learning model for chemical formula or adduct assignment from MS/MS spectra and need to assess whether it offers genuine performance gains over established baselines. Use it specifically when you have access to ground-truth annotations (e.g., NPLIB1, NIST20, CASMI challenge datasets) and must justify publication or deployment claims with comparative accuracy metrics.

## When NOT to use

- Input is a single spectrum or small set without ground-truth annotations — use case-by-case evaluation instead.
- Baseline tool (e.g., SIRIUS) is not available or cannot be run on your hardware/OS — benchmarking requires all competing methods executable in the same environment.
- You are comparing only internal model variants (e.g., ablation study) without a published external baseline — use internal ablation evaluation rather than full benchmarking.

## Inputs

- MS/MS spectra in MGF format with precursor m/z values
- Ground-truth chemical formula and adduct type annotations
- Candidate formula lists (generated by SIRIUS decomp or chemical enumeration)
- Trained neural network weights for MIST-CF, FFN, Xformer baselines
- Preprocessed spectral data (normalized intensities, noise-filtered peaks)

## Outputs

- Per-spectrum ranking predictions (ranked list of candidate formulas with adduct types)
- Top-1, top-3, top-k accuracy metrics for formula assignment
- Top-1, top-3, top-k accuracy metrics for adduct assignment
- Performance summary table (model × accuracy metric)
- Ranking accuracy plots (e.g., cumulative accuracy vs. rank position)
- Comparative analysis report (MIST-CF vs. SIRIUS, FFN, Xformer)

## How to apply

Run inference predictions from all competing models (MIST-CF, SIRIUS decomp, FFN, Xformer) on the same test dataset, ensuring identical MS/MS preprocessing (intensity normalization, noise filtering, precursor m/z extraction). For each spectrum, compute ranked predictions for chemical formula and adduct type; evaluate using ranking metrics (top-1 accuracy: first ranked candidate matches ground truth; top-3, top-k: ground truth appears in top k predictions). Tabulate accuracy results side-by-side and generate ranking accuracy plots showing cumulative performance across the test set. On proprietary datasets (e.g., NIST20), train on disjoint splits; on public benchmarks (NPLIB1, CASMI22), use published train-test boundaries. Document hyperparameter tuning (via hyperopt or grid search) for all models on validation data to ensure fair comparison.

## Related tools

- **MIST-CF** (Novel energy-based neural network model for ranking chemical formulas and adduct types from MS/MS spectra; main method under evaluation.) — https://github.com/samgoldman97/mist-cf
- **SIRIUS** (Baseline tool for formula enumeration via dynamic programming algorithm (SIRIUS decomp); generates candidate formula lists and provides ranking predictions for comparison.) — https://bio.informatik.uni-jena.de/software/sirius/
- **FFN** (Feed-forward neural network baseline model for formula ranking; included in comparative benchmarking experiments.)
- **Xformer** (Transformer baseline model for formula ranking; included in comparative benchmarking experiments.)
- **SCARF** (Related work on sinusoidal formula embeddings; used as feature representation in MIST-CF architecture.) — https://arxiv.org/abs/2303.06470

## Examples

```
. run_scripts/benchmarking/train_mist_cf.sh && . run_scripts/benchmarking/train_ffn.sh && . run_scripts/benchmarking/train_xformer.sh && python run_scripts/benchmarking/eval_models.py
```

## Evaluation signals

- Top-k accuracy values are monotonically non-decreasing as k increases (top-1 ≤ top-3 ≤ top-5, etc.).
- MIST-CF top-k accuracy meets or exceeds SIRIUS baseline on retrospective (NPLIB1, NIST20) and prospective (CASMI22) test sets, demonstrating improvement over fragmentation-tree-based method.
- Performance metrics are computed on a consistent test split with no data leakage between training and evaluation sets; hyperparameter tuning used only validation data.
- Adduct assignment accuracy is reported separately and is non-trivial (>baseline random selection of adduct types).
- Comparison tables show that all models were tuned to convergence (no model has obviously suboptimal hyperparameters relative to published results or validation curves).

## Limitations

- MIST-CF trained on public NPLIB1 dataset may be less performant than NIST20-trained model (particularly for Orbitrap or higher-resolution MS/MS data), limiting generalizability claims.
- Benchmarking focuses on positive-mode MS/MS only; adduct types are still limited beyond [M+H]+, so evaluation does not cover negative-mode or diverse adduct scenarios.
- SIRIUS baseline still required for initial formula enumeration (candidate generation); MIST-CF learns ranking but does not replace formula discovery, limiting end-to-end de novo capability.
- Prospective evaluation (CASMI22) is a single challenge dataset; retrospective benchmarking on NPLIB1 may overfit to that data distribution.

## Evidence

- [other] How does MIST-CF's end-to-end energy-based formula transformer perform on ranking chemical formula and adduct assignments compared to SIRIUS-based approaches on benchmark MS/MS data?: "research_question: How does MIST-CF's end-to-end energy-based formula transformer perform on ranking chemical formula and adduct assignments compared to SIRIUS-based approaches on benchmark MS/MS"
- [other] Evaluate ranking performance by computing top-1, top-3, and top-k accuracy metrics for formula and adduct predictions.: "4. Evaluate ranking performance by computing top-1, top-3, and top-k accuracy metrics for formula and adduct predictions."
- [other] Compare MIST-CF top-k accuracy results against SIRIUS baseline predictions on the same dataset.: "5. Compare MIST-CF top-k accuracy results against SIRIUS baseline predictions on the same dataset."
- [intro] MIST-CF ranks chemical formula and adduct assignments for an unknown mass spectrum using an end-to-end energy based modeling approach, without referencing any spectrum databases: "MIST-CF ranks chemical formula and adduct assignments for an unknown mass spectrum using an end-to-end energy based modeling approach, without referencing any spectrum databases"
- [intro] Instead of computing fragmentation trees, MIST-CF adopts a formula transformer neural network architecture and learns in a data dependent fashion: "Instead of computing fragmentation trees, MIST-CF adopts a formula transformer neural network architecture and learns in a data dependent fashion"
- [readme] Retrospective benchmarking: Benchmark MIST-CF performance with baseline models.: "Retrospective benchmarking
Benchmark MIST-CF performance with baseline models."
- [readme] Prospective analysis on CASMI-2022: Compare MIST-CF and SIRIUS on [CASMI-2022](http://www.casmi-contest.org/2022/index.shtml).: "Prospective analysis: CASMI 2022
Compare MIST-CF and SIRIUS on [CASMI-2022](http://www.casmi-contest.org/2022/index.shtml)."
- [readme] This model may be less performant than the model trained on the commercial NIST20 Library (particularly for Orbitrap or higher resolution data).: "This model may be less performant than the model trained on the commercial NIST20 Library (particularly for Orbitrap or higher resolution data)."
- [intro] Considering multiple adduct types beyond [M+H]+ (still only positive mode): "Considering multiple adduct types beyond [M+H]+ (still only positive mode)"