Spectral Scoring And Significance Assessment

---
name: spectral-scoring-and-significance-assessment
description: 'Use when you have executed multiple database search pipelines (Dereplicator, VarQuest, and/or Dereplicator+) on centroided LC-MS/MS spectra (in MGF, mzXML, mzML, or mzData format) against a chemical structure database of known natural products, and you need to: (1) identify which pipeline(s).'
license: CC-BY-4.0
metadata:
  edam_operation: http://edamontology.org/operation_3632
  edam_topics:
  - http://edamontology.org/topic_0121
  - http://edamontology.org/topic_3520
  - http://edamontology.org/topic_0154
  tools:
  - NPDtools 2.5.0
  - MetaMiner
  - Dereplicator
  - ProteoWizard
  - Python
  - VarQuest
  - Dereplicator+
derived_from:
- doi: 10.1038/s41467-018-06082-8
  title: dereplicator
evidence_spans:
- The latest version is available in the Natural Product Discovery toolkit (NPDtools) at https://github.com/ablab/npdtools
- MetaMiner is a metabologenomic pipeline which integrates metabolomic (tandem mass spectra) and genomic data to identify novel RiPPs
- MetaMiner is a metabologenomic pipeline which integrates metabolomic (tandem mass spectra) and genomic data to identify novel Ribosmally synthesized and Post-translationally modified Peptides (RiPPs)
- matches tandem mass spectra against the constructed post-translationally modified RiPPs structure database using Dereplicator
- MetaMiner natively supports MGF, mzXML, mzData and uses msconvert utility from the ProteoWizard package to convert spectra in other formats to MGF
- uses msconvert utility from the ProteoWizard package to convert spectra in other formats to MGF
claims: []
provenance:
  collection: https://w3id.org/holobiomicslab/asb-skill/collection/metabolomics/v1
  assembled_by: scripts/collect_metabolomics_collection.py
  sources:
  - build: coll_dereplicator
    doi: 10.1038/s41467-018-06082-8
    title: dereplicator
  dedup_kept_from: coll_dereplicator
schema_version: 0.2.0
---

# spectral-scoring-and-significance-assessment

## Summary

Evaluate and compare database search results from natural product mass spectrometry pipelines by computing match scores, p-values, and false discovery rates (FDR) to distinguish true identifications from noise. This skill is essential for validating which database search pipeline (Dereplicator, VarQuest, Dereplicator+) provides reliable identifications and assessing the statistical confidence of spectrum-to-structure matches.

## When to use

You have executed multiple database search pipelines (Dereplicator, VarQuest, and/or Dereplicator+) on centroided LC-MS/MS spectra (in MGF, mzXML, mzML, or mzData format) against a chemical structure database of known natural products, and you need to: (1) identify which pipeline(s) successfully detected matches in your test spectra, (2) rank matches by statistical significance rather than raw score alone, (3) cross-compare hit patterns and sensitivity across search modes (e.g., standard vs. blind/unbiased modification search), or (4) control false discovery rate when reporting identifications.

## When NOT to use

- Spectra have not yet been searched against a database (this skill assumes search results already exist; it is a post-processing step, not a search tool itself).
- You are using only a single database search pipeline and have no cross-pipeline comparison goal — in that case, focus on single-tool result interpretation rather than comparative analysis.
- Raw, uncentroided spectra or spectra in proprietary vendor formats that have not been converted to MGF/mzXML/mzML — preprocessing and format conversion must precede this skill.

## Inputs

- significant_matches.tsv output from Dereplicator pipeline
- significant_matches.tsv output from VarQuest pipeline
- significant_matches.tsv output from Dereplicator+ pipeline
- Scan identifiers from test spectra (MSV000080102 .mzML files converted to MGF)
- Match scores (numeric)
- P-values (numeric, one per match)
- False discovery rates (numeric, one per match)

## Outputs

- Cross-compared hit set table (scan ID, pipeline, score, p-value, FDR, common vs. unique)
- Tool-specific detections list (matches found by only one pipeline)
- Common identifications list (matches found by ≥2 pipelines)
- Mode-dependent sensitivity analysis (standard vs. blind/modification-tolerant mode comparison)
- Ranked match list filtered by p-value and FDR thresholds

## How to apply

Parse the significant_matches.tsv output files generated by each database search tool, extracting scan identifiers, match scores, p-values, and false discovery rates for each spectrum–structure pair. Tabulate these metrics side-by-side to enable tool-to-tool comparison. Filter matches using both score and statistical thresholds (p-value and FDR cutoffs should be justified based on your false positive tolerance). Identify tool-specific detections (matches unique to one pipeline), common identifications (hits shared across pipelines, which increase confidence), and mode-dependent sensitivity differences (e.g., standard Dereplicator vs. blind-mode MetaMiner). Cross-reference scan identifiers to ensure you are comparing the same spectra across tools. Use p-values and FDR to rank and report matches in order of statistical confidence rather than raw score alone, as this guards against spurious high-scoring alignments.

## Related tools

- **Dereplicator** (database search pipeline for peptidic natural products; generates match scores and results for tabulation and cross-comparison) — https://github.com/ablab/npdtools
- **VarQuest** (modification-tolerant database search pipeline; produces alternative hits and sensitivity profile for comparison with Dereplicator and Dereplicator+) — https://github.com/ablab/npdtools
- **Dereplicator+** (extended database search pipeline for both peptidic and non-peptidic metabolites; provides an alternative hit set for cross-validation) — https://github.com/ablab/npdtools
- **MetaMiner** (metabologenomic pipeline that includes database search via Dereplicator; supports standard and blind (unbiased modification) search modes for mode-dependent sensitivity comparison) — https://github.com/ablab/npdtools
- **Python** (scripting language used to parse significant_matches.tsv, tabulate results, and cross-compare metrics across tools)

## Examples

```
python parse_matches.py --dereplicator significant_matches.tsv --varquest significant_matches.tsv --dereplicate_plus significant_matches.tsv --output cross_comparison.tsv
```

## Evaluation signals

- Cross-comparison table is complete: every scan ID from the test spectra appears in the output with entries for all pipelines that detected a match (or an explicit 'no match' entry).
- P-values and FDR values are present and finite (not NaN or infinite) for all reported matches; FDR is monotonically non-decreasing when matches are sorted by p-value within each tool.
- Tool-specific and common identification lists are mutually exclusive and exhaustive (every match belongs to exactly one category: unique to one tool, or shared by ≥2 tools).
- Matches ranked by p-value show expected rank order (lower p-value = more confident match); matches with identical p-values can be sub-ranked by score if reported.
- Mode-dependent comparison (standard vs. blind search) shows that blind/modification-tolerant modes detect additional matches not found in standard mode, validating the added sensitivity of flexible search.

## Limitations

- P-values and FDR are meaningful only if the database search pipeline computed them; if a pipeline outputs only raw scores without statistical tests, comparison requires separate statistical modeling (e.g., permutation or null-model fitting).
- Cross-pipeline comparison assumes all tools searched the same spectrum file set and the same (or comparable) chemical structure database; mismatches in database version or content will confound interpretation of tool-specific vs. true differences.
- Scan identifier matching across tool outputs requires careful attention to format and naming conventions; if pipelines report scan IDs differently (e.g., with or without file prefixes), a mapping step is necessary before tabulation.
- False discovery rate control is valid only if applied uniformly across all comparisons; if tools use different FDR estimation methods or null models, direct FDR comparison may be misleading.
- Mode-dependent sensitivity assessment (e.g., standard vs. blind MetaMiner) is only informative if both modes are run on identical input spectra and with matched search parameters; parameter differences can dominate mode effects.

## Evidence

- [methods] Execute Dereplicator pipeline on the same test spectra and RiPP structure database, output match results: "Execute Dereplicator pipeline on the same test spectra and RiPP structure database, output match results."
- [methods] Execute VarQuest pipeline on the same inputs with standard parameters: "Execute VarQuest pipeline on the same inputs with standard parameters."
- [methods] Parse and tabulate significant_matches.tsv outputs from all three tools, recording scan identifiers, match scores, p-values, and false discovery rates: "Parse and tabulate significant_matches.tsv outputs from all three tools, recording scan identifiers, match scores, p-values, and false discovery rates."
- [methods] Cross-compare hit sets to identify tool-specific detections, common identifications, and mode-dependent sensitivity differences: "Cross-compare hit sets to identify tool-specific detections, common identifications, and mode-dependent sensitivity differences."
- [readme] Dereplicator — a tool for identification of peptidic natural products (PNPs) through database search of mass spectra: "Dereplicator — a tool for identification of peptidic natural products (PNPs) through database search of mass spectra"
- [readme] VarQuest — a tool for modification-tolerant identification of novel variants of PNPs: "VarQuest — a tool for modification-tolerant identification of novel variants of PNPs"
- [readme] Dereplicator+ — a tool for identification of metabolites (both peptidic and non-peptidic) through database search of mass spectra: "Dereplicator+ — a tool for identification of metabolites (both peptidic and non-peptidic) through database search of mass spectra"