Ms Modification Site Evaluation

---
name: ms-modification-site-evaluation
description: Use when after ModiFinder has generated modification site probability
  scores for an unknown compound by comparing its MS/MS spectrum to a known analog,
  and you have access to the true structure of the unknown compound (oracle mode)
  or a reference modification site annotation.
license: CC-BY-4.0
metadata:
  edam_operation: http://edamontology.org/operation_3802
  edam_topics:
  - http://edamontology.org/topic_3520
  - http://edamontology.org/topic_0121
  tools:
  - ModiFinder
  - BasicEvaluationEngine
  - Python
  - RDKit
  - matplotlib
  - Pillow
  techniques:
  - LC-MS
  license_tier: restricted
  provenance_tier: literature
derived_from:
- doi: 10.1021/jasms.4c00061
  title: ModiFinder
evidence_spans:
- mf = ModiFinder(known_compound, modified_compound, mz_tolerance=0.01, ppm_tolerance=40)
- mf = ModiFinder(known_compound, modified_compound, helpers=helpers_array, **args)
- eval_engine = BasicEvaluationEngine(default_method="is_max")
- eval_engine = BasicEvaluationEngine(default_method="average_distance")
- ModiFinder requires Python 3.9 or above.
- ModiFinder requires Python 3.9 or above
claims: []
provenance:
  collection: https://w3id.org/holobiomicslab/asb-skill/collection/metabolomics/v2
  assembled_by: scripts/collect_metabolomics_collection.py
  sources:
  - build: coll_modifinder_cq
    doi: 10.1021/jasms.4c00061
    title: ModiFinder
  dedup_kept_from: coll_modifinder_cq
schema_version: 0.2.0
attribution:
  generator: AgenticScienceBuilder
  original_doi: 10.1021/jasms.4c00061
  all_source_dois:
  - 10.1021/jasms.4c00061
  zenodo_doi: 10.5281/zenodo.20794027
  curators: []
  promoter: Louis-Félix Nothias
  sponsor: CNRS & Université Côte d'Azur
---

# ms-modification-site-evaluation

> **License: restricted** — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution. <!-- asb-license-banner -->
## Summary

Quantitatively evaluate the accuracy of predicted modification sites on unknown compounds by comparing ModiFinder-generated probability distributions against known structural modifications using BasicEvaluationEngine scoring metrics (is_max and average_distance). This skill validates whether the predicted site probabilities correctly localize the actual modification.

## When to use

After ModiFinder has generated modification site probability scores for an unknown compound by comparing its MS/MS spectrum to a known analog, and you have access to the true structure of the unknown compound (oracle mode) or a reference modification site annotation. Use this skill to quantify whether the prediction succeeded—i.e., whether the highest-probability site(s) align with the true modification location.

## When NOT to use

- The modified compound structure is unknown and oracle-mode refinement has not been applied; evaluation requires ground truth.
- ModiFinder probabilities have not yet been generated; evaluation requires a complete probability distribution over modification sites.
- You are only interested in visual inspection of the alignment and do not need quantitative validation.

## Inputs

- analog_structure (RDKit molecule object representing the known compound)
- target_structure (RDKit molecule object representing the modified/unknown compound)
- modification_probabilities (numpy array or dict mapping atoms to probability scores, generated by ModiFinder.generate_probabilities())
- evaluation_method (string: 'is_max' or 'average_distance')

## Outputs

- evaluation_score (float: 1.0 or 0.0 for is_max; 0.0–1.0 for average_distance)
- evaluation_results (dictionary or JSON object containing score, method, and optionally the input probability vector)

## How to apply

Instantiate BasicEvaluationEngine with the desired scoring method (either 'is_max' to test whether the true site received the maximum probability, or 'average_distance' to measure the average atom-level distance from predicted high-probability atoms to the true modification site). Call evaluate_single() with the known/analog structure, modified/unknown structure, and the ModiFinder-generated probabilities. The is_max score (binary: 0 or 1) succeeds if the true site is correctly ranked highest; average_distance (continuous: 0–1, lower is better) provides sensitivity to near-misses by penalizing predictions that are structurally close but not exact. Compare returned scores against project baselines (e.g., is_max=1.0 and average_distance=0.514 for ModiFinder under default Modified Cosine + MAGMa) to assess whether your configuration matches expected performance.

## Related tools

- **ModiFinder** (Generates modification site probability distributions via generate_probabilities() that are passed to BasicEvaluationEngine for scoring) — https://github.com/Wang-Bioinformatics-Lab/ModiFinder_base
- **BasicEvaluationEngine** (Computes is_max and average_distance evaluation metrics by comparing predicted probabilities against known structures) — https://github.com/Wang-Bioinformatics-Lab/ModiFinder_base
- **RDKit** (Provides molecule representation and structure comparison primitives used by BasicEvaluationEngine) — http://www.rdkit.org/
- **Python** (Required runtime for ModiFinder and BasicEvaluationEngine (≥3.9))

## Examples

```
from modifinder import BasicEvaluationEngine; engine = BasicEvaluationEngine(default_method='average_distance'); score = engine.evaluate_single(analog_structure, target_structure, probabilities); print(f'average_distance={score}')
```

## Evaluation signals

- is_max score equals 1.0 if and only if the true modification site receives the maximum probability in the predicted distribution.
- average_distance score is in the range [0, 1] with lower values indicating predictions closer to the true modification site; baseline of 0.514 for ModiFinder default configuration provides a reference benchmark.
- Consistency check: is_max=1.0 should generally yield a low average_distance (since the true site is ranked first); if both metrics are available, inspect their correlation.
- JSON output includes the input probability vector to enable post-hoc tracing and visualization of which atoms received high confidence.
- Evaluation score reproducibility: re-running evaluate_single() with identical inputs should yield identical scores (deterministic).

## Limitations

- Evaluation requires knowledge of the true modification site; if the ground truth structure is incorrect or ambiguous, the score may misleadingly penalize correct predictions.
- average_distance depends on the choice of atom-alignment algorithm used by BasicEvaluationEngine; different structure normalization or graph matching strategies could yield different scores.
- is_max is binary and does not reward near-correct predictions; average_distance is more forgiving but can mask failure modes where multiple high-probability atoms are far from the truth.
- The skill does not validate the quality of the input spectra, tolerances (mz_tolerance=0.01, ppm_tolerance=40, ratio_to_base_peak=0.01), or ModiFinder configuration; poor upstream choices will lead to poor scores that do not reflect a failure of evaluation itself.

## Evidence

- [other] Instantiate BasicEvaluationEngine with default_method='average_distance': "Instantiate BasicEvaluationEngine with default_method='average_distance'"
- [other] Call evaluate_single() with analog and target structures and probabilities: "Call evaluate_single(analog_structure, target_structure, probabilities) to compute average_distance score; call again with evaluation_method='is_max' to compute is_max score"
- [other] ModiFinder baseline scores: is_max=1.0 and average_distance=0.514: "ModiFinder achieves BasicEvaluationEngine scores of is_max = 1.0 and average_distance = 0.514 under the default Modified Cosine + MAGMa condition"
- [other] Baseline comparison to validate configuration: "Compare the returned score against the reference value of 0.514 and document the result"
- [other] Output includes probability vector for tracing: "Write both scores and intermediate probability vector to evaluation_results.json"