Embedding Vector Similarity Ranking

---
name: embedding-vector-similarity-ranking
description: Use when you have a query mass spectrum and need to retrieve the most chemically similar reference spectra from a large experimental or predicted library (thousands to millions of entries).
license: CC-BY-4.0
metadata:
  edam_operation: http://edamontology.org/operation_3814
  edam_topics:
  - http://edamontology.org/topic_0091
  - http://edamontology.org/topic_3520
  tools:
  - hnswlib
  - Python
  - gensim
  - Python 3.7
  techniques:
  - mass-spectrometry
derived_from:
- doi: 10.1038/s41467-023-39279-7
  title: FastEI
evidence_spans:
- conda install -c conda-forge hnswlib
- Anaconda for Python 3.7
- conda install -c conda-forge gensim
claims: []
provenance:
  collection: https://w3id.org/holobiomicslab/asb-skill/collection/metabolomics/v2
  assembled_by: scripts/collect_metabolomics_collection.py
  sources:
  - build: coll_fastei_cq
    doi: 10.1038/s41467-023-39279-7
    title: FastEI
  dedup_kept_from: coll_fastei_cq
schema_version: 0.2.0
attribution:
  generator: AgenticScienceBuilder
  original_doi: 10.1038/s41467-023-39279-7
  all_source_dois:
  - 10.1038/s41467-023-39279-7
  zenodo_doi: 10.5281/zenodo.20794027
  curators: []
  promoter: Louis-Félix Nothias
  sponsor: CNRS & Université Côte d'Azur
---

# embedding-vector-similarity-ranking

## Summary

Rank spectrum candidates by computing cosine similarity between a query spectrum's Word2vec embedding and a pre-indexed set of reference spectrum embeddings using HNSW approximate nearest-neighbor search. This enables ultra-fast molecular identification from million-scale in-silico libraries while maintaining accuracy.

## When to use

You have a query mass spectrum and need to retrieve the most chemically similar reference spectra from a large experimental or predicted library (thousands to millions of entries). The query spectrum has been converted to a Word2vec embedding vector, and reference spectra embeddings are already indexed in an HNSW structure. Use this skill to obtain a ranked list of candidate matches ordered by similarity, enabling rapid molecular identification without exhaustive pairwise comparison.

## When NOT to use

- Query spectrum has not yet been embedded using the same Word2vec model as the reference library; embeddings must be generated by the identical pre-trained model to ensure comparability.
- HNSW index has not been built from the reference spectra; you must first construct the index using hnswlib before querying.
- Reference library size is very small (< 1000 spectra); exhaustive nearest-neighbor search may be simpler and equally fast without index overhead.

## Inputs

- Query spectrum embedding vector (Word2vec-encoded, floating-point)
- Pre-built HNSW index binary file (e.g., references_index.bin)
- Reference spectrum embeddings database (pre-indexed)
- Number of candidates k to retrieve

## Outputs

- Ranked list of nearest-neighbor spectrum identifiers
- Similarity scores (cosine distance) for each retrieved spectrum
- Top-k candidate spectra ordered by chemical similarity

## How to apply

Load the pre-built HNSW index from a binary file (e.g., 'references_index.bin') and the query spectrum embedding vector (Word2vec-encoded). Call the HNSW approximate nearest-neighbor search method to retrieve the k most similar candidate spectra, ranked by cosine distance. The HNSW index uses default navigable small world graph parameters (M and ef settings) to balance speed and accuracy during construction and query time. Return the ranked list of spectrum identifiers with their corresponding similarity scores. Cosine distance is the default metric; inspect the returned rankings to confirm high-confidence matches (typically cosine similarity > 0.7 for correct molecular identifications in mass spectrometry).

## Related tools

- **hnswlib** (Implements hierarchical navigable small world graph data structure for approximate nearest-neighbor search on embedding vectors with configurable M and ef parameters) — https://github.com/nmslib/hnswlib
- **gensim** (Provides Word2vec model training and vector encoding for spectrum embedding generation)
- **Python 3.7** (Runtime environment for loading and querying HNSW indexes and spectrum embeddings)

## Examples

```
import hnswlib; index = hnswlib.Index(space='cosine', dim=100); index.load_index('references_index.bin'); labels, distances = index.knn_query(query_embedding, k=10)
```

## Evaluation signals

- Retrieved spectrum identifiers are unique and belong to the indexed reference library (no out-of-bounds or duplicate returns).
- Similarity scores are in valid cosine distance range (0.0 to 1.0 or equivalent normalized metric); scores decrease monotonically across the ranked list.
- Top-ranked spectrum matches correspond to known correct molecular identifications when validated against experimental or curated reference data (e.g., NIST 2017 or MassBank libraries).
- Query runtime is sub-second for k ≤ 100 on million-scale libraries, confirming HNSW acceleration over brute-force search.
- Recall of true matches within top-k is > 95% compared to exhaustive nearest-neighbor search, validating approximate search approximation quality.

## Limitations

- HNSW is an approximate nearest-neighbor method; rank order may differ from exhaustive nearest-neighbor search, particularly for lower-ranked candidates.
- Query accuracy depends critically on the quality of the Word2vec embedding model and consistency between query and reference embeddings; mismatched or poorly trained models degrade ranking quality.
- HNSW index is immutable after construction; adding new reference spectra requires rebuilding the entire index, which may be computationally expensive for very large libraries.
- Platform support is currently limited; FastEI software is tested only on Windows 7 and Windows 10 64-bit systems as of the latest release.

## Evidence

- [other] For a given query spectrum embedding, perform approximate nearest-neighbor search on the HNSW index to retrieve the k most similar candidate spectra (ranked by cosine distance).: "For a given query spectrum embedding, perform approximate nearest-neighbor search on the HNSW index to retrieve the k most similar candidate spectra (ranked by cosine distance)."
- [other] FastEI implements HNSW-based indexing to enable approximate nearest-neighbor search on Word2vec-embedded spectrum vectors, allowing rapid retrieval from million-scale in-silico libraries while maintaining accuracy.: "FastEI implements HNSW-based indexing to enable approximate nearest-neighbor search on Word2vec-embedded spectrum vectors, allowing rapid retrieval from million-scale in-silico libraries while"
- [other] Initialize an HNSW index using hnswlib with default parameters (M and ef settings for navigable small world graph construction).: "Initialize an HNSW index using hnswlib with default parameters (M and ef settings for navigable small world graph construction)."
- [readme] FastEI is an ultra-fast and accurate spectrum matching method, proposed to improve accuracy by Word2vec-based spectrum embedding and boost the speed using hierarchical navigable small world graph (HNSW): "FastEI is an ultra-fast and accurate spectrum matching method, proposed to improve accuracy by Word2vec-based spectrum embedding and boost the speed using hierarchical navigable small world graph"
- [readme] a million-molecule scale *in-silico* library has been builded and an ultra-fast and accurate search method has been developed (FastEI): "a million-molecule scale *in-silico* library has been builded and an ultra-fast and accurate search method has been developed (FastEI)"