Spectral Array Extraction And Reconstruction

---
name: spectral-array-extraction-and-reconstruction
description: Use when your input is an mzPeak archive (ZIP of Parquet files) and you need to recover spectrum signal data (m/z values and intensities) for downstream analysis, visualization, or format conversion. Use this skill when working with profile or centroid mode spectra stored in `spectra_data.
license: CC-BY-4.0
metadata:
  edam_operation: http://edamontology.org/operation_3645
  edam_topics:
  - http://edamontology.org/topic_3520
  - http://edamontology.org/topic_0121
  tools:
  - pyarrow
  - arrow (R)
  - Rust mzPeak library
  - NumPy
  - pandas
  techniques:
  - mass-spectrometry
derived_from:
- doi: 10.1021/acs.jproteome.5c00435
  title: mzpeak
evidence_spans:
- complete re-implementation for _reading_ mzPeak files using [`pyarrow`]
claims: []
provenance:
  collection: https://w3id.org/holobiomicslab/asb-skill/collection/metabolomics/v2
  assembled_by: scripts/collect_metabolomics_collection.py
  sources:
  - build: coll_mzpeak_cq
    doi: 10.1021/acs.jproteome.5c00435
    title: mzpeak
  dedup_kept_from: coll_mzpeak_cq
schema_version: 0.2.0
attribution:
  generator: AgenticScienceBuilder
  original_doi: 10.1021/acs.jproteome.5c00435
  all_source_dois:
  - 10.1021/acs.jproteome.5c00435
  zenodo_doi: 10.5281/zenodo.20794027
  curators: []
  promoter: Louis-Félix Nothias
  sponsor: CNRS & Université Côte d'Azur
---

# spectral-array-extraction-and-reconstruction

## Summary

Extract m/z and intensity arrays from mzPeak Parquet-based mass spectrometry archives and reconstruct the original spectral signal using Arrow-backed columnar storage, with support for point or chunked layouts and optional null-marking restoration. This skill is essential for reading scalable, interoperable mzPeak files in Python, R, or Rust environments.

## When to use

Your input is an mzPeak archive (ZIP of Parquet files) and you need to recover spectrum signal data (m/z values and intensities) for downstream analysis, visualization, or format conversion. Use this skill when working with profile or centroid mode spectra stored in `spectra_data.parquet` or `spectra_peaks.parquet`, especially when the data has been subjected to zero-run stripping or null-marking compression to reduce file size.

## When NOT to use

- Input is not an mzPeak file (e.g., mzML, mzXML, NetCDF, or raw vendor formats) — use format-specific readers instead.
- You only need spectrum-level metadata without signal data — use `spectra_metadata.parquet` directly without invoking array reconstruction.
- You require write access to mzPeak files — the Python and R implementations support reading only; use the Rust implementation for write operations.

## Inputs

- mzPeak archive (ZIP file containing Parquet files)
- spectra_metadata.parquet (spectrum index and metadata)
- spectra_data.parquet (m/z and intensity arrays in point or chunked layout)
- spectra_peaks.parquet (optional centroid data)
- mzpeak_index.json (file resolution and schema definitions)

## Outputs

- Decoded spectrum signal arrays (m/z values, intensity values)
- Spectrum index / run information
- NumPy arrays or pandas DataFrames or Arrow Tables with reconstructed spectra
- Metadata for peak layout (profile vs. centroid mode)

## How to apply

Load the mzPeak archive using Arrow/pyarrow libraries to read the Parquet metadata index (`mzpeak_index.json`) and identify the data layout (point vs. chunked). For point layout, read parallel m/z and intensity arrays with a repeated spectrum index column; for chunked layout, unpack the columnar chunks. If null-marking is present, fit or retrieve a learned m/z spacing model (quadratic polynomial: δmz ~ β₀ + β₁·mz + β₂·mz²) and use the local median or model to reconstruct the m/z values for null-marked points. Return decoded spectra as Arrow Tables or convert to NumPy arrays / pandas DataFrames for downstream use. Validate reconstruction against reference test files to confirm peak apex and centroid accuracy remain unaffected by the compression.

## Related tools

- **pyarrow** (Read and deserialize Parquet columnar data structures from mzPeak archives; decode Arrow-backed m/z and intensity arrays into Python objects) — https://arrow.apache.org/docs/python/index.html
- **arrow (R)** (Read and deserialize Parquet data from mzPeak archives; decode spectra into R data structures (Arrow Tables or tibbles)) — https://arrow.apache.org/docs/r/
- **Rust mzPeak library** (Native read and write support for mzPeak files; command-line tools for format conversion and array extraction) — https://github.com/HUPO-PSI/mzPeak
- **NumPy** (Convert reconstructed spectral arrays to NumPy array objects for numerical analysis and visualization)
- **pandas** (Convert reconstructed spectral arrays to DataFrame format for tabular manipulation and export)

## Examples

```
import pyarrow.parquet as pq; table = pq.read_table('spectra_data.parquet', filters=[('spectrum_index', '==', 1)]); mz = table.column('mz').to_numpy(); intensity = table.column('intensity').to_numpy()
```

## Evaluation signals

- Reconstructed m/z and intensity arrays have non-null values for all signal points and match the spectrum index dimensions.
- Peak apex (maximum intensity point) and centroid (intensity-weighted mean m/z) of reconstructed spectra are numerically identical to reference mzPeak test files (within floating-point tolerance).
- For null-marked spectra, the refitted m/z spacing model (quadratic fit residuals) is < 1 ppm RMSE; reconstructed signal shows no visible distortion in mass accuracy plots.
- Data type consistency: m/z values are 64-bit or 32-bit floats (as per schema), intensities are unsigned integers or floats; no type mismatches or NaN propagation except at intentional null locations.
- File reading completes without schema validation errors; Parquet metadata checksums pass and no corrupt columns are detected.

## Limitations

- The Python and R implementations support reading mzPeak files only; write functionality is not yet available (Rust implementation required for format conversion or authoring).
- Null-marking reconstruction accuracy depends on the quality of the learned m/z spacing model; if the spectrum m/z axis deviates substantially from the quadratic model (e.g., due to instrument calibration drift), reconstruction error may exceed 1 ppm.
- Zero-run stripping is only meaningful for profile mode data; centroid mode spectra stored in `spectra_peaks.parquet` bypass this optimization.
- The mzPeak format is currently a HUPO-PSI working draft with no stability guarantee; future schema changes may break compatibility with older implementations.
- Random access to individual spectral arrays in large archives may require scanning the entire Parquet file's row groups; chunked layout offers better access characteristics than point layout but at the cost of larger file size.

## Evidence

- [intro] complete re-implementation for _reading_ mzPeak files using [`pyarrow`]: "complete re-implementation for _reading_ mzPeak files using [`pyarrow`], and the PyData stack"
- [readme] mzPeak is a archive of multiple [Parquet] files stored in an _uncompressed_ [ZIP] archive: "mzPeak is a archive of multiple [Parquet] files, stored directly in an _uncompressed_ [ZIP] archive"
- [readme] Point layout stores data as-is in parallel arrays alongside a repeated index column: "the point layout stores the data as-is in parallel arrays alongside a repeated index column"
- [readme] Null marking replaces flanking zero intensity points with null m/z and intensity values: "replace the flanking zero intensity points with `null` m/z and intensity values"
- [readme] use either the local median δ mz or the learned model for that spectrum to compute m/z spacing: "use either the local median δ mz or the learned model for that spectrum to compute the m/z spacing for singleton points"
- [readme] The Python codebase does not support writing at this time: "The Python codebase does not support writing at this time although this is subject to change in the future"
- [other] Decode spectral data arrays (m/z values, intensities) from Arrow-backed storage: "Decode spectral data arrays (m/z values, intensities) from Arrow-backed storage"