177 Reference D1c26264

# Proteomics and Metabolomics File Formats Reference

This reference covers file formats specific to proteomics, metabolomics, lipidomics, and related omics workflows.

## Mass Spectrometry-Based Proteomics

### .mzML - Mass Spectrometry Markup Language
**Description:** Standard XML format for MS data
**Typical Data:** MS1 and MS2 spectra, retention times, intensities
**Use Cases:** Proteomics, metabolomics pipelines
**Python Libraries:**
- `pymzml`: `pymzml.run.Reader('file.mzML')`
- `pyteomics.mzml`: `pyteomics.mzml.read('file.mzML')`
- `pyopenms`: OpenMS Python bindings
**EDA Approach:**
- Scan count and MS level distribution
- Total ion chromatogram (TIC) analysis
- Base peak chromatogram (BPC)
- m/z coverage and resolution
- Retention time range
- Precursor selection patterns
- Data completeness
- Quality control metrics (lock mass, standards)

### .mzXML - Legacy MS XML Format
**Description:** Older XML-based MS format
**Typical Data:** Mass spectra with metadata
**Use Cases:** Legacy proteomics data
**Python Libraries:**
- `pyteomics.mzxml`
- `pymzml`: Can read mzXML
**EDA Approach:**
- Similar to mzML
- Format version compatibility
- Conversion quality validation
- Metadata preservation check

### .mzIdentML - Peptide Identification Format
**Description:** PSI standard for peptide identifications
**Typical Data:** Peptide-spectrum matches, proteins, scores
**Use Cases:** Search engine results, proteomics workflows
**Python Libraries:**
- `pyteomics.mzid`
- `pyopenms`: MzIdentML support
**EDA Approach:**
- PSM count and score distribution
- FDR calculation and filtering
- Modification analysis
- Missed cleavage statistics
- Protein inference results
- Search parameters validation
- Decoy hit analysis
- Rank-1 vs lower ranks

### .pepXML - Trans-Proteomic Pipeline Peptide XML
**Description:** TPP format for peptide identifications
**Typical Data:** Search results with statistical validation
**Use Cases:** Proteomics database search output
**Python Libraries:**
- `pyteomics.pepxml`
**EDA Approach:**
- Search engine comparison
- Score distributions (XCorr, expect value, etc.)
- Charge state analysis
- Modification frequencies
- PeptideProphet probabilities
- Protein coverage
- Spectral counting

### .protXML - Protein Inference Results
**Description:** TPP protein-level identifications
**Typical Data:** Protein groups, probabilities, peptides
**Use Cases:** Protein-level analysis
**Python Libraries:**
- `pyteomics.protxml`
**EDA Approach:**
- Protein group statistics
- Parsimonious protein sets
- ProteinProphet probabilities
- Coverage and peptide count per protein
- Unique vs shared peptides
- Protein molecular weight distribution
- GO term enrichment preparation

### .pride.xml - PRIDE XML Format
**Description:** Proteomics Identifications Database format
**Typical Data:** Complete proteomics experiment data
**Use Cases:** Public data deposition (legacy)
**Python Libraries:**
- `pyteomics.pride`
- Custom XML parsers
**EDA Approach:**
- Experiment metadata extraction
- Identification completeness
- Cross-linking to spectra
- Protocol information
- Instrument details

### .tsv / .csv (Proteomics)
**Description:** Tab or comma-separated proteomics results
**Typical Data:** Peptide or protein quantification tables
**Use Cases:** MaxQuant, Proteome Discoverer, Skyline output
**Python Libraries:**
- `pandas`: `pd.read_csv()` or `pd.read_table()`
**EDA Approach:**
- Identification counts
- Quantitative value distributions
- Missing value patterns
- Intensity-based analysis
- Label-free quantification assessment
- Isobaric tag ratio analysis
- Coefficient of variation
- Batch effects

### .msf - Thermo MSF Database
**Description:** Proteome Discoverer results database
**Typical Data:** SQLite database with search results
**Use Cases:** Thermo Proteome Discoverer workflows
**Python Libraries:**
- `sqlite3`: Database access
- Custom MSF parsers
**EDA Approach:**
- Database schema exploration
- Peptide and protein tables
- Score thresholds
- Quantification data
- Processing node information
- Confidence levels

### .pdResult - Proteome Discoverer Result
**Description:** Proteome Discoverer study results
**Typical Data:** Comprehensive search and quantification
**Use Cases:** PD study exports
**Python Libraries:**
- Vendor tools for conversion
- Export to TSV for Python analysis
**EDA Approach:**
- Study design validation
- Result filtering criteria
- Quantitative comparison groups
- Imputation strategies

### .pep.xml - Peptide Summary
**Description:** Compact peptide identification format
**Typical Data:** Peptide sequences, modifications, scores
**Use Cases:** Downstream analysis input
**Python Libraries:**
- `pyteomics`: XML parsing
**EDA Approach:**
- Unique peptide counting
- PTM site localization
- Retention time predictability
- Charge state preferences

## Quantitative Proteomics

### .sky - Skyline Document
**Description:** Skyline targeted proteomics document
**Typical Data:** Transition lists, chromatograms, results
**Use Cases:** Targeted proteomics (SRM/MRM/PRM)
**Python Libraries:**
- `skyline`: Python API (limited)
- Export to CSV for analysis
**EDA Approach:**
- Transition selection validation
- Chromatographic peak quality
- Interference detection
- Retention time consistency
- Calibration curve assessment
- Replicate correlation
- LOD/LOQ determination

### .sky.zip - Zipped Skyline Document
**Description:** Skyline document with external files
**Typical Data:** Complete Skyline analysis
**Use Cases:** Sharing Skyline projects
**Python Libraries:**
- `zipfile`: Extract for processing
**EDA Approach:**
- Document structure
- External file references
- Result export and analysis

### .wiff - SCIEX WIFF Format
**Description:** SCIEX instrument data with quantitation
**Typical Data:** LC-MS/MS with MRM transitions
**Use Cases:** SCIEX QTRAP, TripleTOF data
**Python Libraries:**
- Vendor tools (limited Python access)
- Conversion to mzML
**EDA Approach:**
- MRM transition performance
- Dwell time optimization
- Cycle time analysis
- Peak integration quality

### .raw (Thermo)
**Description:** Thermo raw instrument file
**Typical Data:** Full MS data from Orbitrap, Q Exactive
**Use Cases:** Label-free and TMT quantification
**Python Libraries:**
- `pymsfilereader`: Thermo RawFileReader
- `ThermoRawFileParser`: Cross-platform CLI
**EDA Approach:**
- MS1 and MS2 acquisition rates
- AGC target and fill times
- Resolution settings
- Isolation window validation
- SPS ion selection (TMT)
- Contamination assessment

### .d (Agilent)
**Description:** Agilent data directory
**Typical Data:** LC-MS and GC-MS data
**Use Cases:** Agilent instrument workflows
**Python Libraries:**
- Community parsers
- Export to mzML
**EDA Approach:**
- Method consistency
- Calibration status
- Sequence run information
- Retention time stability

## Metabolomics and Lipidomics

### .mzML (Metabolomics)
**Description:** Standard MS format for metabolomics
**Typical Data:** Full scan MS, targeted MS/MS
**Use Cases:** Untargeted and targeted metabolomics
**Python Libraries:**
- Same as proteomics mzML tools
**EDA Approach:**
- Feature detection quality
- Mass accuracy assessment
- Retention time alignment
- Blank subtraction
- QC sample consistency
- Isotope pattern validation
- Adduct formation analysis
- In-source fragmentation check

### .cdf / .netCDF - ANDI-MS
**Description:** Analytical Data Interchange for MS
**Typical Data:** GC-MS, LC-MS chromatography data
**Use Cases:** Metabolomics, GC-MS workflows
**Python Libraries:**
- `netCDF4`: Low-level access
- `pyopenms`: CDF support
- `xcms` via R integration
**EDA Approach:**
- TIC and extracted ion chromatograms
- Peak detection across samples
- Retention index calculation
- Mass spectral matching
- Library search preparation

### .msp - Mass Spectral Format (NIST)
**Description:** NIST spectral library format
**Typical Data:** Reference mass spectra
**Use Cases:** Metabolite identification, library matching
**Python Libraries:**
- `matchms`: Spectral matching
- Custom MSP parsers
**EDA Approach:**
- Library coverage
- Metadata completeness (InChI, SMILES)
- Spectral quality metrics
- Collision energy standardization
- Precursor type annotation

### .mgf (Metabolomics)
**Description:** Mascot Generic Format for MS/MS
**Typical Data:** MS/MS spectra for metabolite ID
**Use Cases:** Spectral library searching
**Python Libraries:**
- `matchms`: Metabolomics spectral analysis
- `pyteomics.mgf`
**EDA Approach:**
- Spectrum quality filtering
- Precursor isolation purity
- Fragment m/z accuracy
- Neutral loss patterns
- MS/MS completeness

### .nmrML - NMR Markup Language
**Description:** Standard XML format for NMR metabolomics
**Typical Data:** 1D/2D NMR spectra with metadata
**Use Cases:** NMR-based metabolomics
**Python Libraries:**
- `nmrml2isa`: Format conversion
- Custom XML parsers
**EDA Approach:**
- Spectral quality metrics
- Binning consistency
- Reference compound validation
- pH and temperature effects
- Metabolite identification confidence

### .json (Metabolomics)
**Description:** JSON format for metabolomics results
**Typical Data:** Feature tables, annotations, metadata
**Use Cases:** GNPS, MetaboAnalyst, web tools
**Python Libraries:**
- `json`: Standard library
- `pandas`: JSON normalization
**EDA Approach:**
- Feature annotation coverage
- GNPS clustering results
- Molecular networking statistics
- Adduct and in-source fragment linkage
- Putative identification confidence

### .txt (Metabolomics Tables)
**Description:** Tab-delimited feature tables
**Typical Data:** m/z, RT, intensities across samples
**Use Cases:** MZmine, XCMS, MS-DIAL output
**Python Libraries:**
- `pandas`: Text file reading
**EDA Approach:**
- Feature count and quality
- Missing value imputation
- Data normalization assessment
- Batch correction validation
- PCA and clustering for QC
- Fold change calculations
- Statistical test preparation

### .featureXML - OpenMS Feature Format
**Description:** OpenMS detected features
**Typical Data:** LC-MS features with quality scores
**Use Cases:** OpenMS workflows
**Python Libraries:**
- `pyopenms`: FeatureXML support
**EDA Approach:**
- Feature detection parameters
- Quality metrics per feature
- Isotope pattern fitting
- Charge state assignment
- FWHM and asymmetry

### .consensusXML - OpenMS Consensus Features
**Description:** Linked features across samples
**Typical Data:** Aligned features with group info
**Use Cases:** Multi-sample LC-MS analysis
**Python Libraries:**
- `pyopenms`: ConsensusXML reading
**EDA Approach:**
- Feature correspondence quality
- Retention time alignment
- Missing value patterns
- Intensity normalization needs
- Batch-wise feature agreement

### .idXML - OpenMS Identification Format
**Description:** Peptide/metabolite identifications
**Typical Data:** MS/MS identifications with scores
**Use Cases:** OpenMS ID workflows
**Python Libraries:**
- `pyopenms`: IdXML support
**EDA Approach:**
- Identification rate
- Score distribution
- Spectral match quality
- False discovery assessment
- Annotation transfer validation

## Lipidomics-Specific Formats

### .lcb - LipidCreator Batch
**Description:** LipidCreator transition list
**Typical Data:** Lipid transitions for targeted MS
**Use Cases:** Targeted lipidomics
**Python Libraries:**
- Export to CSV for processing
**EDA Approach:**
- Transition coverage per lipid class
- Retention time prediction
- Collision energy optimization
- Class-specific fragmentation patterns

### .mzTab - Proteomics/Metabolomics Tabular Format
**Description:** PSI tabular summary format
**Typical Data:** Protein/peptide/metabolite quantification
**Use Cases:** Publication and data sharing
**Python Libraries:**
- `pyteomics.mztab`
- `pandas` for TSV-like structure
**EDA Approach:**
- Data completeness
- Metadata section validation
- Quantification method
- Identification confidence
- Software and parameters
- Quality metrics summary

### .csv (LipidSearch, LipidMatch)
**Description:** Lipid identification results
**Typical Data:** Lipid annotations, grades, intensities
**Use Cases:** Lipidomics software output
**Python Libraries:**
- `pandas`: CSV reading
**EDA Approach:**
- Lipid class distribution
- Identification grade/confidence
- Fatty acid composition analysis
- Double bond and chain length patterns
- Intensity correlations
- Normalization to internal standards

### .sdf (Metabolomics)
**Description:** Structure data file for metabolites
**Typical Data:** Chemical structures with properties
**Use Cases:** Metabolite database creation
**Python Libraries:**
- `RDKit`: `Chem.SDMolSupplier('file.sdf')`
**EDA Approach:**
- Structure validation
- Property calculation (logP, MW, TPSA)
- Molecular formula consistency
- Tautomer enumeration
- Retention time prediction features

### .mol (Metabolomics)
**Description:** Single molecule structure files
**Typical Data:** Metabolite chemical structure
**Use Cases:** Structure-based searches
**Python Libraries:**
- `RDKit`: `Chem.MolFromMolFile('file.mol')`
**EDA Approach:**
- Structure correctness
- Stereochemistry validation
- Charge state
- Implicit hydrogen handling

## Data Processing and Analysis

### .h5 / .hdf5 (Omics)
**Description:** HDF5 for large omics datasets
**Typical Data:** Feature matrices, spectra, metadata
**Use Cases:** Large-scale studies, cloud computing
**Python Libraries:**
- `h5py`: HDF5 access
- `anndata`: For single-cell proteomics
**EDA Approach:**
- Dataset organization
- Chunking and compression
- Metadata structure
- Efficient data access patterns
- Sample and feature annotations

### .Rdata / .rds - R Objects
**Description:** Serialized R analysis objects
**Typical Data:** Processed omics results from R packages
**Use Cases:** xcms, CAMERA, MSnbase workflows
**Python Libraries:**
- `pyreadr`: `pyreadr.read_r('file.Rdata')`
- `rpy2`: R-Python integration
**EDA Approach:**
- Object structure exploration
- Data extraction
- Method parameter review
- Conversion to Python-native formats

### .mzTab-M - Metabolomics mzTab
**Description:** mzTab specific to metabolomics
**Typical Data:** Small molecule quantification
**Use Cases:** Metabolomics data sharing
**Python Libraries:**
- `pyteomics.mztab`: Can parse mzTab-M
**EDA Approach:**
- Small molecule evidence
- Feature quantification
- Database references (HMDB, KEGG, etc.)
- Adduct and charge annotation
- MS level information

### .parquet (Omics)
**Description:** Columnar storage for large tables
**Typical Data:** Feature matrices, metadata
**Use Cases:** Efficient big data omics
**Python Libraries:**
- `pandas`: `pd.read_parquet()`
- `pyarrow`: Direct parquet access
**EDA Approach:**
- Compression efficiency
- Column-wise statistics
- Partition structure
- Schema validation
- Fast filtering and aggregation

### .pkl (Omics Models)
**Description:** Pickled Python objects
**Typical Data:** ML models, processed data
**Use Cases:** Workflow intermediate storage
**Python Libraries:**
- `pickle`: Standard serialization
- `joblib`: Enhanced pickling
**EDA Approach:**
- Object type and structure
- Model parameters
- Feature importance (if ML model)
- Data shapes and types
- Deserialization validation

### .zarr (Omics)
**Description:** Chunked, compressed array storage
**Typical Data:** Multi-dimensional omics data
**Use Cases:** Cloud-optimized analysis
**Python Libraries:**
- `zarr`: Array storage
**EDA Approach:**
- Chunk optimization
- Compression codecs
- Multi-scale data
- Parallel access patterns
- Metadata annotations