Machine Learning Hyperparameter Extraction

---
name: machine-learning-hyperparameter-extraction
description: Use when you have completed cross-validation tuning of one or more machine-learning
  models (e.g., AdaBoost, SVM, Random Forest) on a development dataset using the caret
  package and need to identify which specific hyperparameter values were selected
  as optimal.
license: CC-BY-4.0
metadata:
  edam_operation: http://edamontology.org/operation_3664
  edam_topics:
  - http://edamontology.org/topic_0091
  - http://edamontology.org/topic_3673
  tools:
  - MetaClean
  - R
  - caret
  license_tier: restricted
  provenance_tier: literature
derived_from:
- doi: 10.1007/s11306-020-01738-3
  title: MetaClean
- doi: 10.1186/1471-2105-15-s11-s5
  title: ''
evidence_spans:
- MetaClean is a package for building classifiers to identify low quality integrations
  in untargeted metabolomics data.
- '`MetaClean` provides 8 classification algorithms (implemented with the R package
  `caret`) for building a predictive model.'
- getEvalObj is called to extract the relevant data from the three objects provided
  by ther user and store them in an object of class evalObj
- It is an R package and can be easily incorporated
- MetaClean provides 8 classification algorithms (implemented with the R package caret)
claims: []
provenance:
  collection: https://w3id.org/holobiomicslab/asb-skill/collection/metabolomics/v2
  assembled_by: scripts/collect_metabolomics_collection.py
  sources:
  - build: coll_metaclean_cq
    doi: 10.1007/s11306-020-01738-3
    title: MetaClean
  dedup_kept_from: coll_metaclean_cq
schema_version: 0.2.0
attribution:
  generator: AgenticScienceBuilder
  original_doi: 10.1007/s11306-020-01738-3
  all_source_dois:
  - 10.1007/s11306-020-01738-3
  - 10.1186/1471-2105-15-s11-s5
  zenodo_doi: 10.5281/zenodo.20794027
  curators: []
  promoter: Louis-Félix Nothias
  sponsor: CNRS & Université Côte d'Azur
---

# Machine-Learning Hyperparameter Extraction

> **License: restricted** — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution. <!-- asb-license-banner -->
## Summary

Extract and validate optimized hyperparameter values (e.g., nIter, method) for trained machine-learning classifiers from cross-validation output. This skill identifies which hyperparameter combinations were selected during model tuning and documents them for reproducibility and model retraining.

## When to use

You have completed cross-validation tuning of one or more machine-learning models (e.g., AdaBoost, SVM, Random Forest) on a development dataset using the caret package and need to identify which specific hyperparameter values were selected as optimal. This is necessary before training a final classifier on the full development set or before applying the model to held-out test data.

## When NOT to use

- You are using pre-trained models from external sources without access to their cross-validation history or prediction data frames.
- Hyperparameter tuning was not performed; the model uses default hyperparameters and no runCrossValidation() output is available.
- The evaluation measures output is unavailable or does not align with the pred data frame (mismatched cross-validation runs).

## Inputs

- List object returned by runCrossValidation() containing trained models
- Prediction data frame from models$[AlgorithmName]$pred
- Evaluation measures data frame from getEvaluationMeasures()
- Cross-validation parameters (k-fold number, repetition number, metric set)

## Outputs

- Extracted hyperparameter combinations (e.g., nIter, method values)
- Identified optimized hyperparameter set for best-performing model configuration
- Validated correspondence between hyperparameters and evaluation metrics

## How to apply

After running runCrossValidation() with specified k-fold and repetition parameters (e.g., k=5, repNum=10) and a selected metric set (e.g., metricSet=c('M4','M7','M11')), access the prediction data frame returned by the trained model object (e.g., models$AdaBoost_M11$pred). Extract the unique hyperparameter columns (nIter, method, etc.) from this pred data frame to identify all candidate hyperparameter combinations tested. Cross-reference the extracted combinations against the evaluation measures output (generated by getEvaluationMeasures()) to confirm which hyperparameter set corresponds to the best-performing configuration according to your chosen evaluation metric. Document the optimized values for use in the subsequent trainClassifier() call on the full development set.

## Related tools

- **caret** (R package used to implement machine-learning algorithms and manage cross-validation hyperparameter tuning)
- **MetaClean** (Peak-quality classifier package providing wrapper functions runCrossValidation(), getEvaluationMeasures(), and trainClassifier() that manage hyperparameter tuning and extraction workflow) — https://github.com/KelseyChetnik/MetaClean
- **R** (Statistical computing environment in which hyperparameter extraction, data frame manipulation, and validation are performed)

## Examples

```
models <- runCrossValidation(trainData = pqm_development, k = 5, repNum = 10, rand.seed = 512, models = 'all', metricSet = c('M4','M7','M11')); hyperparams <- unique(models$AdaBoost_M11$pred[, c('nIter', 'method')]); eval_measures <- getEvaluationMeasures(models); best_idx <- which.max(eval_measures$Accuracy); optimal_hyperparams <- hyperparams[best_idx, ]
```

## Evaluation signals

- The extracted hyperparameter columns (nIter, method) are present and non-null in the pred data frame.
- The unique hyperparameter combinations extracted match the set of combinations that were iterated during cross-validation (no spurious or missing values).
- The optimized hyperparameter set identified can be cross-referenced to the row(s) in the evaluation measures output with the best (highest or lowest, depending on metric direction) performance score.
- The hyperparameter values are numeric or character strings matching the data types expected by trainClassifier() (e.g., nIter is an integer, method is a character string such as 'Adaboost.M1').
- When the optimized hyperparameters are passed to trainClassifier() on the full development set, model training succeeds without parameter-type or range errors.

## Limitations

- The article does not provide the hyperparameter values in the reported text; extraction relies entirely on programmatic access to the pred data frame, which is not published in the paper.
- Cross-validation outputs are sensitive to random seed and sampling strategy; reproducibility requires recording the exact runCrossValidation() parameters (k, repNum, rand.seed) used during tuning.
- If multiple hyperparameter combinations achieve similar evaluation scores, the extraction method does not disambiguate; practitioners must apply domain knowledge or additional criteria to select among tied candidates.
- The skill applies to MetaClean's wrapper functions and caret-compatible algorithms; extraction workflows may differ for other machine-learning frameworks or packages.

## Evidence

- [other] Load the trained models list returned by runCrossValidation() with parameters k=5, repNum=10, rand.seed=512, models='all', metricSet=c('M4','M7','M11') on pqm_development (500 peaks, 89 samples). 2. Access the prediction data frame from models$AdaBoost_M11$pred. 3. Extract the unique hyperparameter combinations (nIter and method columns) from the pred data frame.: "Access the prediction data frame from models$AdaBoost_M11$pred. 3. Extract the unique hyperparameter combinations (nIter and method columns) from the pred data frame."
- [other] Identify and record the optimized hyperparameters determined during cross-validation (nIter=150, method='Adaboost.M1' per the example). 5. Validate that these hyperparameters correspond to the best-performing AdaBoost configuration by cross-referencing with evaluation measures output.: "Validate that these hyperparameters correspond to the best-performing AdaBoost configuration by cross-referencing with evaluation measures output."
- [methods] The runCrossValidation function is a wrapper function that uses cross-validation to train a user-selected subset of the 8 available algorithms.: "The runCrossValidation function is a wrapper function that uses cross-validation to train a user-selected subset of the 8 available algorithms."
- [methods] We use the getEvaluationMeasures function to do this.: "We use the getEvaluationMeasures function to do this."
- [methods] metaclean_model <- trainClassifier(trainData = pqm_development, model = "AdaBoost", metricSet = "M11", hyperparameters = hyperparameters): "trainClassifier(trainData = pqm_development, model = "AdaBoost", metricSet = "M11", hyperparameters = hyperparameters)"