Method Ml Track

---
name: method-ml-track
description: "Adds reproducible ML experiment tracking (data hashes, splits, metrics, SAP links) with MLflow, W&B, DVC, or an offline log. Use when building or reviewing ML experiments."
argument-hint: "[--backend mlflow|wandb|dvc|offline, --project name]"
---

# ML Experiment Tracking Setup


## Skill Value

- **Problem it solves:** ML analyses lose reproducibility when splits, seeds, metrics, and artifacts are not tracked consistently.
- **Use when:** The user builds or reviews ML experiments, prediction models, validation runs, or tracking setup.
- **Output:** Experiment tracking plan or scaffold with run metadata, metrics, artifacts, and reproducibility checks.
- **Ask only if:** Only when tracking backend, task type, or required metrics cannot be inferred.
- **Do not do:** Do not make a tracking service mandatory when local artifacts are sufficient.
- **Interaction:** Check repo/config/chat evidence first. Ask one decision-changing question at a time; use the current harness's blocking question UI when available, otherwise present numbered choices and wait.

Most ML papers in biomedicine are unreproducible because runs are not tracked. By the time the analyst writes the manuscript, "the version that got 0.89 AUC" is on no disk, with no record of hyperparameters or training-data hash. This skill wires tracking from day 1.

## When this skill activates

- New ML project (immediately after `method-cohort` and before model training)
- Mid-project transition from "exploratory" to "manuscript-bound" runs
- After `method-data-leakage-reviewer` flagged irreproducible eval
- Manual: `method-ml-track --backend mlflow --project sepsis-risk`

## Prerequisites

- A Python or R ML project (most modern frameworks have tracking integrations)
- For mlflow: pip-installed; tracking-server URL OR local `mlruns/` directory
- For wandb: API key in `~/.netrc` or env; project quota
- For dvc: dvc-installed; remote configured
- For offline: just file-system writable

## Core workflow

### Step 1: Pick the backend

| Backend | When to pick |
|---------|--------------|
| **mlflow** | Self-hostable, file or DB backend, multi-language (Python + R + Java). Default for most academic settings |
| **wandb** | SaaS-first, hosted UI, requires network (check the current plan terms for academic use). Pick when sharing with collaborators in real time and no PHI leaves the site |
| **dvc** | Git-native, versions data + model artifacts together. Pick for dataset-heavy projects with strict reproducibility |
| **offline** | YAML run logs in a `runs/` folder, no network, no UI. Pick when working on air-gapped or PHI-restricted machines |

If `--backend` not passed, infer from existing config (presence of `mlflow`, `wandb`, `.dvc/`); else prompt.

### Step 2: Generate boilerplate

Write `analysis/ml/tracking.py` (or `tracking.R`) with backend-specific init code:

**mlflow example:**

```python
import mlflow
import hashlib
from pathlib import Path

def sha256_file(path, chunk=1 << 20):
    """Stream the file so large extracts are not read into memory at once."""
    h = hashlib.sha256()
    with Path(path).open("rb") as fh:
        for block in iter(lambda: fh.read(chunk), b""):
            h.update(block)
    return h.hexdigest()

def init_run(train_path, experiment="sepsis-risk", run_name=None):
    # train_path comes from the project config, never a hard-coded path
    mlflow.set_tracking_uri("file:./mlruns")  # or http://tracking-server:5000
    mlflow.set_experiment(experiment)
    run = mlflow.start_run(run_name=run_name)
    # Always log data hash
    mlflow.log_param("train_data_sha", sha256_file(train_path)[:16])
    mlflow.log_param("git_sha", _git_sha())
    mlflow.log_param("study_id", _study_id())
    return run
```

### Step 3: Define the required-log schema

Write `analysis/ml/run-schema.yaml` -- the minimum fields every run must log:

```yaml
required:
  params:
    - train_data_sha
    - git_sha
    - study_id
    - random_seed
    - split_strategy          # group / temporal / random
    - n_configurations_evaluated  # running count of configurations tried in this experiment
  metrics:
    - auc_validation          # tuning/CV metric used for model selection
    - auc_validation_scheme   # e.g., grouped 5-fold CV by patient
    - n_train
    - n_events_train
  # Held-out test metrics (test_data_sha, auc_test, brier_test,
  # calibration_intercept_test, calibration_slope_test, n_test, n_events_test,
  # auc_external when an external set exists) are logged ONLY for the locked
  # final model, once; record test_set_unlocked_at.
  artifacts:
    - model.pkl (or equivalent)
    - eval-results.json
    - calibration-plot.png
    - feature-importance.csv (or shap_values.npz)
recommended:
  params:
    - hyperparam_search_strategy
    - model_class
  metrics:
    - subgroup_auc.<subgroup>
    - decision_curve.net_benefit_at_<threshold>
```

A run that does not log all required fields → not eligible for manuscript inclusion. A run that logs test-set metrics before the model is locked is not eligible either: scoring exploratory runs on the test set turns it into a selection set (REFORMS, Kapoor et al., Sci Adv 2024, PMID 38691601). Report `n_configurations_evaluated` with the final model (TRIPOD+AI item 12c, PMID 38626948).

### Step 4: Write the linkage to the SAP

Add to `analysis/sap.md`:

> "All ML runs are tracked via {{ backend }} at {{ uri }}. The run-id of the final model reported in the manuscript is logged in `analysis/ml/manuscript-run.txt` and committed alongside the SAP."

### Step 5: Provide the run-log fallback

For any backend, also write `runs/<run-id>.yaml` -- a YAML mirror of the tracked run. This way even if the tracking server is lost, the local repo has a record. `runs/` is committed to git.

```yaml
run_id: <uuid>
started: <ISO>
completed: <ISO>
backend: mlflow
backend_run_id: <mlflow run id>
params:
  train_data_sha: <sha256 of the training extract>
  # one key per hyperparameter
metrics:
  auc_validation: 0.84
  # one key per reported metric
artifacts:
  - name: model.pkl
    path: mlruns/.../model.pkl
    sha: ...
study_id: <id>
```

### Step 6: Emit signal

`__CE_ML_TRACK__ backend=<name> uri=<uri> schema=<path>` so subsequent skills know tracking exists.

## What this skill does NOT do

- Does not train models (use the `method-work` skill and write training scripts that call into the tracker)
- Does not enforce schema at training-time -- the discipline is the analyst's. The reviewer agent (eventually) checks runs/*.yaml against schema
- Does not version the data itself beyond hashing (use DVC for data versioning)
- Does not push runs to a public registry

## References

@./references/backends-comparison.md

@./references/run-log-schema.yaml