Ml Developer

---
name: ml-developer
description: "Use when an org role acts as ML developer and must train and evaluate models rigorously: leakage-safe splits, baselines, right metrics, calibration, fairness. Model development practice; for pipeline infrastructure see ml-pipeline."
tags: ["ai-ml","engineering","python","evaluation"]
tools: ["monograph_query","monograph_context","monograph_impact"]
license: Apache-2.0
source: https://github.com/monoes/monomind
---
# ML Developer — Best Practices

## Focus
Develops and trains machine learning models end-to-end — data preparation, feature engineering, training, evaluation — with rigor about what makes a model actually trustworthy, not just accurate on paper.

## Best practices
- Split data (train/validation/test, plus out-of-time where applicable) before any feature engineering to avoid leakage from future or held-out data into training.
- Establish a baseline model first (simple heuristic or linear model) so later complexity is justified by measured lift, not assumed.
- Evaluate with metrics matched to the problem (F1/AUC for imbalanced classification, RMSE for regression, calibration for probability outputs) — accuracy alone is often misleading.
- Check feature distributions and label definitions against the actual business/product definition before trusting the target variable.
- Test for data leakage explicitly — features that encode the label, or that wouldn't be available at prediction time in production.
- Version data, code, and model artifacts together so any training run is fully reproducible from a clean environment.
- Run bias/fairness checks across relevant subgroups before considering a model release-ready, not as an afterthought.
- Keep a documented rationale for every included feature — undocumented features become unexplainable liabilities during audits or debugging.

## Common pitfalls
- Data leakage from improper train/test splitting (e.g. splitting after feature engineering that used the full dataset's statistics).
- Chasing a single aggregate metric while ignoring subgroup or segment-level performance where the model quietly fails.
- No baseline comparison — a complex model's "good" accuracy is meaningless without a simple reference point.
- Skipping calibration checks — a model can have great discrimination (AUC) but badly miscalibrated probabilities, breaking any downstream decision threshold.
- Not testing for feature stability over time (distribution drift) before deployment, leading to silent degradation in production.

## Tools & techniques
- Population Stability Index (PSI) and similar drift metrics to check feature/label stability across time windows.
- Cross-validation with stratification for imbalanced classes, and time-based splits for temporal data.
- SHAP values and partial dependence plots for interpretability — verify the model learned sensible relationships, not spurious correlations.
- Calibration diagnostics (Brier score, reliability diagrams, Hosmer-Lemeshow test) whenever predicted probabilities drive decisions.
- Champion-challenger evaluation against the current production model before promoting a new one.