Machine Learning

---
name: machine-learning
description: Machine learning development patterns, model training, evaluation, and
  deployment. Use when building ML pipelines, training models, feature engineering,
  model evaluation, or deploying ML systems to production.
author: Joseph OBrien
status: unpublished
updated: '2025-12-23'
version: 1.0.1
tag: skill
type: skill
---

# Machine Learning

Comprehensive machine learning skill covering the full ML lifecycle from experimentation to production deployment.

## When to Use This Skill

- Building machine learning pipelines
- Feature engineering and data preprocessing
- Model training, evaluation, and selection
- Hyperparameter tuning and optimization
- Model deployment and serving
- ML experiment tracking and versioning
- Production ML monitoring and maintenance

## ML Development Lifecycle

### 1. Problem Definition

**Classification Types:**

- Binary classification (spam/not spam)
- Multi-class classification (image categories)
- Multi-label classification (document tags)
- Regression (price prediction)
- Clustering (customer segmentation)
- Ranking (search results)
- Anomaly detection (fraud detection)

**Success Metrics by Problem Type:**

| Problem Type | Primary Metrics | Secondary Metrics |
|--------------|-----------------|-------------------|
| Binary Classification | AUC-ROC, F1 | Precision, Recall, PR-AUC |
| Multi-class | Macro F1, Accuracy | Per-class metrics |
| Regression | RMSE, MAE | R², MAPE |
| Ranking | NDCG, MAP | MRR |
| Clustering | Silhouette, Calinski-Harabasz | Davies-Bouldin |

### 2. Data Preparation

**Data Quality Checks:**

- Missing value analysis and imputation strategies
- Outlier detection and handling
- Data type validation
- Distribution analysis
- Target leakage detection

**Feature Engineering Patterns:**

- Numerical: scaling, binning, log transforms, polynomial features
- Categorical: one-hot, target encoding, frequency encoding, embeddings
- Temporal: lag features, rolling statistics, cyclical encoding
- Text: TF-IDF, word embeddings, transformer embeddings
- Geospatial: distance features, clustering, grid encoding

**Train/Test Split Strategies:**

- Random split (standard)
- Stratified split (imbalanced classes)
- Time-based split (temporal data)
- Group split (prevent data leakage)
- K-fold cross-validation

### 3. Model Selection

**Algorithm Selection Guide:**

| Data Size | Problem | Recommended Models |
|-----------|---------|-------------------|
| Small (<10K) | Classification | Logistic Regression, SVM, Random Forest |
| Small (<10K) | Regression | Linear Regression, Ridge, SVR |
| Medium (10K-1M) | Classification | XGBoost, LightGBM, Neural Networks |
| Medium (10K-1M) | Regression | XGBoost, LightGBM, Neural Networks |
| Large (>1M) | Any | Deep Learning, Distributed training |
| Tabular | Any | Gradient Boosting (XGBoost, LightGBM, CatBoost) |
| Images | Classification | CNN, ResNet, EfficientNet, Vision Transformers |
| Text | NLP | Transformers (BERT, RoBERTa, GPT) |
| Sequential | Time Series | LSTM, Transformer, Prophet |

### 4. Model Training

**Hyperparameter Tuning:**

- Grid Search: exhaustive, good for small spaces
- Random Search: efficient, good for large spaces
- Bayesian Optimization: smart exploration (Optuna, Hyperopt)
- Early stopping: prevent overfitting

**Common Hyperparameters:**

| Model | Key Parameters |
|-------|---------------|
| XGBoost | learning_rate, max_depth, n_estimators, subsample |
| LightGBM | num_leaves, learning_rate, n_estimators, feature_fraction |
| Random Forest | n_estimators, max_depth, min_samples_split |
| Neural Networks | learning_rate, batch_size, layers, dropout |

### 5. Model Evaluation

**Evaluation Best Practices:**

- Always use held-out test set for final evaluation
- Use cross-validation during development
- Check for overfitting (train vs validation gap)
- Evaluate on multiple metrics
- Analyze errors qualitatively

**Handling Imbalanced Data:**

- Resampling: SMOTE, undersampling
- Class weights: weighted loss functions
- Threshold tuning: optimize decision threshold
- Evaluation: use PR-AUC over ROC-AUC

### 6. Production Deployment

**Model Serving Patterns:**

- REST API (Flask, FastAPI, TF Serving)
- Batch inference (scheduled jobs)
- Streaming (real-time predictions)
- Edge deployment (mobile, IoT)

**Production Considerations:**

- Latency requirements (p50, p95, p99)
- Throughput (requests per second)
- Model size and memory footprint
- Fallback strategies
- A/B testing framework

### 7. Monitoring & Maintenance

**What to Monitor:**

- Prediction latency
- Input feature distributions (data drift)
- Prediction distributions (concept drift)
- Model performance metrics
- Error rates and types

**Retraining Triggers:**

- Performance degradation below threshold
- Significant data drift detected
- Scheduled retraining (daily, weekly)
- New training data available

## MLOps Best Practices

### Experiment Tracking

Track for every experiment:

- Code version (git commit)
- Data version (hash or version ID)
- Hyperparameters
- Metrics (train, validation, test)
- Model artifacts
- Environment (packages, versions)

### Model Versioning

```
models/
├── model_v1.0.0/
│   ├── model.pkl
│   ├── metadata.json
│   ├── requirements.txt
│   └── metrics.json
├── model_v1.1.0/
└── model_v2.0.0/
```

### CI/CD for ML

1. **Continuous Integration:**
   - Data validation tests
   - Model training tests
   - Performance regression tests

2. **Continuous Deployment:**
   - Staging environment validation
   - Shadow mode testing
   - Gradual rollout (canary)
   - Automatic rollback

## Reference Files

For detailed patterns and code examples, load reference files as needed:

- **`references/preprocessing.md`** - Data preprocessing patterns and feature engineering techniques
- **`references/model_patterns.md`** - Model architecture patterns and implementation examples
- **`references/evaluation.md`** - Comprehensive evaluation strategies and metrics

## Integration with Other Skills

- **performance** - For optimizing inference latency
- **testing** - For ML-specific testing patterns
- **database-optimization** - For feature store queries
- **debugging** - For model debugging and error analysis