Model Development

---
name: model_development
description: The general model-building method for ANY market — framing, sample-size discipline, feature selection (user priors + data-driven), baselines, leakage-safe validation, calibration, persistence.
triggers: [build a model, train a model, model development, predictive model, new model, build me a model]
---
# Model development — the general method

This is the method for every market (winners, totals, lines, props). Market recipes
(`build_a_totals_model`, `build_a_h2h_model`) are worked examples OF this method,
never replacements for it.

## 1. Frame the problem

- What exactly is being predicted? One binary probability per selection is the
  default (P(home win), P(over)). Name the market convention (whose line, which
  book) before any code.
- What is the decision the probability feeds? A backtest entry needs the prob
  BEFORE the price moves — timestamps are part of the model contract
  (`record_predictions` takes `predicted_at`; never backdate dishonestly).

## 2. Sample size BEFORE cleverness

- Rule of thumb: **10–20 outcomes per model parameter** as a floor. A logistic
  regression with 5 features wants 100+ decided events; anything fancier wants
  several times that.
- "Last 20 games" is rarely enough — it estimates a mean with ±20%+ noise and one
  parameter eats half of it. Prefer **multiple seasons with recency weighting**
  (exponential decay) over tiny recent windows.
- Sport cadence changes everything: an MLB team plays 162 games/season, an NFL team
  17 — the same "two seasons of data" is 324 events in one sport and 34 in the
  other. Count EVENTS, not calendar time.
- Watch regime changes: rule changes, roster turnover, venue moves. Old data is
  only valuable while the process that generated it still operates — say so when
  you truncate history and why.
- When the data cannot support the model requested, SAY THAT PLAINLY and build the
  smaller model that it can support.

## 3. Features: ask AND measure

- When the request is open-ended, **ask the user which stats they believe matter**
  for this market — domain priors are real information and it is their model.
- Independently **measure importance from the data** (univariate signal, simple
  permutation importance in run_python) — then report where the user's priors and
  the data disagree, with numbers. Do not silently drop either.
- Fewer features beat more: every feature is a parameter and §2 already priced
  parameters. Justify each one in a sentence.

## 4. Baseline first

- The market's vig-removed probabilities ARE the baseline (`value_finder` shows
  them). A model that cannot beat the closing line has no edge regardless of its
  Brier — report your model AND the market baseline side by side.
- The second baseline is the base rate (home teams win X%). Beat both or say so.

## 5. Validate without lying to yourself

- Time-ordered splits only: train on the past, test on the future. **Never shuffle
  time**; walk-forward (rolling refit) is the gold standard when you have volume.
- Leakage checklist before quoting any holdout number: no post-game stats in
  pre-game features, no closing prices as features for an entry-time model, no
  target-derived columns, holdout events strictly after every training event.
- Report holdout Brier/log-loss via `calibration_metrics` (see `quant_concepts`
  for how to read them), then `save_model` — it refuses uncalibrated models by
  design. Record forward picks with `record_predictions`.

## 6. Honesty rules

- Print every number in run_python; quote only printed numbers.
- Probabilities only — sizing belongs to the bankroll manager; no locks, no
  guarantees, the user decides.