Installs into .claude/skills of the current project.
Are you the author of Calibrate Probabilities?
Add the live security badge to your README. It updates with every re-scan.
[](https://www.skillsdirectory.com/skills/danieltomaro13-calibrate-probabilities)
---
name: calibrate_probabilities
description: How to evaluate and fix probability calibration — holdout discipline, Brier/log-loss reading, Platt-style rescaling.
triggers: [calibrate, calibration, brier, log loss, log-loss, overconfident]
---
# Calibrating probabilities
A model that says 70% should be right ~70% of the time. Calibration is measured,
never assumed.
## Measure (always on holdout)
- `calibration_metrics(pairs)` where pairs = holdout `{prob, outcome}` rows.
- **Brier**: mean squared error. 0 = oracle; 0.25 = coin flip on a balanced set;
beating the market baseline matters more than the absolute number.
- **Log-loss**: punishes confident wrongness. If log-loss looks much worse than
Brier, the model is overconfident in its tails.
## Fix overconfidence (in run_python)
- Shrink toward the base rate: `p' = w * p + (1 - w) * base_rate`, fit w on a
validation slice (grid over w ∈ [0.5..1.0] minimising log-loss is fine).
- Platt-style: fit logistic regression of outcome on logit(p) — two parameters,
hard to overfit; refuse fancier recalibration without more than ~200 samples.
- Re-run `calibration_metrics` AFTER rescaling and report both before/after.
## Persist
`save_model` with the post-calibration metrics and the rescaling parameters in
`params` — the next session must be able to reproduce the pipeline from the row.
