Reproducible Training Runs

---
name: reproducible-training-runs
description: Use when reviewing or modifying ML training scripts that must produce identical results across runs or machines -- runs with the "same" config differ, or a past result must be reconstructed exactly.
---

## Purpose

Enforce seed setting, deterministic operations, and environment tracking so a training run can be reproduced exactly.

## When to Use

- Reviewing or modifying a training script that must be deterministic
- Two runs with the "same" configuration produced different results
- A past result needs to be reconstructed exactly

## Inputs

- A target Python training script

## Workflow

1. **Global seed initialization**: ensure a single function sets seeds for all relevant libraries (`random`, `numpy`, `torch`, `tensorflow`).
2. **Deterministic algorithms**: for PyTorch or TensorFlow, check that deterministic algorithms are enabled (e.g., `torch.use_deterministic_algorithms(True)`).
3. **Data loading**: verify data loaders use deterministic shuffling and that worker processes are seeded correctly to avoid identical augmentations.
4. **Environment & config tracking**: ensure the script logs the exact configuration, dependency versions, and data hashes.
5. **Review first**: point out missing reproducibility guards before rewriting the script. Provide the exact seed-initialization snippet — do not hide side effects.

## Output

- A list of missing reproducibility guards with the exact code snippets to add, plus any performance trade-off warnings

## Verification

- [ ] All library seeds set from one place
- [ ] Deterministic-algorithm flags enabled (or the gap explicitly accepted)
- [ ] Loader shuffling and worker seeding deterministic
- [ ] Configuration, dependency versions, and data hashes logged
- [ ] User warned if determinism flags significantly slow training

## Failure Modes

- **Partial seeding** — seeding `random` but not the framework or loader workers still yields nondeterminism.
- **Silent slowdown** — enabling deterministic algorithms can cost real training speed; surface the trade-off instead of hiding it.
- **Rewriting before reviewing** — changing the script without first listing the gaps loses the audit trail.