Optimization Project Structure

---
name: optimization-project-structure
description: When the user wants to organize optimization research code into a reproducible project — separating the src package, scripts, configs, and results; JSON/YAML config systems; factory registration of algorithms and problems; seed discipline; atomic result writing; and light testing. Also use when the user mentions "project structure," "research code organization," "config file," "factory pattern," "reproducible runs," or "random seed," or when results can no longer be traced to the exact code and parameters that produced them. For result tables and aggregation, see pandas-experiment-management; for commit and tag discipline, see git-for-research-code.
---

# Optimization Project Structure

You are an expert in structuring research code for combinatorial optimization: package layout,
configuration systems, algorithm/problem factories, seeding discipline, atomic result writing,
and a light testing tier. This skill is a pattern catalog: each pattern gives a short
motivation, a complete implementation, and the pitfall that most often invalidates experiments.
Use the framework below to size the structure to the project, then adapt the matching patterns.

## Initial Assessment

Establish the following before recommending any structure:

- **Project lifetime and stakes.** A one-week exploration tolerates a single script. A thesis
  or paper project will produce thousands of runs over months and must support the question
  "which code and parameters produced this number in Table 3?"
- **Count the axes of variation.** How many problems, algorithms, parameter settings,
  instances, and seeds will be combined? The product of these axes is the number of runs the
  structure must name, store, and aggregate without collisions.
- **Compute environment.** Laptop only, or also a cluster/colleague's machine? Anything beyond
  one machine forbids absolute paths, machine-local state, and manual run bookkeeping.
- **Current state of the code.** Greenfield, a pile of notebooks, or a working flat script?
  Migration order matters: extract the library first, configs second, orchestration last.
- **Result volume and format.** Hundreds of scalar records fit JSON-per-run plus a CSV
  aggregate. Millions of rows or per-iteration traces need parquet and a separate trace
  directory from day one.
- **Failure tolerance of campaigns.** Will runs take seconds or hours? Long campaigns need
  resumability (skip completed runs) and per-run failure isolation; short ones do not.
- **Sources of randomness.** List them: metaheuristic moves, instance generation, tie-breaking,
  solver internals. Every source must trace back to a recorded seed.
- **External solvers and licenses.** Solver version and parameter files are part of provenance;
  a Gurobi version bump can change every number in a results table.
- **Who else runs this code.** Solo use allows conventions in your head; a second user (advisor,
  reviewer, future you in 18 months) requires the conventions to be in the repository: README,
  configs, and tests.
- **What the paper needs.** Identify the final artifacts (tables, figures, per-instance bests)
  and design the results directory and record schema backward from them.

## Project Anatomy and the Run Contract

The reference layout, sized for a typical metaheuristic-vs-baseline paper project:

```text
myproject/
├── pyproject.toml            # package metadata; enables `pip install -e .`
├── README.md                 # how to install, run one experiment, run tests
├── configs/
│   ├── base.yaml             # campaign definitions: data, not code
│   └── sweeps/
│       └── ils_kick.yaml
├── data/
│   └── instances/            # read-only inputs; never written by runs
├── results/                  # write-only outputs; gitignored; regenerable
│   ├── raw/                  # one JSON record per run (immutable)
│   ├── tables/               # aggregated CSV/parquet, regenerated by scripts
│   └── figures/
├── scripts/
│   ├── run_experiment.py     # thin entry point: args -> config -> library
│   ├── aggregate_results.py
│   └── make_figures.py
├── src/
│   └── myopt/
│       ├── __init__.py       # imports problem/algorithm modules -> fills registries
│       ├── algorithms/       # one module per algorithm
│       ├── problems/         # one module per problem class
│       ├── io/               # instance parsing, atomic result writing
│       ├── config.py
│       ├── provenance.py
│       ├── registry.py
│       └── seeding.py
└── tests/
    └── test_core.py          # fast tier; slow tests marked separately
```

Each layer has one rule that gives the separation its value:

| Layer | Lives in | Rule |
|---|---|---|
| Library code | `src/myopt/` | Importable, no import-time work, no hard-coded paths, no I/O surprises |
| Entry points | `scripts/` | Thin: parse arguments, load config, call the library, nothing else |
| Experiment definitions | `configs/` | Data, not code; committed to git; one file per campaign |
| Inputs | `data/instances/` | Read-only; runs never modify instances |
| Outputs | `results/` | Write-only by runs; gitignored; raw records immutable |
| Tests | `tests/` | Fast tier runs in seconds, always; slow tier opt-in |

**The run contract.** Everything in this skill serves one formal statement. A run is a function

$$ r = \mathrm{run}(I,\; A,\; \theta,\; s) $$

mapping an instance $I$, an algorithm $A$, a parameter vector $\theta$, and a seed $s$ to a
result record $r$ (objective, solution, runtime, status, provenance). Reproducibility means:
given the same $(I, A, \theta, s)$ and the same code version, $\mathrm{run}$ returns the same
record up to wall-clock fields. The project structure exists to make every argument of this
function explicit, recorded, and recoverable — instances by name in `data/`, algorithms by
registered name, $\theta$ in a config file, $s$ in the record, and the code version as a git
hash inside $r$. This is the operational core of Sandve et al. (2013), "Ten Simple Rules for
Reproducible Computational Research," and of the reporting standards argued by Johnson (2002),
"A Theoretician's Guide to the Experimental Analysis of Algorithms."

**Decision guidance — how much structure:**

- **Single flat script** when: one problem, one algorithm, exploration that will be discarded.
  Still use `default_rng(seed)` and print the seed; throwaway code has a habit of surviving.
- **Script + config file** when: one algorithm but parameter studies begin. The moment you edit
  a constant in code to launch a second run, move constants to a config.
- **Full layout above** when: two or more algorithms or problems will be compared, or any
  result might appear in a paper. Half a day of migration, amortized over every later run.
- **Do not add** plugin systems, abstract base class towers, or microservice splits. Research
  code optimizes for change velocity and auditability, not for extensibility by strangers
  (Wilson et al. 2017, "Good Enough Practices in Scientific Computing").

Complexity note: runtime overhead is negligible — config parsing and JSON writing cost
microseconds against seconds-to-hours solves. The cost is cognitive, paid once, and repaid on
every run, debug session, and reviewer question.

## Configuration Patterns

### Pattern: frozen dataclass configs loaded from YAML/JSON

Configs as plain dicts spread `cfg["algorithm"]["params"]["kick_strenght"]` typos through the
code and fail silently. A frozen dataclass gives attribute access, type hints, one schema
definition, and immutability — nothing downstream can quietly edit the experiment definition.
Unknown keys must be rejected at load time: a misspelled YAML key must crash, not be ignored.

```python
# src/myopt/config.py
from __future__ import annotations

import json
from dataclasses import dataclass, field, fields
from pathlib import Path
from typing import Any

import yaml


@dataclass(frozen=True)
class AlgorithmConfig:
    """Which algorithm to build and with which parameters."""

    name: str
    params: dict[str, Any] = field(default_factory=dict)


@dataclass(frozen=True)
class ExperimentConfig:
    """Full description of one experiment campaign."""

    experiment_name: str
    instances: list[str]
    algorithm: AlgorithmConfig
    seeds: list[int]
    time_limit_s: float
    results_dir: str = "results/raw"


def _build(cls: type, raw: dict[str, Any]) -> Any:
    """Construct a dataclass from a dict, rejecting unknown keys loudly."""
    known = {f.name for f in fields(cls)}
    unknown = set(raw) - known
    if unknown:
        raise ValueError(f"unknown config keys for {cls.__name__}: {sorted(unknown)}")
    kwargs = dict(raw)
    if isinstance(kwargs.get("algorithm"), dict):
        kwargs["algorithm"] = _build(AlgorithmConfig, kwargs["algorithm"])
    return cls(**kwargs)


def load_config(path: str | Path) -> ExperimentConfig:
    """Load an ExperimentConfig from a .yaml/.yml or .json file."""
    path = Path(path)
    text = path.read_text(encoding="utf-8")
    raw = yaml.safe_load(text) if path.suffix in {".yaml", ".yml"} else json.loads(text)
    return _build(ExperimentConfig, raw)


# Tiny synthetic instance: parse a config from text and read it back.
cfg_text = """
experiment_name: ils_baseline
instances: [rand50_a, rand50_b]
algorithm: {name: ils, params: {kick_strength: 4, max_iters: 2000}}
seeds: [0, 1, 2]
time_limit_s: 10.0
"""
cfg = _build(ExperimentConfig, yaml.safe_load(cfg_text))
print(cfg.algorithm.name, cfg.seeds, cfg.results_dir)
# Expected: "ils [0, 1, 2] results/raw"; a misspelled key such as 'sedes' raises ValueError
```

**Pitfall:** defaults defined in two places. If `kick_strength` defaults to 3 in the
algorithm's `__init__` and is 4 in `base.yaml`, the recorded config and the executed parameters
diverge whenever a config omits the key. Rule: defaults live in exactly one place (usually the
constructor), and the result record stores the *resolved* parameters actually passed to the
constructor, never the raw config fragment.

### Pattern: run identity from a canonical config hash

Runs need stable, collision-free names for files and skip-completed logic. Deriving the name
from a hash of the canonical JSON of (parameters, instance, seed) makes identity follow
content: the same experiment always maps to the same filename, any change produces a new one.

```python
# src/myopt/run_id.py
import hashlib
import json
from typing import Any


def canonical_json(obj: Any) -> str:
    """Serialize with sorted keys so semantically equal configs hash equally."""
    return json.dumps(obj, sort_keys=True, separators=(",", ":"), default=str)


def run_id(algo_config: dict[str, Any], instance: str, seed: int) -> str:
    """Stable short identifier for one (algorithm config, instance, seed) run."""
    payload = canonical_json({"algorithm": algo_config, "instance": instance, "seed": seed})
    digest = hashlib.sha256(payload.encode("utf-8")).hexdigest()[:12]
    return f"{instance}_s{seed}_{digest}"


algo = {"name": "ils", "params": {"kick_strength": 4, "max_iters": 2000}}
reordered = {"params": {"max_iters": 2000, "kick_strength": 4}, "name": "ils"}
print(run_id(algo, "rand50_a", 7))
print(run_id(algo, "rand50_a", 7) == run_id(reordered, "rand50_a", 7))
# Expected: an id like "rand50_a_s7_<12 hex chars>", then True (key order is irrelevant)
```

**Pitfall:** hashing objects with unstable string forms. `default=str` makes numpy scalars and
`Path` objects serializable, but a Python `set` serializes in arbitrary order and a float
computed two ways may differ in the last bit. Keep config values JSON-native (str, int, float,
bool, list, dict), and hash the inputs that generate derived floats, never the floats.

## Factory and Interface Patterns

### Pattern: registry with decorator registration

The run script must turn the string `"ils"` from a config file into an algorithm object. A
chain of `if name == "ils": ...` branches grows stale and hides what is available. A registry
maps names to factories, registration happens next to the class definition, and the error
message for an unknown name lists every valid option.

```python
# src/myopt/registry.py
from __future__ import annotations

from typing import Callable, TypeVar

T = TypeVar("T")


class Registry:
    """Name -> factory mapping with decorator-based registration."""

    def __init__(self, kind: str) -> None:
        self.kind = kind
        self._factories: dict[str, Callable[..., object]] = {}

    def register(self, name: str) -> Callable[[Callable[..., T]], Callable[..., T]]:
        """Decorator that registers a class or factory function under `name`."""
        def wrap(factory: Callable[..., T]) -> Callable[..., T]:
            if name in self._factories:
                raise ValueError(f"{self.kind} '{name}' registered twice")
            self._factories[name] = factory
            return factory
        return wrap

    def create(self, name: str, **kwargs: object) -> object:
        """Instantiate the registered factory, failing with the list of known names."""
        if name not in self._factories:
            raise KeyError(f"unknown {self.kind} '{name}'; known: {self.names()}")
        return self._factories[name](**kwargs)

    def names(self) -> list[str]:
        """Sorted registered names — useful in tests and error messages."""
        return sorted(self._factories)


ALGORITHMS = Registry("algorithm")
PROBLEMS = Registry("problem")


@ALGORITHMS.register("ils")
class IteratedLocalSearch:
    """Stand-in algorithm class; real moves live in their own module."""

    def __init__(self, kick_strength: int = 3, max_iters: int = 1000) -> None:
        self.kick_strength = kick_strength
        self.max_iters = max_iters


algo = ALGORITHMS.create("ils", kick_strength=5)
print(type(algo).__name__, algo.kick_strength, ALGORITHMS.names())
# Expected: "IteratedLocalSearch 5 ['ils']"; ALGORITHMS.create("tabu") raises KeyError naming 'ils'
```

**Pitfall:** the empty-registry trap. Registration runs when the defining module is imported;
if nothing imports `myopt/algorithms/ils.py`, the registry stays empty and `create("ils")`
fails only at run time. Fix structurally: `myopt/__init__.py` imports every algorithm and
problem module explicitly, and a one-line test asserts the expected names exist. Avoid
`importlib` directory scans — they trade an explicit list for import-order surprises.

### Pattern: Protocol interfaces between problems and algorithms

Algorithms should depend on a minimal problem interface, not on concrete classes, so a new
problem plugs into every existing algorithm. `typing.Protocol` expresses the contract with
static checking and zero inheritance coupling — concrete classes do not subclass anything.

```python
# src/myopt/problems/base.py (Protocol) and a concrete pair to show the fit
from __future__ import annotations

from typing import Protocol

import numpy as np


class Problem(Protocol):
    """Contract every problem satisfies; algorithms depend only on this."""

    name: str

    def evaluate(self, solution: np.ndarray) -> float:
        """Objective value of one solution (minimization)."""
        ...

    def random_solution(self, rng: np.random.Generator) -> np.ndarray:
        """Sample one feasible solution."""
        ...


class TSPProblem:
    """Symmetric TSP on a distance matrix; solutions are permutations of range(n)."""

    def __init__(self, name: str, dist: np.ndarray) -> None:
        self.name = name
        self.dist = dist
        self.n = dist.shape[0]

    def evaluate(self, solution: np.ndarray) -> float:
        """Tour length, closing the cycle, fully vectorized."""
        return float(self.dist[solution, np.roll(solution, -1)].sum())

    def random_solution(self, rng: np.random.Generator) -> np.ndarray:
        """Uniform random permutation."""
        return rng.permutation(self.n)


class RandomRestartSearch:
    """Baseline algorithm: best of n_samples random solutions."""

    def __init__(self, n_samples: int = 1000) -> None:
        self.n_samples = n_samples

    def solve(self, problem: Problem, rng: np.random.Generator) -> tuple[np.ndarray, float]:
        """Return (best_solution, best_objective) using only the Problem protocol."""
        best_sol = problem.random_solution(rng)
        best_obj = problem.evaluate(best_sol)
        for _ in range(self.n_samples - 1):
            sol = problem.random_solution(rng)
            obj = problem.evaluate(sol)
            if obj < best_obj:
                best_sol, best_obj = sol, obj
        return best_sol, best_obj


rng = np.random.default_rng(0)
pts = rng.random((8, 2))
dist = np.linalg.norm(pts[:, None, :] - pts[None, :, :], axis=2)
problem = TSPProblem("rand8", dist)
sol, obj = RandomRestartSearch(n_samples=500).solve(problem, np.random.default_rng(42))
print(sorted(sol.tolist()) == list(range(8)), round(obj, 3))
# Expected: "True <tour length>" — a valid permutation and a length near 2.5-3.0 for 8 unit-square points
```

**Pitfall:** fat interfaces. Adding `neighbors()`, `crossover()`, `plot()`, and `to_mip()` to
the Problem protocol forces every problem to implement methods most algorithms never call, and
soon a "problem" import drags in matplotlib and gurobipy. Keep the shared protocol to what
*every* algorithm needs; operator-specific capabilities (neighborhood moves, decoders) belong
to small extra protocols required only by the algorithms that use them.

## Reproducibility Patterns

### Pattern: seed discipline with SeedSequence spawning

"Seeds everywhere" fails when implemented as `np.random.seed(42)` once at program start: any
change in execution order — a new log line that draws a number, a reordered loop — shifts the
global stream and changes every later result. The discipline: one `Generator` per run, created
from a per-run seed, passed explicitly to everything that draws randomness. Derive per-run
seeds from one campaign seed with `SeedSequence.spawn`, which guarantees independent streams
(better than `base_seed + i`, which correlates generators for some bit patterns).

```python
# src/myopt/seeding.py
import numpy as np


def spawn_run_seeds(base_seed: int, n_runs: int) -> list[int]:
    """Derive independent, reproducible per-run seeds from one campaign seed."""
    seq = np.random.SeedSequence(base_seed)
    return [int(child.generate_state(1)[0]) for child in seq.spawn(n_runs)]


def rng_for(run_seed: int) -> np.random.Generator:
    """The single construction point for all randomness in a run."""
    return np.random.default_rng(run_seed)


seeds = spawn_run_seeds(base_seed=2026, n_runs=4)
print(len(set(seeds)) == 4, spawn_run_seeds(2026, 4) == seeds)
a = rng_for(seeds[0]).integers(0, 100, size=3)
b = rng_for(seeds[0]).integers(0, 100, size=3)
print(np.array_equal(a, b))
# Expected: "True True" then "True" — distinct seeds, stable derivation, reproducible streams
```

**Pitfall:** hidden randomness outside the passed generator. `random.shuffle`, set/dict
iteration under `PYTHONHASHSEED`, legacy `np.random.*` global state, sklearn `random_state`
defaults, and MIP solvers (Gurobi `Seed`, `Threads`) all draw from streams your `Generator`
does not control. Grep for `import random` and `np.random.` outside `seeding.py`; pass `rng`
into every constructor; set and record solver seeds. The determinism test catches regressions.

### Pattern: provenance capture in every record

A result record that cannot answer "which code produced you" is unusable for a paper revision
eight months later. Capture the git commit, Python and key package versions, platform, and a
UTC timestamp at campaign start, and merge that dict into every record. The subprocess and
package lookups are real external boundaries, so the fallbacks below are legitimate.

```python
# src/myopt/provenance.py
import platform
import subprocess
import sys
from datetime import datetime, timezone
from importlib import metadata


def git_commit() -> str:
    """Short commit hash of the repository, '<hash>-dirty' if uncommitted changes exist."""
    try:
        head = subprocess.run(["git", "rev-parse", "--short", "HEAD"],
                              capture_output=True, text=True, check=True).stdout.strip()
        status = subprocess.run(["git", "status", "--porcelain"],
                                capture_output=True, text=True, check=True).stdout.strip()
        return f"{head}-dirty" if status else head
    except (OSError, subprocess.CalledProcessError):
        return "nogit"


def provenance(packages: tuple[str, ...] = ("numpy", "pandas", "gurobipy")) -> dict[str, str]:
    """Environment snapshot merged into every result record."""
    versions: dict[str, str] = {}
    for pkg in packages:
        try:
            versions[f"pkg_{pkg}"] = metadata.version(pkg)
        except metadata.PackageNotFoundError:
            versions[f"pkg_{pkg}"] = "absent"
    return {
        "git_commit": git_commit(),
        "python": sys.version.split()[0],
        "platform": platform.platform(),
        "timestamp_utc": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        **versions,
    }


snap = provenance(packages=("numpy",))
print(sorted(snap))
# Expected: ['git_commit', 'pkg_numpy', 'platform', 'python', 'timestamp_utc'] with real values
```

**Pitfall:** running paper experiments from a dirty working tree. The commit hash is worthless
if uncommitted edits changed the algorithm. The `-dirty` suffix above makes this visible;
stricter projects refuse to start a campaign when the tree is dirty unless the config sets an
explicit `allow_dirty: true` for debugging. Pair this with the tagging workflow in
**git-for-research-code** so each paper table maps to a tag.

## Result Writing and Run Orchestration

### Pattern: atomic result files

A run killed mid-write (Ctrl-C, cluster preemption, out-of-memory) must not leave a truncated
JSON file: half-written files poison aggregation and, worse, make skip-completed logic treat a
broken run as done. The POSIX-and-Windows-safe recipe: write to a temporary file in the *same
directory*, flush and fsync, then `os.replace` onto the final name — the rename is atomic on
one filesystem, so readers see either the old state or the complete new file, never a fragment.

```python
# src/myopt/io/results.py
import json
import os
import tempfile
from pathlib import Path
from typing import Any


def write_json_atomic(record: dict[str, Any], path: Path) -> None:
    """Write a JSON record so that readers can never observe a partial file."""
    path.parent.mkdir(parents=True, exist_ok=True)
    fd, tmp_name = tempfile.mkstemp(dir=path.parent, suffix=".tmp")
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as fh:
            json.dump(record, fh, indent=2)
            fh.flush()
            os.fsync(fh.fileno())
        os.replace(tmp_name, path)
    except BaseException:
        os.unlink(tmp_name)
        raise


record = {"run_id": "demo_s0_abc123def456", "objective": 42.0, "status": "ok"}
with tempfile.TemporaryDirectory() as tmp:
    target = Path(tmp) / "raw" / "demo_s0_abc123def456.json"
    write_json_atomic(record, target)
    print(json.loads(target.read_text(encoding="utf-8"))["objective"])
# Expected: 42.0; a crash before os.replace leaves only a .tmp file, never a partial record
```

**Pitfall:** appending many runs to one shared CSV/JSONL from parallel workers. Concurrent
appends interleave bytes, and a crash mid-append truncates the tail. The robust pattern is one
file per run named by `run_id` (writes never contend), with aggregation into a single table as
a separate, rerunnable step (see **pandas-experiment-management**). If one append-only JSONL is
genuinely required, serialize writers with `filelock` and write each line in a single call.

### Pattern: the campaign runner script

The entry point ties the previous patterns together: load config, iterate instances × seeds,
skip completed runs, isolate per-run failures as recorded error records, write each record
atomically. Note what it does *not* contain: algorithm logic, hard-coded paths, global state.

```python
# scripts/run_experiment.py
"""Run one experiment campaign. Usage: python scripts/run_experiment.py configs/base.yaml"""
from __future__ import annotations

import argparse
import time
import traceback
from pathlib import Path

import numpy as np

from myopt.config import ExperimentConfig, load_config
from myopt.io.instances import load_instance
from myopt.io.results import write_json_atomic
from myopt.provenance import provenance
from myopt.registry import ALGORITHMS
from myopt.run_id import run_id


def run_campaign(cfg: ExperimentConfig) -> None:
    """Loop over instances x seeds; one atomic JSON record per run; resumable."""
    results_dir = Path(cfg.results_dir) / cfg.experiment_name
    env = provenance()
    algo_dict = {"name": cfg.algorithm.name, "params": dict(cfg.algorithm.params)}
    for instance_name in cfg.instances:
        problem = load_instance(instance_name)
        for seed in cfg.seeds:
            rid = run_id(algo_dict, instance_name, seed)
            out_path = results_dir / f"{rid}.json"
            if out_path.exists():
                continue  # resumable: a completed run is identified by its file
            algorithm = ALGORITHMS.create(cfg.algorithm.name, **cfg.algorithm.params)
            rng = np.random.default_rng(seed)
            base = {"run_id": rid, "instance": instance_name, "seed": seed,
                    "algorithm": algo_dict, **env}
            t0 = time.perf_counter()
            try:
                solution, objective = algorithm.solve(problem, rng, cfg.time_limit_s)
                record = {**base, "status": "ok", "objective": float(objective),
                          "solution": np.asarray(solution).tolist(),
                          "runtime_s": time.perf_counter() - t0}
            except Exception:
                record = {**base, "status": "error",
                          "traceback": traceback.format_exc(),
                          "runtime_s": time.perf_counter() - t0}
            write_json_atomic(record, out_path)


def main() -> None:
    """Thin CLI wrapper around the library."""
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("config", type=Path, help="path to a campaign YAML/JSON file")
    args = parser.parse_args()
    run_campaign(load_config(args.config))


if __name__ == "__main__":
    main()
# Expected: results/raw/<experiment_name>/<run_id>.json per run; rerunning skips finished runs
```

**Pitfall:** the broad `except Exception` is correct here and only here. At the per-run
boundary of an unattended thousand-run campaign, recording the traceback and continuing is
right — one bad instance must not kill 900 pending runs, and `status == "error"` rows surface
in aggregation. Inside library code the same construct hides bugs. Keep exactly one such catch,
at the run boundary, and make error records loud in the aggregate report.

### Pattern: the light test tier

Research code does not need 90% coverage; it needs the handful of tests that catch the bugs
that *invalidate experiments*: a config typo silently ignored, a missing registry import, a
nondeterministic run, an objective function that disagrees with brute force on a tiny instance.
These five tests run in about a second, so they actually get run before every campaign.

```python
# tests/test_core.py
"""Fast test tier: every test guards a way an experiment can become invalid."""
import itertools

import numpy as np
import pytest

from myopt.config import load_config
from myopt.registry import ALGORITHMS, PROBLEMS
from myopt.seeding import spawn_run_seeds


def test_config_rejects_unknown_keys(tmp_path) -> None:
    """A typo in a YAML key must crash at load time, not be ignored."""
    bad = tmp_path / "bad.yaml"
    bad.write_text("experiment_name: x\ninstances: []\nseeds: [0]\n"
                   "time_limit_s: 1.0\nalgorithm: {name: ils}\nsedes: [1]\n",
                   encoding="utf-8")
    with pytest.raises(ValueError, match="unknown config keys"):
        load_config(bad)


def test_registry_contains_paper_algorithms() -> None:
    """Guards the empty-registry trap: imports in myopt/__init__.py fill these."""
    for name in ("ils", "tabu", "ga"):
        assert name in ALGORITHMS.names()


def test_same_seed_same_result() -> None:
    """Iteration-capped so the time limit never binds; wall clock must not matter."""
    problem = PROBLEMS.create("random_tsp", n=15, seed=3)
    algo = ALGORITHMS.create("ils", max_iters=200)
    obj_a = algo.solve(problem, np.random.default_rng(7), time_limit_s=60.0)[1]
    obj_b = algo.solve(problem, np.random.default_rng(7), time_limit_s=60.0)[1]
    assert obj_a == obj_b


def test_seed_spawning_is_stable() -> None:
    """Per-run seeds derived from the campaign seed must never change."""
    assert spawn_run_seeds(2026, 3) == spawn_run_seeds(2026, 3)


def test_tiny_instance_matches_brute_force() -> None:
    """On n=6 the heuristic must find the enumerated optimum."""
    problem = PROBLEMS.create("random_tsp", n=6, seed=0)
    algo = ALGORITHMS.create("ils", max_iters=500)
    obj = algo.solve(problem, np.random.default_rng(0), time_limit_s=60.0)[1]
    brute = min(problem.evaluate(np.array(p)) for p in itertools.permutations(range(6)))
    assert obj == pytest.approx(brute)
# Expected: `pytest -q tests/` reports 5 passed in about a second
```

**Pitfall:** time-based assertions. Any test whose pass/fail depends on wall clock (a real
time limit binding, "finishes within 2 s") flakes on loaded machines and CI. Keep the default
tier iteration-capped and tiny-instance only; mark anything slower with `@pytest.mark.slow` and
run it deliberately. Deeper validation — independent feasibility checkers, known-optimum
regression suites — belongs to **solution-validation-testing**.

## Advanced Techniques

### Config sweeps with dotted-path overrides

Parameter studies should be generated, not hand-edited: a base config plus a grid of dotted
keys yields one resolved config per grid point, each hashing to its own `run_id`.

```python
import copy
import itertools
from typing import Any


def expand_grid(base: dict[str, Any], grid: dict[str, list[Any]]) -> list[dict[str, Any]]:
    """One config dict per point of the Cartesian grid of dotted-key overrides."""
    def set_dotted(cfg: dict[str, Any], dotted: str, value: Any) -> None:
        parts = dotted.split(".")
        node = cfg
        for part in parts[:-1]:
            node = node.setdefault(part, {})
        node[parts[-1]] = value

    keys = list(grid)
    configs: list[dict[str, Any]] = []
    for combo in itertools.product(*(grid[k] for k in keys)):
        cfg = copy.deepcopy(base)
        for key, value in zip(keys, combo):
            set_dotted(cfg, key, value)
        configs.append(cfg)
    return configs


base = {"algorithm": {"name": "ils", "params": {"kick_strength": 3}}, "time_limit_s": 10.0}
grid = {"algorithm.params.kick_strength": [2, 4, 8], "time_limit_s": [10.0, 60.0]}
sweep = expand_grid(base, grid)
print(len(sweep), sweep[0]["algorithm"]["params"]["kick_strength"], sweep[-1]["time_limit_s"])
# Expected: "6 2 60.0" — the full 3x2 grid with nested overrides applied
```

Grids explode combinatorially; for more than ~3 swept parameters, replace grid search with a
tuning study (**optuna-hyperparameter-tuning**) and keep `expand_grid` for the final 2-3
sensitivity sweeps reported in the paper.

### Determinism beyond the seed

A recorded seed is necessary, not sufficient. Remaining nondeterminism sources, by frequency:
solver threading (set Gurobi `Threads=1` and `Seed` explicitly for paper runs; record both),
BLAS threading in numpy reductions (`OMP_NUM_THREADS=1` for bit-exact reruns),
`PYTHONHASHSEED` changing `set`/`dict`-of-object iteration order (sort before iterating), and
wall-clock termination (iterations reached in 10 s vary across machines — record iterations
performed and prefer evaluation budgets when comparing algorithms). State in the README which
level the project guarantees: statistical reproducibility (same distribution) is honest and
often sufficient; bit-exact reproducibility costs the thread restrictions above.

### Packaging with pyproject.toml and editable installs

`sys.path.append("..")` at the top of scripts breaks the moment a script moves or runs from
another directory. A minimal package definition fixes imports everywhere — scripts, tests,
notebooks — with one `pip install -e .`:

```toml
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"

[project]
name = "myopt"
version = "0.1.0"
requires-python = ">=3.10"
dependencies = ["numpy>=1.26", "pyyaml>=6.0", "pandas>=2.2"]

[project.optional-dependencies]
dev = ["pytest>=8.0"]

[tool.setuptools.packages.find]
where = ["src"]
```

The `src/` layout (importable only after installation) makes tests exercise the installed
package, so an import that "worked from the repo root" fails before a cluster run does. Pin
exact versions in a lock file at submission time and record it next to the tagged results.

### Immutable raw results, regenerable derivatives

Treat `results/raw/` as append-only: no script modifies or deletes a raw record; renaming an
experiment means a new `experiment_name`, not editing files. Everything in `results/tables/`
and `results/figures/` must be regenerable by rerunning `aggregate_results.py` and
`make_figures.py`. A plotting bug then never forces rerunning a week of solves, and a reviewer
request ("medians instead of means") is a 30-second aggregation rerun. Archive published raw
records outside the working tree, keyed by the paper's git tag.

### Run manifests for large campaigns

Beyond roughly 10^4 runs, globbing JSON files to find pending work becomes slow and opaque.
Generate the run list up front — `scripts/plan_campaign.py` expands configs × instances × seeds
into a manifest parquet with one row per `run_id` and a status column — then workers claim rows
and the aggregator joins results against the manifest. One pandas query then answers "what is
missing, what failed, what is running" instead of directory archaeology.

## Practical Challenges

**The logic lives in notebooks and every experiment is a manual cell sequence.** Migrate in
one direction only: move functions (not cells) into `src/myopt/` modules, import them back into
the notebook, and re-verify outputs after each move. Nothing a result depends on may exist
*only* in a notebook; notebooks stay useful for exploration and figure prototyping, calling the
same library and reading the same `results/raw/` records as the scripts.

**A re-run silently overwrites last month's results.** Content-addressed `run_id`s prevent
collisions between *different* experiments, but rerunning the same config targets the same
files. Make raw records write-once: the runner skips existing files, and an explicit
`--force-rerun` flag that first moves old records to `results/raw/_superseded/<date>/` is the
only path to replacement — never destroy the evidence a submitted table was built on.

**Results exist but nobody knows which code produced them.** This is fatal at revision time.
Provenance in every record (commit hash, dirty flag, package versions) plus the tag-per-table
workflow of **git-for-research-code** solves it forward; for orphaned legacy results, the only
honest fix is rerunning them under the new regime — treat unprovenanced numbers as lost.

**`np.random.seed` deep inside a module makes results depend on call order.** Symptom: results
change when an unrelated feature is added, or when two algorithms run in one process in a
different order. Grep for `np.random.seed`, `np.random.<draw>`, and bare `import random`;
replace every hit with an explicit `Generator` parameter threaded from the runner. Add a
two-algorithm variant of the determinism test if this bites once.

**Defaults in code disagree with the config file.** A config omits `tenure`, a refactor
silently changed the constructor default from 8 to 12, and three weeks of "tenure=8" results
are actually mixed. Record resolved parameters (read back from the constructed object) in every
record, and let aggregation group by *recorded* parameters, never by what the config claims.

**Absolute paths break on the cluster.** `C:/Users/.../instances/` or `/home/alice/...` in
code or configs guarantees failure on the second machine. Config paths are relative to the
repository root; the runner resolves them against `Path(__file__).resolve().parents[1]` or an
explicit `--data-root` argument, and nothing else is machine-specific.

**Run 900 of 1000 crashes and the campaign dies overnight.** Two structural fixes, both in the
runner pattern: per-run exception capture writes an error record and continues, and
skip-completed logic makes a restart cost nothing. For cluster array jobs, the manifest pattern
lets each task claim disjoint rows so a single failed task is re-submittable in isolation.

**Two runs share mutable state through a cached instance object.** An algorithm "repairs" the
distance matrix in place or appends to `problem.history`, and every later run on that instance
sees modified data. Rules: problems are immutable after construction (`arr.flags.writeable =
False` makes violations crash loudly), algorithms own their mutable state, and the runner
constructs a fresh algorithm object per run — reuse across runs saves nothing.

**The registry is empty in the cluster job but fine on the laptop.** Locally the notebook
imported `myopt.algorithms.ils` at some point; the cluster entry point did not. The fix is the
explicit import block in `myopt/__init__.py` plus `test_registry_contains_paper_algorithms`,
which turns a KeyError at hour three into a red test before submission.

**Tests are slow, so they stop being run.** A suite that solves real instances takes minutes
and gets skipped exactly when it matters. Enforce the two-tier split: the default tier (config,
registry, determinism, tiny brute-force) stays under ~2 s; everything touching real instances
or real time limits is `@pytest.mark.slow`. Run the fast tier before every campaign launch.

## Tools & Libraries

| Library / tool | When to use | Note |
|---|---|---|
| `dataclasses` (stdlib) | Config schemas, result records | Frozen dataclasses + a strict builder cover most projects; zero dependencies |
| `pydantic` | Configs needing coercion/validation | Heavier; worth it when configs come from many hands or web forms |
| `PyYAML` (`safe_load`) | Human-edited campaign configs | Always `safe_load`; YAML 1.1 quirk: unquoted `no`/`yes` parse as booleans |
| `json` (stdlib) | Machine-generated configs, result records | Canonical dumps (sorted keys) for hashing; JSONL for traces |
| `argparse` (stdlib) | Script CLIs | Sufficient for thin entry points; resist framework creep |
| `hydra` / `omegaconf` | Many-layered config composition | Powerful overrides and sweeps; adds magic — adopt only when plain YAML hurts |
| `pathlib` (stdlib) | All filesystem paths | Cross-platform; never concatenate path strings |
| `pytest` | The light test tier | `tmp_path` fixture, `pytest.approx`, markers for the slow tier |
| `filelock` | Shared-file writes from parallel workers | Only needed if the one-file-per-run pattern is abandoned |
| `pandas` + `pyarrow` | Aggregating raw records into tables | Parquet for big traces; see pandas-experiment-management |
| `pip` + `venv` / `uv` | Environment per project | Export a lock file at submission time; record it with the tag |

## Output Format

A structured-project deliverable consists of checklists and templates, not prose. Provide:

**New-project checklist:**

- [ ] `pyproject.toml` with `src/` layout; `pip install -e .[dev]` succeeds in a fresh venv
- [ ] Directory skeleton: `configs/`, `data/instances/`, `results/{raw,tables,figures}/`, `scripts/`, `src/`, `tests/`
- [ ] `.gitignore` covers `results/`, solver logs, `*.tmp`, venv, `__pycache__`
- [ ] `Registry` for algorithms and problems; explicit imports in `myopt/__init__.py`
- [ ] `seeding.py` is the only module that constructs generators; no `np.random.seed` anywhere
- [ ] `write_json_atomic` is the only way results reach disk
- [ ] Fast test tier passes: config rejection, registry contents, determinism, tiny brute-force
- [ ] README: install, run one campaign, run tests — three commands, copy-pasteable

**Campaign config template (`configs/base.yaml`):**

```yaml
experiment_name: ils_baseline_v1     # becomes the results subdirectory
instances: [rand50_a, rand50_b, rand100_a]
algorithm:
  name: ils                          # must be a registered name
  params: {kick_strength: 4, max_iters: 20000}
seeds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
time_limit_s: 60.0
results_dir: results/raw
```

**Per-run record field checklist** (every record carries all of these):

- [ ] `run_id`, `instance`, `seed`, `algorithm` (name + resolved params)
- [ ] `status` (`ok`/`error`), `objective`, `solution`, `runtime_s` (+ iterations performed)
- [ ] Provenance: `git_commit` (with dirty flag), `python`, `platform`, package versions, `timestamp_utc`

**Pre-submission reproducibility checklist:**

- [ ] Working tree clean; campaign commit tagged (e.g. `paper-table3-v1`)
- [ ] Solver `Seed`/`Threads` recorded; lock file exported next to the tag
- [ ] Error-status records counted and reported, not silently dropped
- [ ] `aggregate_results.py` + `make_figures.py` regenerate every table and figure from `results/raw/` alone
- [ ] Raw records of published numbers archived outside the working tree

## Questions to Ask

- How many algorithms, problems, instances, and seeds will be combined — what is the total run count?
- Will any of these results appear in a paper or thesis, and which tables/figures are needed?
- What machines run this — laptop only, or also a cluster or a colleague's environment?
- How long is one run, and can a thousand-run campaign survive a crash at run 900?
- Where does randomness enter (moves, instance generation, solver), and is each source seeded and recorded?
- Is there existing code to migrate — notebooks, a flat script, or nothing yet?
- Do configs need composition and sweeps, or is one YAML per campaign enough?
- What result volume is expected — scalar records, full solutions, or per-iteration traces?

## Related Skills

- **pandas-experiment-management** — when the per-run JSON records must become tidy result tables, aggregations over instances and seeds, and pivot tables for the paper.
- **git-for-research-code** — when commits, tags, and .gitignore rules must link every result table to the exact code version that produced it.
- **solution-validation-testing** — when the light test tier should grow into independent feasibility checkers, objective recomputation, and known-optimum regression tests.
- **instance-generation-and-benchmarks** — when `data/instances/` needs standard benchmark parsers or seeded synthetic generators with controlled hardness.
- **optuna-hyperparameter-tuning** — when hand-edited parameter sweeps should become a proper tuning study fed by the same config and seeding system.