Validation And Experiments

---
name: validation-and-experiments
description: >-
  Design the smallest falsifying test loop for an algorithmic solver, including
  brute oracles, differential and metamorphic tests, champion/challenger
  comparisons, paired seeds, holdouts, resource probes, and release gates. Use
  as an internal algorithmic-problem-solving recovery route when the smallest
  counterexample is unclear, comparisons are noisy, or a correction needs
  independent evidence before promotion. Do not use to implement a doubtful
  checker or to diagnose a known compiler or runtime-engineering failure.
---

# Validation and Experiments

Use the smallest test loop that can falsify the current algorithmic hypothesis. Do not begin with a generic benchmark platform or a large campaign.

## Ownership and handoffs

This skill owns oracle choice, counterexample search, controlled candidate comparison, stochastic evidence, and promotion/release gates. Use:

- [checker and local evaluation](../checker-and-local-evaluation/SKILL.md) to implement or repair an independent checker, scorer, interactor, simulator, or process harness;
- [model and route algorithms](../model-and-route-algorithms/SKILL.md) when evidence contradicts the contract, state, reduction, proof, or route class;
- [contest solver engineering](../contest-solver-engineering/SKILL.md) for compiler, overflow, memory-layout, I/O, or measured deadline failures;
- the shared [heuristic search](../../references/heuristic-search.md) for representation, moves, deltas, reachability, and optimizer mechanics.

Do not start a local generated-data campaign merely because a candidate is non-full or because a local harness could be built. If accepted online submissions provide stable legality/protocol evidence and coherent scores, defer local construction while the unresolved question is algorithmic quality. Start local testing when a named hypothesis needs a counterexample, official feedback cannot isolate the failure, paired evidence is needed for promotion, a rare/resource boundary must be probed, or Checker and Local Evaluation has been evidence-triggered.

When generated inputs are justified, this skill chooses the coverage families, oracle relationships, development/holdout split, and comparison design. Execute them through [Checker and Local Evaluation's end-to-end data workflow](../checker-and-local-evaluation/SKILL.md#build-problem-finding-data-end-to-end): parameterized generator code, independent input validation, solver/evaluator execution, and a reproducible seed/failure manifest. Pair with [testlib C++ judging](../testlib-cpp-judging/SKILL.md) when using Testlib generators or validators. Randomized, batch, and maximum-scale inputs must be generated by code rather than maintained as hand-edited data.

## 1. Validation ladder

Build evidence in this order:

1. target-language compile with the official flags;
2. samples and hand-computed examples;
3. empty/minimal, maximum, equality, duplicate, and overflow boundaries;
4. one case for each hard constraint and failure mode;
5. brute or independent oracle on tiny instances;
6. randomized or metamorphic tests;
7. representative runtime and memory probes;
8. official evaluator or legal remote feedback.

Stop and fix the first failing layer before running a larger campaign.

## 2. Exact and constructive problems

For an exact solver, prefer a tiny obviously correct oracle over a second optimized implementation. Generate cases small enough for full enumeration and compare the complete answer or a canonical witness.

Check:

- empty/nonempty distinctions and impossible cases;
- equality, duplicates, disconnected components, and symmetric solutions;
- reconstruction independently from the optimum value;
- overflow before narrowing and exact division/modulo semantics;
- dense and sparse extremes;
- the first size where an optimization changes behavior.

For constructive output, validate syntax, hard constraints, and final serialization independently. A promised-feasible input does not prove the construction reaches a valid witness.

When a mismatch appears, minimize it and identify the first violated model or proof premise before changing downstream implementation details.

## 3. Scored solvers

Always keep:

```
fallback: simplest guaranteed-valid emergency output
challenger/current: mutable experimental candidate
champion: best independently validated legal artifact
```

Before using a routed local campaign to tune quality or promote a challenger:

- validate every output;
- recompute the true raw objective;
- compare incremental deltas with full recomputation;
- test apply/undo round trips;
- confirm score direction and any clamp, threshold, or normalization;
- verify the internal deadline includes a safety margin.

Use Checker and Local Evaluation when the official checker/scorer is absent or doubtful and that uncertainty blocks the current decision. Do not compare solver quality under an evaluator whose contract is still contradicted by official or hand-computed evidence.

## 4. Minimal champion/challenger loop

Record one line per material attempt:

```
champion | hypothesis | one change | fixed cases/seeds
validity | per-case score/runtime delta | keep/revert | next action
```

Use a direct shell command or a short task-specific script. Default to:

- a few representative development cases;
- the same small set of solver seeds for champion and challenger;
- one untouched boundary or holdout set;
- saved failing inputs and outputs;
- the full generator parameter tuple, instance seed, validator result, and input hash for every generated case;
- per-case results, not only an aggregate.

Promote only when the challenger stays legal, fits the budget with margin, and improves the relevant fixed evidence. Revert the challenger otherwise.

## 5. Randomness and holdouts

Keep randomness sources distinct:

```
instance seed | solver seed | judge/noise seed | host/runtime noise
```

Pair champion and challenger on the same inputs and allowed noise seeds. Tune on development cases, then check an untouched set. Do not repeatedly select against one public set or one lucky solver seed.

Three to five fixed seeds are usually enough to reject a clearly weak idea. Expand only when results are close, variance is material, or remote scoring is noisy.

## 6. Metamorphic tests

When an exact oracle is hard, transform an instance in a way with a known effect:

- permute labels or input order when semantics are invariant;
- add an isolated or dominated object with predictable behavior;
- scale or translate geometry when the contract preserves it;
- duplicate independent components and combine their objectives;
- reverse, rotate, or complement only when the contract has that symmetry.

Verify the expected relationship, not necessarily a fixed output string.

## 7. Interactive and reactive evaluator validation

Only when Checker and Local Evaluation has been evidence-triggered or a Plateau Escape phase-local gate is active, use it to build and self-test the required process-level simulator, interactor, reactive episode replayer, scorer, real-pipe runner, or transcript contract. That sub-skill owns evaluator implementation, the C++ implementation policy, adversarial fixtures, and the route to concrete `testlib.h` APIs. If accepted official interactions are stable and coherent, defer these local artifacts rather than entering Checker through this section alone.

This skill owns how those evaluator artifacts enter the broader experiment loop. Pin their build/run commands and evaluator version, compare policies or strategies on the same allowed hidden states and noise seeds, retain failing transcripts, and report protocol failure separately from legality, reward, runtime, and strategy quality. Do not promote a solver merely because it performs well under one uncorroborated local simulator.

## 8. When heavier evidence is justified

Use larger seed sets, quantiles, confidence intervals, or formal benchmark automation only when:

- stochastic variance can reverse the decision;
- candidates differ by a small margin;
- failures have expensive lower-tail risk;
- the work is a long-running solver project rather than one problem;
- a formal report requires reproducible aggregate statistics.

Even then, begin with paired per-case deltas and explicit invalid/timeout counts. Do not let statistical machinery hide contract or checker errors.

## 9. Failure-directed next action

| Symptom | Next route or check |
|---|---|
| Invalid output | Parser, bounds, reconstruction, checker assumptions |
| Exact mismatch | Model-and-route: smallest brute counterexample and proof premise |
| TLE/MLE | Contest-solver-engineering: measured hot path, state count, copying, allocation |
| Legal but very low score | Objective direction, external reference/bound gap, representation, construction |
| Local search plateau | Delta correctness, reachability, compound moves |
| Seed-sensitive gain | Per-instance deltas, outliers, overfit |
| Local/remote disagreement | Checker contract and score transform |

If the current route appears structurally weak but the admission criteria in [plateau escape](../plateau-escape/SKILL.md) are not yet met, test one falsifiable structural challenger. Once they are met, use that sub-skill before choosing another model, representation, or algorithm family. Do not answer every failure with another parameter or bypass the plateau gate by expanding the test campaign.

## 10. Release gate

Before delivery:

- rebuild the champion cleanly;
- rerun samples and saved regressions;
- rerun the smallest independent oracle/checker suite already justified or available; do not build a new local campaign solely for release when coherent accepted official evidence leaves no evaluator-facing uncertainty;
- validate the final output or transcript;
- probe worst or representative runtime and memory;
- confirm deterministic seeds and release flags;
- ensure the delivered artifact is the champion, not the latest edit.

Return the falsified or supported hypothesis, exact commands and fixtures used, per-case evidence, promotion/rejection decision, and remaining material uncertainty.