Dueling Autoresearch

---
name: dueling-autoresearch
description: >
  Use when the user wants two approaches raced head-to-head on a single shared metric — e.g.
  a classical/algorithmic lane vs an ML/learned lane, or any two strategies for the same task.
  Each lane runs its own analysis-first research loop confined to its lane, the lanes share a
  scoreboard and may borrow ideas across the boundary without abandoning their identity, and a
  shared eval keeps the head-to-head honest; loops until interrupted, reporting the current
  leader. Not for improving a single approach in isolation (use a single-track research loop),
  and not for picking between two finished artifacts in one shot (that is a one-time comparison).
compatibility: Requires Python 3.9+
metadata:
  version: "0.1.0"
---

# Dueling Autoresearch Loop

Two lanes work the **same** objective in parallel and **race the same metric** — by default a
**classical/algorithmic** lane against an **ML/learned** lane (the lanes are user-named). Each lane
runs its own analysis-first iteration via `roles/TrackAgent.md`, **confined to its lane**. Every
round both lanes post to a shared `duel_log.md` scoreboard and may **borrow ideas** across the lane
boundary — but each stays in its lane. The feedback signal is the shared `<metric>` on a shared
eval: if the classical lane wins, that is a real result. Lanes support **mixed code locations** — a
`codebase` lane edits existing repo files, a `sandbox` lane authors its own code — and an
**eval-parity gate** keeps the scores comparable.

You are the orchestrator: each round you advance both lanes, update the scoreboard, and keep both
honest. Do not pause for permission once the loop is running.

## When to use

Use this to race two genuinely different approaches on one metric and keep them honest against the
same eval — classical vs learned, two model families, two query strategies. Default to spawning both
lanes in parallel and letting the scoreboard drive cross-lane idea borrowing; if a lane runs dry,
push it to a more radical in-lane change or to borrow a fresh idea from the log. Not for tuning a
single approach (use a single-track loop), and not for a one-shot comparison of two finished things.

The cast (all in this folder):
- `roles/TrackAgent.md` — the per-lane researcher, instantiated once per lane.

## Setup

Resolve bindings interactively. If `loop.run.yaml` exists in the working dir, load it, confirm the
values in one line, and skip to the loop. Otherwise: on Claude Code (the `AskUserQuestion` tool is
available) infer a likely value for each binding and present it as the recommended option; on other
hosts ask each as a quoted plain-text prompt. Then write `loop.run.yaml` (format:
`examples/run.example.yaml`) and confirm the values before creating any other files.

The host also decides **spawn-or-degrade**: on Claude Code spawn a real `Agent` per lane so the two
run **in parallel**; otherwise adopt `roles/TrackAgent.md` inline and run the lanes sequentially.

**Shared bindings** (identical for both lanes — the honesty anchor):

| binding | meaning | default | how to infer |
|---|---|---|---|
| `<metric>` | the single metric both lanes race; ground truth of the duel | — | ask; scan run logs for a printed score |
| `<metric_direction>` | `minimize` or `maximize` | — | from the metric's nature (loss vs accuracy) |
| `<gate>` | run budget unit: `time` or `epochs` | `epochs` | the artifact's runner |
| `<budget>` | epochs per run (or minutes if `gate: time`) | 5 | — |
| `<sandbox_root>` | where snapshots, ledgers, and the duel log live | `./sandbox` | — |
| `<iter_strategy>` | `snapshots` or `branches` (snapshots recommended — two lanes on one branch is simplest) | `snapshots` | — |

**Per-lane bindings** (two lanes, default names `classical` and `learned`). Each lane has a
`code_location` that decides which other fields it needs — **never add code to the codebase**:

| field | meaning | when |
|---|---|---|
| `name` | lane name | always |
| `code_location` | `codebase` or `sandbox` | always |
| `run_cmd` | existing entrypoint to run, e.g. `python train.py` | if `codebase` |
| `editable_files` | existing repo files this lane may edit | if `codebase` |
| `entry` | command run from inside `<sandbox_root>/<lane>/iter<N>/`, e.g. `python run.py` | if `sandbox` |

- **`codebase`** — the lane maps to existing code: edits its `editable_files` and runs `run_cmd`.
  Two `codebase` lanes must have **non-overlapping** `editable_files`.
- **`sandbox`** — no implementation exists and none is added to the repo: the lane **authors and runs
  its code inside `<sandbox_root>/<lane>/iter<N>/`** via `entry`.

> **Typical duel on a repo with one existing model:** the **learned** lane is `codebase` (edits
> `model.py`/`config.yaml`, runs `train.py`); the **classical** lane is `sandbox` (authors its own
> code under `<sandbox_root>/classical/iter<N>/`). Nothing is added to the codebase, yet the
> classical lane is still built and iterated.

**Eval-parity gate (the honesty anchor).** Before starting, confirm **both lanes report `<metric>`
on the same held-out set, computed the same way**, so the scores are comparable — state how each lane
emits it (e.g. both print `<metric>:` to their run log). If they don't match, fix it first; the duel
is meaningless otherwise. If `gate: time`, write a `run_with_timeout.sh` wrapper per lane
(`timeout $(( <budget> * 60 )) <entry-or-run_cmd> "$@"`).

**Initialise the sandbox** (after confirmation):
```
<sandbox_root>/
├── duel_log.md          ← shared channel + scoreboard (## Scoreboard, ## Round log; headers only)
├── <laneA>/results.tsv  ← lane A ledger, header only
└── <laneB>/results.tsv  ← lane B ledger, header only
```
Each lane's per-iteration work lives in `<sandbox_root>/<lane>/iter<N>/` (`analysis/`, `results/`,
the run log). A `codebase` lane's iter dir also holds `code_snapshot/` (the pre-change copy for
revert); a `sandbox` lane's iter dir holds the lane's **actual code** for that iteration (a kept
iteration carries forward as the next one's starting point).

## The loop (duel)

Each **round** advances **both** lanes by one iteration. On Claude Code, spawn the two TrackAgents in
**parallel** (one turn, two `Agent` calls); otherwise run lane A then lane B inline. A track is one
analysis-first iteration confined to its lane — the 8 steps in `roles/TrackAgent.md`. **Round 1** is
each lane's baseline (a `codebase` lane runs unmodified; a `sandbox` lane authors its initial
implementation in `iter1/`). **One change per lane per round**, so each metric delta is attributable.

Copy this checklist and tick items off each round:
- [ ] **State** — note round N; read `duel_log.md` (both lanes' latest posts + the scoreboard).
- [ ] **Advance each lane** — run a TrackAgent (`roles/TrackAgent.md`) per lane, given its lane
      bindings, the shared `<metric>`/`<metric_direction>`/`<gate>`/`<budget>`, and `duel_log.md`.
- [ ] **Track posts** — each lane appends its round entry to `duel_log.md` (best `<metric>`, one key
      finding, any dead end, one idea the other lane could borrow).
- [ ] **Scoreboard** — update `## Scoreboard`: best `<metric>` per lane and the current leader (per
      `<metric_direction>`); optionally flag one cross-pollination suggestion for next round.
- [ ] **Continue** — go to the next round; never pause to ask whether to continue.

**Spawn-or-degrade per lane.** Where the host supports it, spawn a real isolated TrackAgent per lane
(Claude Code: an `Agent` per lane, both launched in one turn for parallelism). Otherwise adopt
`roles/TrackAgent.md` inline and run the lanes sequentially. Each TrackAgent is confined to its lane
and returns its iteration summary to the orchestrator.

## Ledger

Two ledgers: a **per-lane `results.tsv`** for each lane's experiments, and the **shared
`duel_log.md`** scoreboard + round posts.

**Per-lane `<sandbox_root>/<lane>/results.tsv`** (tab-separated, never commas in free text):
```
iter	<metric>	status	analysis_summary	description
1	0.6320	keep	baseline; classical features, logistic head	baseline
2	0.6610	keep	added HOG features; per-class gains on textured classes	add HOG feature extractor
```
`status` ∈ {`keep`, `discard`, `crash`} (`0.000000` for `<metric>` on crash).

**Shared `<sandbox_root>/duel_log.md`** — scoreboard + per-round posts:
```
## Scoreboard
round	classical_best	learned_best	leader
1	0.6320	0.6480	learned
2	0.6610	0.7050	learned

## Round log
### Round 2
- **classical** — best 0.6610 (this iter 0.6610). Finding: HOG helps textured classes
  (results/per_class.txt). Dead end: raw-pixel kNN plateaus. Borrow: learned's augmentation
  could expand classical's training set.
- **learned** — best 0.7050. Finding: BN fixed conv2 saturation. Dead end: dropout hurt at this
  budget. Borrow: classical's HOG features as an aux input channel.
```
Report the **current leader** (per `<metric_direction>`), never a final winner — a lane that is
behind can still come back. Leave `results.tsv`, `duel_log.md`, and `iter*/` untracked (do not
commit them).

## Constraints
- **Never add code to the codebase.** A `codebase` lane edits only its own `<editable_files>`; a
  `sandbox` lane lives entirely in `<sandbox_root>/<lane>/`. Lanes never touch each other's files —
  every other file is the evaluation ground truth.
- **Stay in lane.** A lane borrows *ideas*, never converts into the other approach — a classical lane
  stays classical even if it borrows a loss/target idea from the learned lane.
- **Same metric, same eval.** Both lanes optimize `<metric>` on the same held-out set computed the
  same way; never compare otherwise. Do not modify the evaluation/metric — it is the shared ground
  truth that makes the duel honest.
- **One change per lane per round**, so each metric delta is attributable.
- Do not install new packages or add dependencies the project lacks; helper code stays stdlib-only.
- Redirect each run's output to its lane's run log; never use `tee`. The sandbox is self-contained —
  no `../` escapes.
- Do not pause the loop to ask for direction; once running, keep both lanes iterating until manually
  interrupted, and report the current leader rather than declaring a final winner.