Inno Experiment Dev

---
name: inno-experiment-dev
description: Creates implementation plan, writes project code with judge feedback loop, and submits final experiment run. Use after code-survey in both Idea and Plan branches.
---

# Inno Experiment Dev (Planning, Implementation, and Submission)

Merges the former `inno-implementation-plan`, `inno-ml-dev-iteration`, and the submit step of `inno-experiment-submit-refine`. Mirrors `_create_implementation_plan` (830-858), `_implement_and_iterate` (861-920), and the submit portion of `_submit_and_refine_experiments` (922-945) in `run_infer_idea_ours.py`.

## Inputs

| Variable | Source | Description |
|----------|--------|-------------|
| `survey_res` | inno-idea-generation or user | The finalized selected idea (or `refined_for_downstream`) |
| `references` | pipeline config | Pre-formatted string of source papers |
| `updated_prepare_res` | inno-prepare-resources | JSON with `reference_codebases` and `reference_paths` |
| `code_survey_res` | inno-code-survey | Comprehensive implementation report / model survey notes |
| `dataset_description` | from prepare step / context | Description of available datasets (not in instance.json) |
| `core_code` | instance.json `Experiment.core_code` | Absolute path when created by Dr. Claw (e.g. `<project_path>/Experiment/core_code`); use as-is or resolve with `path.join(project_path, value)` if relative |
| `code_references` | instance.json `Experiment.code_references` | Absolute path when created by Dr. Claw (e.g. `<project_path>/Experiment/code_references`); use as-is or resolve if relative |
| `max_iter_times` | pipeline config | Max judge-iteration rounds (default 2) |
| `context_variables` | shared state | Mutable dict carrying state across agents |

Plan mode additionally uses `ideas` and survey-specific prompt variants (`build_plan_query_with_survey`, `build_iteration_query_for_plan`, etc.).

## Outputs

| Variable | Description |
|----------|-------------|
| `plan_res` | Detailed implementation plan with dataset, model, training, and testing sections |
| `ml_dev_res` | Final ML Agent implementation result |
| `judge_res` | Final Judge Agent feedback |
| `judge_messages` | Full conversation thread (preserved for inno-experiment-analysis) |
| `submit_res` | Experiment submission result with statistical outputs |
| `context_variables` | Updated with `dataset_plan`, `training_plan`, `testing_plan`, `suggestion_dict`, `raw_error_stats` |

## Cache Artifacts

| File | Agent | Content |
|------|-------|---------|
| `Experiment/core_code/logs/coding_plan_agent.json` | Coding Plan Agent | `context_variables` + `messages` from planning phase |
| `Experiment/core_code/logs/machine_learning_agent.json` | ML Agent | Initial implementation messages (+ `_iter_{N}.json` for judge iterations) |
| `Experiment/core_code/logs/judge_agent.json` | Judge Agent | Evaluation messages (+ `_iter_{N}.json` for iterations) |
| `Experiment/core_code/logs/machine_learning_agent_iter_submit.json` | ML Agent | Submission run messages and results |

## Instructions

### Phase 1: Create Implementation Plan

Mirrors `_create_implementation_plan`.

1. **Optional pre-step (Idea mode only)**: If refining the idea for implementation clarity, call the idea refinement agent to produce `refined_for_downstream` with tensor interfaces and forward-pass sketch.

2. **Build plan query**:
   - **Idea mode**: `plan_query = build_plan_query(survey_res, references, updated_prepare_res, code_survey_res, dataset_description)` (see `prompts/build_plan_query.md`)
   - **Plan mode**: Use `build_plan_query_with_survey(ideas, references, prepare_res, code_survey_res, dataset_description)`

3. **Call Coding Plan Agent** with `messages = [{"role": "user", "content": plan_query}]`.
   - The agent reviews codebases using `tree` / `cat`, then creates structured plans via `plan_dataset`, `plan_training`, `plan_testing`.
   - Calls `case_resolved` to merge plans.
   - Set `plan_res = plan_messages[-1]["content"]`.
   - See `references/coding_plan_agent.md` for agent details.

4. **Verify** the plan has clear sections: dataset, model, training, evaluation, file layout.

### Phase 2: Implement and Iterate

Mirrors `_implement_and_iterate`.

5. **Initial implementation**: Build `ml_dev_query = build_ml_dev_query(survey_res, prepare_res, code_survey_res, plan_res, dataset_description, core_code, code_references)` (see `prompts/build_ml_dev_query.md`). Use paths from `instance.json`: `Experiment.core_code`, `Experiment.code_references` (absolute in Dr. Claw–created projects; use as-is or resolve with project path if relative). Call **ML Agent** with `messages = [{"role": "user", "content": ml_dev_query}]`. Set `ml_dev_res = ml_messages[-1]["content"]`.
   - See `references/ml_agent_instructions.md` for agent details.

6. **Initial judge evaluation**: Build `judge_query = build_judge_query(survey_res, prepare_res, plan_res, ml_dev_res)` (see `prompts/build_judge_query.md`). Call **Judge Agent** with `input_messages = [{"role": "user", "content": judge_query}]`. Set `judge_res = judge_messages[-1]["content"]`.
   - See `references/judge_agent_instructions.md` for agent details.

7. **Iteration loop** (for i in 0..max_iter_times - 1):
   a. Build `iteration_query = build_iteration_query(survey_res, prepare_res, code_survey_res, plan_res, ml_dev_res, judge_res, core_code, code_references)` (see `prompts/build_iteration_query.md`). Use paths from instance.json (absolute in Dr. Claw–created projects; use as-is or resolve if relative). Plan mode uses `build_iteration_query_for_plan`.
   b. Append as user message to `judge_messages`. Call **ML Agent** with `iter_times=i+1`. Update `ml_dev_res`.
   c. Build `judge_simple_query = build_judge_simple_query(survey_res, prepare_res, plan_res, ml_dev_res)` (see `prompts/build_judge_simple_query.md`). Plan mode uses `build_judge_simple_query_for_plan`.
   d. Append as user message to `judge_messages`. Call **Judge Agent** with `iter_times=i+1`. Update `judge_res`.
   e. If `"fully_correct": true` in last message, **break early**.

8. Preserve `judge_messages` for the submit step and for downstream `inno-experiment-analysis`.

### Phase 3: Submit Experiment

Mirrors the submit portion of `_submit_and_refine_experiments`.

9. **Build submit query**: `submit_query = build_submit_query(survey_res, ml_dev_res, judge_res, core_code)` (see `prompts/build_submit_query.md`). Resolve `core_code` from `instance.Experiment.core_code`. Plan mode uses `build_submit_query_for_plan`.

10. **Append** to `judge_messages` as user message. Call **ML Agent** with `iter_times="submit"`.
    - The agent adjusts epochs (3-10), runs `run_training_testing.py`, ensures checkpoints are saved.
    - Set `submit_res = judge_messages[-1]["content"]`.

11. If the implementation is not runnable, ML Agent calls `case_not_resolved`. Otherwise, `case_resolved` with statistical results and analysis.

## Tool Mappings

All custom Python tools map to Claude Code built-in capabilities:

| Original Tool | Claude Code Equivalent |
|---------------|----------------------|
| `execute_command` | Shell tool (direct execution) |
| `run_python` | `python <script>` via Shell tool |
| `create_file` / `write_file` | Write tool |
| `read_file` | Read tool or `cat <path>` |
| `create_directory` | `mkdir -p <path>` |
| `list_files` | `ls <path>` |
| `gen_code_tree_structure` | `tree -L 3 <path>` |
| `diagnose_code_error` | Analyze stderr output + inspect code |
| `rollback_and_reimplement` | Re-write file with different approach |
| `view_error_history` | Track error fingerprints in agent memory |
| `plan_dataset` / `plan_training` / `plan_testing` | Structure plan sections in agent response |
| `case_resolved` / `case_not_resolved` | Agent returns result / failure reason |

## Checklist

- [ ] Optional idea refinement applied if desired (Idea mode).
- [ ] Correct `build_plan_query` variant used for Idea vs Plan mode.
- [ ] Coding Plan Agent called; `plan_res` has clear dataset/model/training/testing sections.
- [ ] ML Agent initial implementation completed; `ml_dev_res` recorded.
- [ ] Judge Agent initial evaluation completed; `judge_res` recorded.
- [ ] Iteration loop runs with correct prompt variants; early exit on `fully_correct`.
- [ ] `judge_messages` preserved across all phases.
- [ ] Submit query appended to `judge_messages`; ML Agent submission run completed.
- [ ] Final model checkpoint saved to `Experiment/core_code/checkpoints/model_final.pth`.
- [ ] Cache artifacts saved to `Experiment/core_code/logs/`: `coding_plan_agent.json`, `machine_learning_agent.json`, `judge_agent.json`, `machine_learning_agent_iter_submit.json`.

## References

- `run_infer_idea_ours.py`: `_create_implementation_plan` (830-858), `_implement_and_iterate` (861-920), `_submit_and_refine_experiments` submit step (922-945)
- `prompt_templates.py`: `build_plan_query` (203-233), `build_ml_dev_query` (236-381), `build_judge_query` (384-417), `build_iteration_query` (420-468), `build_judge_simple_query` (471-494), `build_submit_query` (497-527)
- Agent definitions: `plan_agent.py`, `ml_agent.py`, `judge_agent.py` in `inno/agents/inno_agent/`