2569 Target User Agentic Harness V0 A84a827c

# Product + Technical Spec: Target User Agentic Harness (v0)

Status: PRD draft (not implemented).

## Objective
Build a repeatable harness that simulates how real target users interact with Brood:
- import realistic photo sets,
- provide project context,
- use Mother + Abilities over multiple turns,
- log explicit "open pondering" questions each turn,
- produce scored, reproducible run artifacts.

## Feasibility Snapshot
Feasibility is high because the required primitives already exist:
- `brood chat --out ... --events ...` supports multi-turn interaction.
- Abilities already exist as slash commands (`/diagnose`, `/argue`, `/bridge`, `/swap_dna`, `/triforce`, etc.).
- `events.jsonl` is append-only and machine-readable.
- `brood_engine/harness` already handles deterministic multi-step experiment runs and telemetry patterns.

The missing piece is orchestration for persona-driven user behavior, not core model capability.

## v0 Scope
- Adapter: CLI-first (`brood chat` PTY) for speed and determinism.
- Persona + scenario pack input (JSON).
- Agentic turn loop with explicit reflection logging.
- Ability coverage tracking and stop conditions.
- Scorecard per scenario and aggregate report.

## Non-Goals (v0)
- Pixel-level desktop UI automation.
- Perfect simulation of human behavior.
- Using reflection text as hidden reasoning. Reflection is explicit, user-visible artifact data.

## Architecture
Components:
- Scenario pack loader: validates persona/scenario config.
- Session runner: executes one scenario against one persona.
- Chat adapter: sends utterances/commands to `brood chat`, tails `events.jsonl`.
- Policy layer: chooses next action from scenario goals + recent events.
- Reflection writer: writes explicit turn reflections to `reflections.jsonl`.
- Evaluator: computes success and rubric scores from artifacts/events/reflections.

Suggested code placement:
- `brood_engine/harness/user_sim.py` (core loop + data models)
- `scripts/target_user_harness.py` (CLI entrypoint)
- `docs/target_user_harness.schema.json` (config contract)

## Run Artifacts
For each scenario run:
- `session.json`: resolved persona/scenario/policy.
- `events.jsonl`: raw engine events (from Brood run dir).
- `transcript.jsonl`: adapter-level IO (`user_input`, `assistant_output`, `command`).
- `reflections.jsonl`: explicit per-turn pondering.
- `scorecard.json`: rubric + pass/fail result.
- `summary.json`: compact outcome for dashboarding.

## State Machine
States:
1. `init`
2. `start_chat`
3. `import_inputs`
4. `plan_turn`
5. `act`
6. `observe`
7. `reflect`
8. `score_checkpoint`
9. `done`
10. `error`

Transitions:
- `init -> start_chat` when scenario validates.
- `start_chat -> import_inputs` when PTY and events are live.
- `import_inputs -> plan_turn` after `/use` and required multi-image context setup.
- `plan_turn -> act` when next action is chosen.
- `act -> observe` after command/utterance dispatch.
- `observe -> reflect` after event delta window closes.
- `reflect -> score_checkpoint` every turn.
- `score_checkpoint -> plan_turn` while stop condition is unmet.
- `score_checkpoint -> done` on success, budget exhaustion, or max turns.
- Any state -> `error` on unrecoverable adapter/engine failure.

Stop conditions:
- Success criteria satisfied.
- `max_turns` reached.
- `max_cost_usd` reached.
- `max_runtime_s` reached.
- Consecutive failure threshold reached.

## Action Model
Two action forms are supported:
- `utterance`: natural language sent to Mother.
- `command`: explicit slash command to an Ability.

Supported `command.name` in v0:
- `use`
- `diagnose`
- `describe`
- `canvas_context`
- `blend`
- `bridge`
- `swap_dna`
- `argue`
- `extract_rule`
- `odd_one_out`
- `triforce`
- `recast`
- `quality`
- `fast`
- `cheaper`
- `better`

Interpretation:
- Use `utterance` when simulating ambiguous user intent.
- Use `command` when simulating power-user behavior and explicit Ability invocation.

## Open Pondering Contract
Each turn must write one reflection record:
- `turn`: integer
- `action_taken`: normalized action record
- `observed_signals`: key events/artifacts in that turn
- `what_worked`: short text
- `what_failed`: short text
- `open_questions`: array of unresolved user-style questions
- `next_hypothesis`: what the simulated user will try next
- `confidence`: float `0.0..1.0`

This keeps reasoning explicit and auditable.

## Scoring (v0)
Per scenario score is weighted:
- `goal_progress` (35%)
- `artifact_quality_proxy` (20%)
- `ability_usage_quality` (15%)
- `efficiency_cost_time` (15%)
- `resilience` (10%)
- `reflection_quality` (5%)

Pass gate:
- `goal_progress >= 0.7`
- at least one required ability used
- no fatal adapter errors

## Three Benchmark Scenarios (v0)
1. `drop_lookbook_rescue`
- Persona: indie streetwear founder.
- Inputs: 3 rough iPhone product/lifestyle photos.
- Goal: produce 2 hero images for a launch post in <12 turns.
- Required abilities: `diagnose`, `bridge` or `blend`, `argue`.

2. `tattoo_flash_direction_find`
- Persona: tattoo artist exploring a motif system.
- Inputs: 2 references + 1 sketch.
- Goal: produce one coherent "centroid" concept + rule extraction.
- Required abilities: `triforce`, `extract_rule`, `odd_one_out`.

3. `creator_thumbnail_pack`
- Persona: solo creator shipping a video by tonight.
- Inputs: selfie + screenshot + mood reference.
- Goal: produce 3 viable thumbnail candidates and select one direction.
- Required abilities: `swap_dna` or `bridge`, `diagnose`, `argue`.

Concrete sample config for these scenarios is in `docs/target_user_harness.scenarios.sample.json`.

## Rollout Plan
Phase 1 (engine-level):
- Implement CLI adapter and deterministic loop.
- Run scenario pack using dryrun and one live provider profile.

Phase 2 (evaluation hardening):
- Add score calibration and failure taxonomy.
- Add aggregate report for multi-run comparisons.

Phase 3 (desktop fidelity):
- Add optional desktop adapter for true upload/canvas flow parity.

## Risks and Mitigations
- Simulation drift from real users.
  Mitigation: calibrate scenario pack from real anonymized usage motifs.
- Agent over-optimizes rubric.
  Mitigation: separate actor policy from evaluator policy/model.
- Flaky provider/network responses.
  Mitigation: retries, bounded backoff, and deterministic seeds where possible.

## Acceptance Criteria
- Run all three benchmark scenarios from one JSON pack.
- Produce complete artifact bundle per scenario.
- Emit reflection record every turn.
- Generate deterministic summary for repeated seeded runs.
- Complete without manual intervention in CLI adapter mode.