Flame Harness Evaluator

---
name: flame-harness-evaluator
description: Phase 6 — skeptical QA. Run the game, watch it, then judge against the contract. Default = functional check; --strict adds quality and edge-case passes.
argument-hint: ""
allowed-tools: [Agent, Read, Write, Edit, Bash, Glob, Grep]
---

# flame-harness-evaluator

Phase 6 of the flutter-flame-harness pipeline. Skeptical QA gate that decides PASS or FAIL
against the negotiated contract. Default mode runs the functional check (6.1) only; `--strict`
adds quality scoring (6.2) and an agent-team edge-case sweep (6.3).

All file schemas (`config.md`, `state.md`, `handoff/round-N-gen.md`, `feedback/round-N-qa.md`,
`build-log.md`) and the phase transition table are defined in `docs/harness-protocol.md` — that
document is the single source of truth (§1 for `config.md`; §2 for `state.md`; §3 for
`contract.md`; §4 for handoff layout; §5 for feedback layout; §6 for log schemas; §7 for the
`evaluator → admob / evaluator → build / evaluator → generator` transitions). Do not redefine schemas here.

---

## Critical Rule

**"Run the code, see the app, then judge." Never PASS on code review alone. Execute commands,
launch the game on a simulator, play the core loop, capture and study screenshots. Stub detected
= automatic FAIL, no exceptions.**

---

## Setup — Read Inputs

Before any check, load:

1. `docs/harness/state.md` — extract `current_round` (integer ≥ 1) and confirm
   `next_role: evaluator`.
2. `docs/harness/config.md` — extract `app_slug`, `strict_mode` (bool), `max_rounds` (int).
3. `docs/harness/contract.md` — parse `## Mandatory Hard Gates` and `## Functional Criteria`.
   Confirm `## Status: AGREED` is present; if missing, abort with:
   `flame-harness-evaluator: contract not AGREED — run flame-harness-contract first`.
4. `docs/harness/handoff/round-<N>-gen.md` (where N = `current_round`) per protocol §4.
   If the file is missing, abort with:
   `flame-harness-evaluator: handoff for round <N> not found — generator must run first`.

---

## 6.1 Functional Check (default — always runs)

Run every step in order. A failure on any **Mandatory Hard Gate** from `contract.md` is an
immediate FAIL — do not continue checking remaining criteria.

### Step 1 — Static analysis

```bash
cd <projects-dir>/<app_slug>
flutter analyze
```

Required result: **0 issues**. If any issues are reported, record each with file, line, and
message. This is a Mandatory Hard Gate.

### Step 2 — Tests

```bash
cd <projects-dir>/<app_slug>
flutter test
```

Required result: **0 failures**. Capture the full output. This is a Mandatory Hard Gate.

### Step 3 — Stub / TODO grep

```bash
grep -rn "TODO\|stub\|placeholder\|스텁\|미구현" \
  <projects-dir>/<app_slug>/lib/ --include="*.dart"
```

Any match is an **automatic FAIL**. No exceptions (per `docs/harness-protocol.md` §3 Hard
Gate 3). Record each match with file path, line number, and the matched text.

### Step 4 — game_config.dart centralization

```bash
grep -rn "[0-9]\{3,\}\.\?[0-9]*" \
  <projects-dir>/<app_slug>/lib/game/ --include="*.dart" \
  | grep -v "game_config.dart"
```

Magic numbers (3+ digits) in game logic files other than `game_config.dart` are a Hard Gate
failure (per protocol §3 Hard Gate 4). Exclude intentional non-tuning constants (e.g., HTTP
status codes) if clearly justified in a comment; document any exclusion in the feedback file.

### Step 5 — l10n completeness

```bash
cd <projects-dir>/<app_slug>
flutter gen-l10n
```

Then confirm that every configured `lib/l10n/app_<locale>.arb` (the project's `default_language`,
plus `app_en.arb` when `default_language` ≠ `en`) contains identical key sets:

```bash
python3 -c "
import glob, json
arbs = glob.glob('lib/l10n/app_*.arb')
keysets = {f: set(json.load(open(f))) for f in arbs}
allkeys = set().union(*keysets.values()) if keysets else set()
bad = {f: sorted(allkeys - ks) for f, ks in keysets.items() if allkeys - ks}
if len(arbs) < 1: print('NO ARB FILES'); exit(1)
if bad: print('MISSING KEYS:', bad); exit(1)
print('l10n OK', list(keysets))
"
```

Missing keys in either locale are a Hard Gate failure (protocol §3 Hard Gate 6).

### Step 5a — Platform-robustness gates

Verify the `## Platform-Robustness Gates` (R1–R11) from `contract.md` / `docs/game-gotchas.md`:

```bash
# R1 audio: frequent SFX pooled + audio calls guarded; BGM stop on teardown
grep -rn "AudioPool" lib/ || echo "WARN: no AudioPool — frequent SFX may stutter"
grep -rn "FlameAudio\|AudioPool\|\.bgm" lib/ | grep -i "try\|catch" >/dev/null || echo "CHECK: audio not in try/catch"
grep -rn "bgm.stop\|\.stop()" lib/ || echo "CHECK: BGM stop on teardown/background"
# R2 haptics (if used)
ls lib/systems/haptics.dart 2>/dev/null && grep -nE "kIsWeb|isIOS|isAndroid|enabled|Stopwatch|elapsed" lib/systems/haptics.dart
# R3 lifecycle
grep -rn "WidgetsBindingObserver\|didChangeAppLifecycleState\|pauseEngine" lib/ || echo "FAIL: no lifecycle pause"
# R4 performance: no per-frame whereType in update hot paths
grep -rn "whereType" lib/ | grep -i "update" && echo "CHECK: whereType inside update — verify it's cached, not per-frame"
# R5 branding: custom icon + splash + localized display name (not defaults)
ls assets/icons/icon.png 2>/dev/null || echo "FAIL: no custom app icon"
grep -q "flutter_launcher_icons" pubspec.yaml && grep -q "flutter_native_splash" pubspec.yaml || echo "FAIL: launcher-icons/native-splash not configured"
sips -g hasAlpha assets/icons/icon.png 2>/dev/null | grep -qi "hasAlpha: no" || echo "CHECK: icon may have alpha (App Store rejects)"
# R5 Android adaptive icon: padded foreground + monochrome configured & emitted (not the full-bleed iOS icon → avoids the cropped/"zoomed" launcher look)
grep -q "adaptive_icon_foreground" pubspec.yaml || echo "CHECK: no adaptive_icon_foreground — Android reuses full-bleed icon and looks cropped/zoomed"
grep -q "adaptive_icon_monochrome" pubspec.yaml || echo "CHECK: no adaptive_icon_monochrome — Android 13+ themed icon will look off"
ls android/app/src/main/res/mipmap-anydpi-v26/ic_launcher.xml 2>/dev/null | grep -q . || echo "CHECK: adaptive-icon XML not emitted (run flutter_launcher_icons)"
grep -rn "CFBundleDisplayName" ios/Runner/Info.plist 2>/dev/null || echo "CHECK: iOS display name not set"
grep -rn 'android:label' android/app/src/main/AndroidManifest.xml 2>/dev/null | grep -qiv "runner" || echo "CHECK: Android label still default/runner"
# R6 native config: orientation locked (matches config.orientation, no ~ipad), iPhone-only, export compliance, root back
grep -q "UISupportedInterfaceOrientations" ios/Runner/Info.plist 2>/dev/null && ! grep -q "UISupportedInterfaceOrientations~ipad" ios/Runner/Info.plist || echo "CHECK: orientation not locked / ~ipad still present"
grep -rq "TARGETED_DEVICE_FAMILY = 1" ios/Runner.xcodeproj/project.pbxproj 2>/dev/null || echo "FAIL: not iPhone-only (TARGETED_DEVICE_FAMILY != 1)"
grep -q "ITSAppUsesNonExemptEncryption" ios/Runner/Info.plist 2>/dev/null || echo "CHECK: export-compliance key missing"
grep -rq "PopScope" lib/ || echo "CHECK: no root back-button handler"
# R6 bundle id identical on both platforms (== config.bundle_id, lowercase no _/-)
IOSID=$(grep -oE 'PRODUCT_BUNDLE_IDENTIFIER = [A-Za-z0-9._-]+' ios/Runner.xcodeproj/project.pbxproj 2>/dev/null | head -1 | sed 's/.*= //')
ANDID=$(grep -oE 'applicationId *= *"[^"]+"' android/app/build.gradle.kts 2>/dev/null | head -1 | sed -E 's/.*"([^"]+)"/\1/')
echo "iOS=$IOSID  Android=$ANDID  (must be byte-identical == config.bundle_id)"
{ [ -n "$IOSID" ] && [ "$IOSID" = "$ANDID" ]; } || echo "FAIL: iOS and Android bundle id differ (or unset)"
echo "$IOSID" | grep -qE '^[a-z0-9.]+$' || echo "FAIL: bundle id has uppercase/_/- (must be lowercase [a-z0-9.])"
# R7 assets & CI: audio present, no missing-asset refs, CI workflow exists
ls assets/audio/*.wav assets/audio/*.mp3 assets/audio/*.ogg 2>/dev/null | grep -q . || echo "FAIL: no audio assets — game ships silent"
ls .github/workflows/*.yml .github/workflows/*.yaml 2>/dev/null | grep -q . || echo "FAIL: no CI workflow"
# R8 Play listing graphics: hi-res icon (512) + feature graphic (1024x500) per locale
ls android/fastlane/metadata/android/*/images/icon.png 2>/dev/null | grep -q . || echo "FAIL: no Play hi-res icon (512x512)"
ls android/fastlane/metadata/android/*/images/featureGraphic.png 2>/dev/null | grep -q . || echo "FAIL: no Play feature graphic (1024x500)"
# R9 durable save: SaveRepository mirrors to Keychain + Block Store + prefs; PreferencesService routes through it
grep -q "flutter_secure_storage" pubspec.yaml && grep -q "play_services_block_store" pubspec.yaml || echo "FAIL: durable-save deps missing (saves won't survive reinstall/device on iOS)"
ls lib/data/save_repository.dart 2>/dev/null || echo "FAIL: no SaveRepository (durable save layer)"
grep -q "FlutterSecureStorage" lib/data/save_repository.dart 2>/dev/null && grep -q "PlayServicesBlockStore" lib/data/save_repository.dart 2>/dev/null || echo "FAIL: SaveRepository missing a durable tier (Keychain/Block Store)"
# R10 accessibility & safety: reduce-motion read, menu Semantics present, no >3Hz flashing
grep -rq "disableAnimations" lib/ || echo "CHECK: OS Reduce Motion (MediaQuery.disableAnimations) not respected"
grep -rq "Semantics(" lib/screens/ lib/ui/ 2>/dev/null || echo "CHECK: no Semantics labels on menu/overlay buttons (screen-reader nav)"
grep -rniE "flash|strobe|invert" lib/ | grep -i "update\|timer\|0\.0[0-9]" && echo "CHECK: verify no full-screen flash faster than 3/sec (photosensitive safety)"
# R11 test depth: >=3 system unit tests + >=1 widget test + >=1 integration test
UNIT=$(ls test/*_test.dart 2>/dev/null | grep -viE 'widget' | wc -l | tr -d ' ')
grep -rlq "testWidgets" test/ 2>/dev/null || echo "FAIL: no widget test (R11 needs >=1)"
ls integration_test/*.dart >/dev/null 2>&1 || echo "FAIL: no integration test (R11 needs >=1 core-loop test)"
[ "${UNIT:-0}" -ge 1 ] || echo "CHECK: few unit test files — R11 wants >=3 system unit tests"
# every asset path declared in pubspec must exist on disk
python3 - <<'PY'
import re,glob,os,sys
try: txt=open('pubspec.yaml').read()
except FileNotFoundError: sys.exit(0)
m=re.search(r'\n\s*assets:\s*\n((?:\s*-\s.*\n)+)', txt)
missing=[]
if m:
    for line in m.group(1).splitlines():
        p=line.strip().lstrip('- ').strip().strip('"\'')
        if not p: continue
        if p.endswith('/'):
            if not (os.path.isdir(p) and os.listdir(p)): missing.append(p)
        elif not os.path.exists(p): missing.append(p)
print('MISSING ASSETS:',missing) if missing else print('assets OK')
PY
```

Judge results against `docs/game-gotchas.md`: missing lifecycle pause (R3) or unguarded audio that
can crash on a missing asset (R1) is a FAIL. A per-frame `whereType` in a hot `update` path, or
raw `HapticFeedback.*` in gameplay without the guarded helper, is a FAIL when the game relies on it.
Also confirm the game does **not crash when an audio/image asset is missing** (it should degrade).
**Branding (R5):** default Flutter icon/splash, an alpha-channel icon, or a "Runner"/slug display
name is a FAIL — the app must look shipped. **Native config (R6):** the app must open directly in
`config.orientation` (no rotate flash → orientation locked natively, `~ipad` removed), be iPhone-only,
have `ITSAppUsesNonExemptEncryption=false`, and handle the root back-button (SnackBar double-exit).
**Assets & CI (R7):** no audio (game ships silent), any missing declared asset, or a missing CI
workflow is a FAIL. **Store graphics (R8):** a missing Play hi-res icon (512×512) or feature graphic
(1024×500) is a FAIL — the listing can't publish without them.
**Durable save (R9):** persistence on `shared_preferences` alone is a FAIL — iOS drops it on app
delete, so the player loses progress on reinstall/new device. There must be a `SaveRepository`
mirroring to iOS Keychain + Android Block Store (durable-first read, all-tier write, try/catch), with
`PreferencesService` routing through it.
**Accessibility & safety (R10):** a full-screen effect that flashes faster than 3×/second is a FAIL
(photosensitive-seizure safety). Reduce Motion not read (`MediaQuery.disableAnimations`) and icon-only
menu/overlay buttons with no `Semantics` label are CHECK-level — a FAIL only if the game leans on
heavy motion with no damping. Tap targets on menu buttons should be ≥48×48 dp.
**Test depth (R11):** a passing `flutter test` that only checks `GameConfig` constants is a FAIL —
there must be ≥3 system unit tests + ≥1 widget test + ≥1 integration test, and all must pass.

### Step 6 — Contract criteria evidence

For each criterion in `contract.md` §§ "Mandatory Hard Gates" and "Functional Criteria", record
a row in the feedback evidence table showing the command run, the result, and any screenshot or
log path. Do not mark a criterion DONE without running a command that directly verifies it.

### Step 7 — Launch on simulator and play core loop

**This step is mandatory. A PASS verdict is not valid without completing it.**

Boot the iOS simulator (or Android emulator if iOS is unavailable):

```bash
open -a Simulator
xcrun simctl boot "iPhone 16" 2>/dev/null || true
```

Install and launch the game:

```bash
cd <projects-dir>/<app_slug>
flutter run -d "iPhone 16" --no-pub
```

While the game is running:

0. Create the screenshots directory if it does not already exist:
   ```bash
   mkdir -p <projects-dir>/<app_slug>/docs/harness/screenshots
   ```
1. Navigate to the main menu. Capture a screenshot:
   ```bash
   xcrun simctl io booted screenshot \
     <projects-dir>/<app_slug>/docs/harness/screenshots/round-<N>-ios-menu.png
   ```
2. Start a game session. Play through the core loop (start → play → win/lose → restart).
3. Capture a screenshot of the gameplay state:
   ```bash
   xcrun simctl io booted screenshot \
     <projects-dir>/<app_slug>/docs/harness/screenshots/round-<N>-ios-play.png
   ```
4. Reach a game-over or win condition. Capture the result screen:
   ```bash
   xcrun simctl io booted screenshot \
     <projects-dir>/<app_slug>/docs/harness/screenshots/round-<N>-ios-end.png
   ```
5. Study each screenshot carefully. Check for: blank/white screens, visual glitches, missing
   sprites, overlapping UI elements, wrong language, or any rendering error.
6. Confirm the game runs without crashes and without console errors.

Any crash, blank screen, or console error is a Hard Gate failure (protocol §3 Hard Gates 7–8).

---

## 6.2 Quality Scoring (`--strict` only)

Run this section only when `strict_mode: true` in `config.md` or `--strict` is passed.

Score the game on four axes, each 0–10:

| Axis | What to assess |
|---|---|
| **Game feel / juice** | Responsiveness, animations, feedback on actions, audio cues |
| **Originality** | Freshness relative to competitors identified in the research phase |
| **Craft** | Code quality, absence of jank, visual polish, consistent design tokens |
| **Functionality** | All contract criteria met, no edge-case breakage observed |

Also assess:

- **Interaction states**: Does the game handle loading, error, and empty states gracefully?
- **Responsiveness**: Does the game render correctly on the target device sizes?

Scoring threshold:

- Default (`strict_mode: false`): weighted average ≥ **7 / 10** to advise PASS (advisory only).
- Strict profile (`strict_mode: true`): weighted average ≥ **8 / 10** required for PASS.

Weight: Game feel 30 %, Originality 20 %, Craft 25 %, Functionality 25 %.

Record the score for each axis and the weighted total in the feedback file. If the weighted
total is below threshold, the verdict is FAIL regardless of 6.1 results, and each axis below
7 must have at least one specific, reproducible fix listed.

---

## 6.3 Edge-Case Sweep (`--strict` only)

Run this section only when `strict_mode: true` in `config.md` or `--strict` is passed.

Spawn six specialist agents in parallel via the `Agent` tool. All six must report PASS for the
overall verdict to be PASS. A single FAIL from any agent is a FAIL verdict.

| Agent role | Brief |
|---|---|
| **gameplay-edge** | Find gameplay edge cases: score overflow, negative health, unreachable states, off-screen entities, simultaneous collision resolution. |
| **balance** | Assess difficulty curve: is the game winnable? is it too easy in the first 30 s? does difficulty ramp feel fair? |
| **lifecycle/crash** | Simulate app backgrounding (home button), screen rotation, incoming call interruption; confirm resume works and no crash. |
| **performance** | Run `flutter run --profile`, check frame rendering in DevTools; flag sustained drops below 30 fps. |
| **test-generator** | Identify the three highest-risk untested paths and write widget/unit tests for them; confirm they pass. |
| **adversarial-reviewer** | Adversarially review the generated code looking for security issues, credential leaks, or App Store policy violations. |

Each agent must return a structured PASS/FAIL verdict with evidence. Collect all six verdicts
before proceeding to Judgment.

---

## Judgment

After completing all applicable sections (6.1, and 6.2 + 6.3 if `--strict`), write the verdict.

### Determine verdict

- **PASS** if and only if:
  - All 6.1 Mandatory Hard Gates pass AND all 6.1 Functional Criteria verified with evidence.
  - If `--strict`: 6.2 weighted score ≥ threshold AND all six 6.3 agents report PASS.
- **FAIL** if any Hard Gate fails, any functional criterion lacks evidence, or (when `--strict`)
  6.2 score is below threshold or any 6.3 agent reports FAIL.

### max_rounds check

Before writing the verdict, check: if `current_round == max_rounds`, force the judgment. Do
not return FAIL regardless of results — write the verdict on the current state, record the
forced-advance note in the feedback file, and proceed to PASS transitions (per protocol §7
`evaluator → max_rounds → admob / build`, applying the same `skip_admob` branch as PASS).

### Write feedback/round-N-qa.md

Create `docs/harness/feedback/round-<N>-qa.md` following the layout in
`docs/harness-protocol.md` §5. Fill:

- `## Verdict` — **PASS** or **FAIL** (bold).
- `## Evidence` — one row per criterion checked; include screenshot paths for simulator checks.
- `## Failed Criteria` — for each FAIL, a specific reproducible fix. If PASS, write "none".

Never leave placeholder text. Every criterion must have a real command output or screenshot path
as evidence.

### Update state.md (PASS)

On PASS (or forced advance at max_rounds), the game has been built and has passed QA — but before
any deploy work (admob/build/screenshot/submit), there is a **human-approval gate** by default so
the user can actually play and approve the game.

First read `skip_admob` and `auto_deploy` from `docs/harness/config.md`, and decide `next_role`:
- if `skip_admob: true` → `next_role: build`
- otherwise → `next_role: admob`

Then branch on `auto_deploy`:

**Default (`auto_deploy: false`) — PAUSE for human review.**
Write `state.md` with `status: paused`, `pause_reason: manual_action`, and the `next_role`
decided above (per protocol §7 rule 2). Append a `pipeline-log.md` row and PRINT a review
checklist for the user:

> 게임 빌드 + QA 통과. **배포 전 직접 확인하세요:** `cd <app_slug> && flutter run` 으로 플레이하고,
> `docs/harness/screenshots/` 의 QA 스크린샷과 `docs/harness/feedback/round-<N>-qa.md` 를 확인.
> 만족하면 `/flame-harness --resume` 로 배포(admob→build→screenshot→submit)를 진행합니다.

```yaml
status: paused
current_phase: evaluator
next_role: admob   # or "build" if skip_admob: true
pause_reason: manual_action
updated_at: "<ISO-8601 UTC now>"
```

The orchestrator halts on `status: paused`; on `--resume`, `flame-harness-resume` confirms the
user approved and dispatches the stored `next_role`.

**`auto_deploy: true` — no pause, continue straight to deploy.**
Write `status: running` with the same `next_role` so the orchestrator auto-continues:

```yaml
status: running
current_phase: evaluator
next_role: admob   # or "build" if skip_admob: true
updated_at: "<ISO-8601 UTC now>"
```

Leave `current_round`, `created_at`, and `resume_attempts` unchanged. (In the `auto_deploy: false`
case you set `pause_reason: manual_action`; in the `auto_deploy: true` case leave `pause_reason`
unchanged.)

### Update state.md (FAIL)

On FAIL (and `current_round < max_rounds`), update `docs/harness/state.md` per protocol §2 and
the `evaluator → generator` transition in §7. Increment `current_round` and set
`status: running` atomically (protocol §7 rule 2):

```yaml
status: running
current_phase: evaluator
next_role: generator
current_round: <N+1>
checkpoint: ""
updated_at: "<ISO-8601 UTC now>"
```

Leave `created_at`, `resume_attempts`, and `pause_reason` unchanged. **Reset `checkpoint: ""`** — the
next round is a fresh feedback-driven pass, so the generator must not skip sub-phases (per protocol §2).

### Append to build-log.md

Append one row to `docs/harness/build-log.md` per protocol §6:

```
| <N> | evaluator | PASS/FAIL | <duration> | <one-line summary> |
```

### Append to pipeline-log.md

Append one row to `docs/harness/pipeline-log.md` per protocol §6:

```
| <ISO-8601 UTC now> | PASS/FAIL | evaluator | round <N>; next: admob|build/generator |
```

When the PASS path pauses for the human-review gate (default, `auto_deploy: false`), additionally
append a `pause` event row so `flame-harness-resume` can show the user what they are approving:

```
| <ISO-8601 UTC now> | pause | evaluator | manual_action: play/approve the built game before deploy; next: admob|build |
```

---

## Error Handling

- If `contract.md` is missing or has no `## Status: AGREED`, abort immediately.
- If `handoff/round-<N>-gen.md` is missing, abort with a clear message; do not set a FAIL
  verdict — the generator has not run yet.
- If the simulator cannot be booted (hardware CI), document the failure, skip Step 7, and set
  the verdict to FAIL with fix: "boot a simulator and complete Step 7."
- If any agent in 6.3 returns an error (not FAIL, but a tool error), retry once, then count it
  as FAIL with note "agent error — retry required."