Ios Visual Explore

---
name: ios-visual-explore
description: |
  Tier-2 visual exploration of an iOS app via Gemini computer-use. Invoke when
  accessibility-based XCUITest doesn't surface enough — layout regressions,
  glitches, elements off the accessibility tree. Text-only Markdown report.
---

# iOS Visual Explore

## When to use

Use this skill as a **Tier-2 escalation** after standard accessibility-based testing has already been attempted or is insufficient:

- Visual or layout regressions that accessibility assertions cannot detect
- Flows driven by visual landmarks (images, icons, colour cues) rather than labels
- Exploratory missions ("find any visual issues on the onboarding flow") where the goal is open-ended
- Debugging a user-reported visual issue that doesn't reproduce in unit tests

Do **not** use as a first resort — prefer deterministic XCUITest flows when the app's accessibility tree is well-formed.

## Prerequisites

- A booted iOS simulator (or physical device via idb) with the target app installed
- `idb_companion` available on PATH (or installed via Homebrew)
- `GEMINI_API_KEY` / `gemini-api-key-paid` in macOS Keychain (for the Gemini vision backend)
- `uv` available on PATH (for the shim runner)

## How to invoke

First locate the runner self-relatively — this skill runs in the USER's project,
where `scripts/` does not exist. Derive it from this skill's base directory
(shown when the skill loads); this resolves both in the plugin repo and in a
marketplace install:

```bash
IVE="<this skill's base directory>/../../scripts/ios-visual-explore"
```

```bash
"$IVE" \
  --udid <SIMULATOR_UDID> \
  --bundle <BUNDLE_ID> \
  "<mission description>" \
  [--max-steps N]                    # default: 25
  [--orientation portrait|landscape] # default: portrait (see iPad / orientation below)
  [--dry-run]                        # single planning probe; no actions executed
  [--out <dir>]                      # default: computer-use-runs/run-<timestamp>
```

**Examples:**

```bash
# Full exploration run (up to 15 steps)
"$IVE" \
  --udid "A1B2C3D4-..." \
  --bundle "com.example.MyApp" \
  "trykk gjennom onboarding-flyten og rapporter visuelle problemer" \
  --max-steps 15

# Dry run — shows the planned first action without executing anything
"$IVE" \
  --udid "A1B2C3D4-..." \
  --bundle "com.example.MyApp" \
  "utforsk innstillinger-skjermen" \
  --dry-run

# iPad in landscape — rotate the simulator to landscape FIRST (see below)
"$IVE" \
  --udid "<iPad-UDID>" \
  --bundle "com.example.MyApp" \
  "utforsk delt-visning-layouten i landskap" \
  --orientation landscape
```

## iPad and orientation (Phase 2)

The skill supports iPhone and iPad, in either orientation, but **fullscreen only**.

- **You must rotate the simulator yourself before the run.** Neither `simctl` nor `idb` can rotate a simulator programmatically, so the tool does **not** set orientation — it only *verifies* it. For landscape, rotate **left** (Simulator.app: Device → Rotate Left, or **Cmd+←**) so the app is in landscape, then launch. If the Application frame's aspect doesn't match `--orientation`, preflight fails closed with a "rotate the sim first" message rather than running against the wrong layout.
- **Landscape must be a LEFT rotation (Cmd+←).** `simctl io screenshot` always captures the device's native-portrait framebuffer, so a landscape app lands rotated inside a portrait-shaped PNG. The tool rotates that screenshot upright before the model sees it — but the rotation direction is fixed for a left-rotated device. A right-rotation (Cmd+→) would make the model see an upside-down image and mis-target taps. iPad safe-area is symmetric, so left vs right costs you nothing; just always rotate left.
- **Fullscreen only.** The app must occupy the full screen. Split View, Slide Over, and Stage Manager windows are rejected at preflight: fullscreen is confirmed empirically (screenshot pixel-width ÷ Application frame-width must be a clean @2x/@3x backing scale), so a narrower windowed frame fails closed. There is no split-view or multi-window exploration.
- **Safe-area insets** are looked up from a per-device-class × orientation table (`INSET_TABLE`), since iPad's accessibility tree exposes no status-bar / home-indicator elements to derive them from. Values are HIG-typical (iPad ≈ 24 pt top / 20 pt bottom). Taps and drags whose coordinates fall outside the safe area are **rejected** (logged as `rejected` / "outside safe area") instead of executing, so the run never fires into the status bar or home indicator.
- **Foreground oracle is self-referential.** The target app is identified by its `Application.AXLabel` (its display name, e.g. "Innstillinger"), captured at launch and compared each step — iPad's tree carries no bundle id on the Application element. A mismatch (wrong app *or* home screen, whose label differs) yields `app_left_foreground`. If the app exposes no AXLabel at all, preflight fails closed (the oracle would otherwise never confirm foreground).
- **iPad describe-all is flaky during transitions** (it briefly returns a degenerate tree). Every accessibility query retries until the tree is stable (>3 typed elements). A cold-booted iPad can also take 30–60 s after "booted" before its UI renders — wait for it before launching.

### Known limitations (honest best-effort)

- **Sim orientation persists after the run** — the tool never rotates, so the simulator stays in whatever orientation you left it. Rotate back manually if needed.
- **Slide Over with the target still frontmost** — if another app is in a Slide Over panel while the target remains the frontmost fullscreen Application, the oracle still reports foreground; the overlay is not separately detected.
- **No split/Stage Manager coverage** — by design (see fullscreen-only above); multi-window layouts are out of scope for this phase.

## Output contract

The skill writes **two artefacts** to the output directory and prints a **text-only summary** to stdout:

| Artefakt | Beskrivelse |
|----------|-------------|
| `<out>/report.md` | Full Markdown report: mission, environment info, step journal, critic findings |
| `<out>/screenshots/` | Every step's screenshot is persisted. The vision critic inspects **every screen the run reached** (baseline + each step), so visual issues on successfully-navigated screens are caught, not just on endpoints and error steps. Perceptual near-duplicate suppression (e.g. successive scrolls) is a future `ahash` optimization. |

**Stdout** contains only text: file paths and a structured summary of findings. It never includes inline images or base64 data. This keeps the Claude context clean when the summary is captured or pasted into a conversation.

## Interpreting results

The `status` field in the report and stdout summary is a stable machine token with exactly four possible values:

- **`completed`** — the model signalled it was done before hitting the step limit
- **`step_limit`** — the run exhausted `--max-steps`; consider increasing it or narrowing the mission
- **`app_left_foreground`** — the foreground oracle detected the target app was no longer in the foreground (e.g. backgrounded or crashed); the partial journal is still written
- **`error`** — abnormal failure (idb error, API error, screenshot failure); check the journal in `report.md` for the first failing step

Findings in the report are produced by the Gemini vision critic and may include false positives; treat them as leads, not definitive bugs. A report is always written even on `error` status — the journal up to the point of failure is preserved.

## Architecture note

The skill orchestrates five layers built in Phase 1:
`preflight` → `executor_idb` → `loop` (Gemini computer-use) → `dedup` → `critic` + `report`.
VisionCritic (in `scripts/computer_use/gemini.py`) runs the evidence screenshots through Gemini vision; it degrades gracefully to an empty findings list on any API error — the run never crashes on critic failure.

## Capability notes

- **Perceptual near-duplicate dedup** — WIRED. `ahash_png` (in `dedup.py`) decodes each retained screen via `sips` downscale → 8×8 BMP (stdlib parse, no Pillow) and `perceptual_dedup` collapses visually near-identical successive screens before the critic runs, so the critic isn't billed for redundant pixels (e.g. a scroll that hit the end). A decode failure keeps the screen.
- **Richer action coverage** — WIRED. `long_press` (held zero-length swipe), `go_back` (iOS interactive-pop edge-swipe from the left; not safe-area-checked since it starts at the edge), and `press_key` (idb `key` with a HID mapping for Return/Escape/Backspace/Tab/Space; numeric keycodes pass through) are handled in `actions.py` + `executor_idb.py`.
- **Gemini-3 `signature` echo** — NOT NEEDED. Thought signatures are carried server-side via `previous_interaction_id` (the session driver in `gemini.py`), so echoing them back is redundant; the signature is still preserved verbatim in the journal via each entry's `raw`. Confirmed across many multi-step live runs.