Research

# R5: how an agent reviews a built UI (research behind `opendesigner-review`)

Lane: R5. Written 2026-10-04 from research done 2026-09-29. Trace: `traces/R5-trace.md` (100 sources; every one opened is listed; ids `S-R5-nnn`). The distilled output is the catalogue `synthesis/review-cases.json` (65 cases), generated into `skills/opendesigner-review/references/cases.md`. This file records the question, the method, what was found, what is unsourced, and what is still open.

## The question
Kunal's request: a skill that teaches Claude or any agent to review its own or anyone's work. It must take screenshots, hover and interact, scroll both ways to the end, check padding, margins, type scale and fonts, and cover things a single screenshot misses (animation, motion graphics, infographics), with research on where AI misses the real case.

## Method
Four research agents ran in parallel with web search and page fetches, each answering a fixed list of questions with a URL it had opened, or a tag (`[snippet only]`, `[unverified]`, `[inferred]`, `[tested]`): (A) capture mechanics and where screenshots mislead; (B) motion and change over time; (C) how and why AI agents get visual review wrong, and the process that fixes it; (D) what human design QA and real users check, and the states a review must cover. Claims marked tested were run in Chrome 153 with Playwright 1.58. The eight one-off scripts from A and B were scratch files and were not kept; the behaviours the references rely on were rebuilt as `scripts/test_probe_live.py` (class `ClaimsWeRelyOn`) and pass. One behaviour has no kept test: Chrome wrapping full-page images taller than 16,384 device pixels. The skill's own probe, checker and capture driver are tested against `fixtures/broken.html`, a page with one planted defect per check.

## What the research found (the ten findings that shaped the skill)
1. **Screenshots are samples, and the default sample is the viewport at rest.** Playwright's full-page capture measures the page and captures beyond the viewport without scrolling, so lazy images far down are never requested, scroll-reveal sections stay hidden, fixed headers appear once, and inner scroll areas are not expanded (A, tested; S-R5-081). Playwright's `toHaveScreenshot` has `fullPage` off by default (C; S-R5-016).
2. **Tall images are shrunk past usefulness.** Claude resizes images (long edge 1568 px, 2576 px on the newest tier) and Anthropic says small detail is lost (A, C; S-R5-002, S-R5-003). Computed with Anthropic's reference function, a 1440 x 8000 page becomes about 464 x 2576, so 16 px text is drawn at about 5 px. Chrome also wraps full-page images over 16,384 device pixels with no error (A, tested once).
3. **Vision is weak at what review needs.** Four frontier models averaged 58% on simple geometry (BlindTest, S-R5-001); small icons and targets are the weak spot (best existing model 18.9% on ScreenSpot-Pro, S-R5-004); Design2Code finds missing elements and layout the weakest part (S-R5-005); color understanding is under-served (S-R5-006). Anthropic's docs call coordinates and counts of many small objects approximate. So numbers must come from the DOM.
4. **Measuring beats looking.** Practitioners report `scrollWidth > clientWidth`, bounding boxes and console errors catch bugs that screenshots hide (S-R5-007, S-R5-017: anecdotal, single sources). Negative-margin overflow to the left never raises `scrollWidth` (A, tested), and `document.fonts.check()` is true for a family that does not exist (A, tested; S-R5-054), so the probe measures each element's box and asks Chromium which font was drawn.
5. **The doer is a poor grader.** Models favor their own output (S-R5-008) and cannot self-correct without outside feedback (S-R5-009). Anthropic's harness work found a separate Playwright-driven evaluator, tuned with worked examples, easier to make skeptical than a self-critical generator, and an untuned one talked itself into approving (S-R5-010). Claude Code's guidance: give a check the agent can run, show evidence, use a fresh-context verifier, flag only gaps that affect correctness (S-R5-012). The skill's report format and verifier prompt follow this.
6. **Motion is invisible to a still, and half of it cannot be listed.** `document.getAnimations()` covers CSS animations, CSS transitions and Web Animations, and a finished transition vanishes from it (B, tested). Pause-and-seek (`currentTime`) gives repeatable frames (B, tested; S-R5-059). Canvas, WebGL, video, SMIL, GIF and `requestAnimationFrame` motion are not listed and need frames on a timer or their own seek calls (GSAP, Lottie, `setCurrentTime` on SVG). Claude reads only the first frame of an animated GIF (S-R5-002). Playwright's `animations: "disabled"` hides the middle (S-R5-067).
7. **Some motion checks are objective.** Over 5 s with no pause control (WCAG 2.2.2, level A), more than 3 flashes a second (2.3.1), interaction-triggered motion that cannot be disabled (2.3.3, AAA; the house floor treats it as required). Apple's HIG gives no numeric durations; Material's tokens run 50 to 1000 ms; NN/g says about 100 ms for feedback and 200 to 300 ms for modals (B; an earlier summary that quoted Apple numbers was wrong and was corrected).
8. **A scanner is not a review.** Deque's study found 57% of issues by count; the UK government audit of 13 tools on 142 planted barriers found the best at 40% and axe-core at 29%; axe has one WCAG 2.2 rule (target size) and none for reflow, focus obscured or keyboard traps (D; S-R5-037 to S-R5-039). The skill runs those checks itself: reflow at 320 px, text spacing, 200% text, Tab walk with an `elementFromPoint` test for focus covered by a sticky bar, forced colors.
9. **States and conditions most reviews skip** (D): hover, focus-visible, active, disabled; loading, empty, error, one and many items; long and unbroken strings; text expansion (W3C: strings up to 10 characters can grow 200 to 300%, over 70 about 30%); right-to-left and logical properties; touch has no hover; safe-area insets; the on-screen keyboard; reduced motion, forced colors, dark mode; Safari's `100vh`. Design-QA checklists from tool vendors agree on tokens over hex, computed spacing against the scale, loaded fonts, and per-component states (S-R5-029 to S-R5-031).
10. **A phone is not a narrow window.** Without mobile emulation a narrow viewport ignores the viewport meta tag; with it and no tag the layout width was 980 px (A, tested). In a phone profile `window.innerWidth` grows to fit overflowing content (found while testing the skill on a real page: 390 px reported as 1560), so the probe reads the layout viewport.

## What is unsourced or only inferred
Marked in the catalogue (`inferred: true` on 17 of 65 cases, and `[inferred]` inline where one step in a sourced case is reasoning): the 6 to 12 frame sampling plan; the half-height viewport as an on-screen keyboard stand-in; the Windows font-rendering and sticky-in-overflow Safari claims (dropped, no primary source); the 45 to 75 character line-length range (WCAG gives only 80 as AAA); the `[inferred]` finding-record schema; iOS form zoom below 16 px (one old article, snippet only; S-R5-090); that Motion's spring values do not appear in `getAnimations()` (the docs are silent). The three design-QA checklists are all from tool vendors.

## Verification
Three fact-checking agents, each in a fresh context that had not seen the work, re-opened the cited pages on 2026-10-04 and checked every sourced claim in the catalogue and in `capture.md`, `motion.md` and `report.md`: one for coverage and time (50 claims), one for states, measurement and interaction (55 rows), one for environment, process and the two reference documents (65 rows). Claims marked tested were not re-checked by them; the browser behaviours behind those were rebuilt as tests (above).

| Pass | Supported | Partly | Not supported | Unmarked (fact with no source and no inferred tag) |
|---|---|---|---|---|
| Coverage and time | 34 | 13 | 3 | included in the 3 |
| States, measure, interaction | 29 | 21 | 5 | included in the 5 |
| Environment, process, reference docs | 49 | 14 | 0 | 2 |

Everything flagged was corrected in `synthesis/review-cases.json` or the reference documents, or marked inferred. The corrections that changed meaning, not only wording:
- **Wrong:** the browser default for an `h3` is 18.72 px, not 17.6 (measured in Chrome 154); a 600-weight request with regular and bold loaded is drawn with the real 700 face, not faked; Apple's 44 pt is the default control size and its minimum is 28 pt (Android recommends 48 dp); W3C's text-expansion table gives the new length as a percentage of the old, so short strings end up 200 to 300% of the English length; Thai needs more vertical space, not more width.
- **Overstated:** WCAG 2.4.11 (AA) fails only when the focused control is entirely hidden; fully visible is the AAA criterion 2.4.12, so partly covered is now a warning citing 2.4.12; WCAG 2.2.2 and 2.3.1 have conditions (content beside other content, an essential exception, flash thresholds) that the first wording dropped; a forced-colors page loses shadows and gradient backgrounds, not `url()` images; `getAnimations()` forgets a finished transition, and a fixed wait is not reliable, so wait for `transitionend`; `animations: "disabled"` is the default for `toHaveScreenshot`, not for `page.screenshot`; axe-core's one WCAG 2.2 rule (target size) is off by default, and the "29%" in the UK audit is from a 2017-2018 axe-core.
- **Unsourced, now marked or sourced:** "most defects sit on pages nobody opened", the 320 to 480 px filmstrip cell, "browser tools do not zoom", the 2 px offset arithmetic (about 1.5 px after a 1920 to 1456 px shrink on standard-tier models), the logo-flipping rule.
- **A real bug in the skill, found by the checkers:** the probe flagged only `maximum-scale=1` as blocking zoom, but WCAG needs 2x, so `maximum-scale=1.5` passed. It now flags any cap below 2, with a test.
- **Capture claim kept but relabelled:** Chrome's 16,384-pixel limit. The cited issues cover only WebP; the wrapping is this lane's own test and is now labelled that way.

## Open questions / gaps
- **Owner decision, not made:** `scripts/capture.py` imports Playwright and `review.py serve` listens on 127.0.0.1. Both go against "skills stay offline and standard-library only" (SPEC, REPO-PLAN, SECURITY.md); `tools/scan_pr.py` reports 2 BLOCK and the helpers are not committed. See `_coordination/DECISIONS.md` (2026-10-04).
- Native apps (iOS, Android, desktop): the cases apply, `probe.js` does not. The skill says so and points to the simulator tools; no native probe exists.
- Browsers other than Chromium: the probe is plain DOM and should run anywhere, and `capture.py` is Chromium only (fonts via CDP). Safari and Firefox were not tested.
- Cross-origin iframes are counted, not measured. Closed shadow roots are not entered.
- Trigger quality: whether agents pick this skill for "review my UI" rather than the router's design-system critique has not been measured (SPEC section 13 trigger evals).
- The router row in `skills/opendesigner/SKILL.md` is not added yet: that file has another session's uncommitted edits.
- A real trial on a person's app, with a person reading the report, has not happened (`docs/FIELD-PLAN.md`).

## Confidence
High that the probe measures what it says (54 offline and 21 browser tests; the planted-defect page and the three example previews). High on the standard thresholds (checked by the verification pass below). Medium on how well the verifier prompt works in practice: it follows published guidance but has not been run on a real review. Low-medium on the drift checks being right for every system: they compare computed styles with tokens and treat browser defaults as drift, which is correct but noisy on raw pages.