Pre Push Gate Debugging

---
name: pre-push-gate-debugging
description: "Use when a pre-push quality gate blocks git push or a lane fails: read the gate's own logs first (reproduce one lane), run single lanes for fast iteration, test the working tree you intend to push, handle E2E/infra cross-run state contamination, worktree node_modules gotchas, and build a manual repro harness when lane output hides the real error."
version: 1.0.0
author: ai-badger
license: MIT
platforms: [linux, macos, windows]
scope: default
metadata:
  hermes:
    tags: [gates, debugging, ci, pre-push]
    related_skills: [debug-issue, commit-reminder]
---

# Pre-push verification gate debugging

Repos with heavy local gates block `git push` until every lane passes (lefthook pre-push hook
calling a verify script that runs docs, unit, E2E, infra and frontend lanes). This skill is for
operating and debugging that gate without burning whole-gate cycles. The methods here
are generic; keep a repo-specific playbook beside the gate script (lane list + commands,
log layout, repro steps) as this skill's reference when one exists.

## Detect and understand the gate

- `lefthook.yml` → `pre-push` → `script` + `runner` (+ `root`). Read the script's header
  comments FIRST: usage, lane names, exit codes, env vars (`VERIFY_MODE`, `VERIFY_SKIP`),
  self-test, and the "reproduce / bypass / logs" failure contract.
- A blocked push ends with `🥊 quality gate (Ns)` and `exit status 1`. The hook buffers lane
  output — never diagnose from the push output alone.

## Workflow

1. **Read the gate's own logs first.** The failure block prints:
   - `reproduce: bun <script> <lane>` — run exactly this to re-run one lane (minutes, not the whole gate).
   - `logs: <dir>/<lane>.log` — the lane's full output; `<dir>/progress.log` is the run timeline
     (mode, which lanes ran, PASS/FAIL per lane).
   - Log dir is per-worktree (hash of the worktree path) — find it by mtime (`ls -t`), not by
     recomputing the hash.
2. **Run one lane** for fast iteration: `bun <script> <lane>` (e.g. `api-e2e`). Only re-run the
   full gate (`<script> pre-push`) when the lane is green.
3. **The gate tests the WORKING TREE, not the pushed ref.** If you push from a different
   checkout than the one you want verified, the lanes verify the wrong bytes. Gate-test the
   tree you intend to push.
4. **E2E/infra lanes**:
   - They start their own AppHost (Docker: emulator containers) OR **reuse a live dev stack**
     that is already ready (probed first). Reuse is convenient but carries state forward.
   - **Cross-run state contamination**: emulator data persists across suite runs on a reused
     stack. Durable orchestration instance IDs are PERMANENTLY TAKEN after a terminal state,
     so a later run scheduling the same deterministic ID 409s or silently dedupes. Symptom: a
     test passes alone, then a DIFFERENT test fails on a later full run with no code change.
     Fix: fresh ephemeral volumes (kill the reused stack, let the lane start its own).
5. **Worktree gotchas**: a bare `git worktree add` has NO node_modules → tooling lanes fail
   (`tsc` TS2688 "Cannot find type definition file for 'bun'", bun test resolution errors).
   Fix: symlink the main checkout's node_modules into the worktree. Never commit node_modules.
6. **Manual repro harness** (when the lane's test output hides the real error — e.g. an
   orchestration ends `failed` with an EMPTY error field):
   - Start the infra exactly as the lane does (see the reference / lane source for the AppHost
     command and env).
   - Build the host under test and run it with the E2E settings (the lane copies a
     `local.settings.e2e.json`-style file into the build output before the tests run).
   - Drive the failing flow with curl, sending the app's auth header (see reference).
   - Read the HOST's own logs: orchestration/activity failures appear there with the full
     exception ("Task 'X' failed with an unhandled exception: ..."), never in the HTTP status
     payload. `failed` + empty `error` in the API body = the detail is in the host logs.
7. **Auto-repair**: some gates try to regenerate drifted artefacts (baseline.json, index.json)
   before failing. If repair "declined", the cause is NOT drift — read the lane log.

## Gotchas
- **A push can LOOK hung while the pre-push hook is simply running.** `git push 2>&1 | tail` buffers everything, so the lefthook banner (`🥊 lefthook ... hook: pre-push`) never shows and the full gate (which can take 5+ minutes) looks like a network stall — zero output, no timeout. Diagnose by re-running the push UNPIPED or in background and reading the first lines: a banner means the gate is working, not hanging. If the user has said "skip gates" / "just push", `git push --no-verify` is the sanctioned bypass and returns instantly — do not keep waiting on a gate the user already waived.
- `full` vs `limited` mode changes which lanes run; limited skips infra/e2e and prints that
  coverage is still owed. Unset `VERIFY_MODE` for the real gate.
- `VERIFY_SKIP=<lane>` and `--no-verify` bypass the gate entirely — prefer fixing the lane.
- A 401 with "Missing or invalid x-functions-key" can really mean a missing auth header on your
  curl, not a missing key — check the app's principal/auth scheme before assuming.
- Don't re-run the full gate to test one fix — single-lane runs are minutes faster.
- Force-pushing a rebased PR branch re-triggers remote CI; the local gate still runs against
  your working tree, so keep the local tree on the branch you are pushing.
- **A green full-tree lint does NOT prove the changed-file lint lane passes.** If the gate's
  lint lane lints only changed files (git-diff scope) while your manual run covered every
  tracked file, the lane can flag findings the full run reports clean — e.g. a W0108
  `unnecessary-lambda` on a freshly added `resolve=lambda: f()` that a whole-repo
  `pylint $(git ls-files ...)` rated 10.00/10. Reproduce with the actual lane
  (`<script> <lane>`, e.g. `verify.sh pylint`) before pushing, not with a full-tree equivalent.
- **The interpreter you run the gate with determines what the gate's subprocesses can import.**
  Gates that spawn `sys.executable` (scaffold/regeneration lanes) inherit the python that
  launched the gate. On macOS the bare `python3` is often the CommandLineTools build (no
  site-packages, e.g. `ModuleNotFoundError: No module named 'jsonschema'`) — always run lanes
  with the repo's venv python, or rely on the gate's own `_resolve_python` fallback.
- **A worktree's own broken `.venv` shadows the main checkout's good one.** If the gate's
  python resolution prefers `$PWD/.venv` and the worktree has a bare/broken venv (e.g. no pip,
  no deps), every lane fails with import errors even though the main checkout's venv is
  populated. Fix: delete the worktree `.venv` so resolution falls back to the main checkout's
  venv. A stray venv you did not create is the first suspect when a fresh worktree's lanes
  fail on missing modules.
- **`git rebase --continue` hangs: the pre-commit hook wedges in rebase context.** Lefthook's
  pre-commit can hang indefinitely while a rebase is being continued (observed with the
  code-review-graph hook; the git process times out waiting on it and the rebase state survives
  the kill). Replay the rest of the rebase with hooks off:
  `GIT_EDITOR=true git -c core.hooksPath=/dev/null rebase --continue`. This is safe because the
  pre-push gate re-runs every relevant lane on the pushed tree — enforcement happens at push,
  not at rebase. A normal commit on a clean branch is unaffected; only the rebase context hangs.
- **The gate runs SCOPED lanes, not the full list.** `progress.log` shows e.g. `PUSH lanes: docs terraform`
  when a branch only touches docs+infra — the gate picks lanes by changed paths. Don't read a
  scoped run as proof the E2E/dotnet lanes passed, and don't be surprised a scoped run finished in
  seconds. `self-test` tooling tests ride inside the `docs` lane.
- **Playwright-style e2e lanes reuse a live dev server on a fixed port** (`reuseExistingServer:
  !CI`). A wholesale failure pattern — every page test timing out while the [WebServer] log floods
  `Cannot find module` / TS2307 for imports — is a SERVER-level failure, not your change. The
  reused server may be a zombie from a removed worktree (its node_modules symlink is gone), or the
  gate's own server hit a transient gap while a concurrent `npm install` in the main checkout
  churned package dirs. Diagnose in this order: (1) who owns the port — `lsof -nP -iTCP:<port>
  -sTCP:LISTEN`, then `lsof -p <pid> | grep cwd` (never kill a process whose cwd is the owner's
  checkout); (2) does the failing module exist on disk; (3) re-run the e2e lane alone for fast
  feedback; (4) re-push. Keep a repo-specific playbook (port, lane commands, worktree
  symlink quirks) beside the gate when the project has one.
- **After an auto-repair commit, push AGAIN.** The gate's repair path regenerates drifted
  artifacts (e.g. `docs/meta/baseline.json`, `trust-index.json`), commits
  `chore(gate): regenerate stale artifacts`, and exits non-zero with "push again" — the first
  push deliberately fails. Re-push the same ref (plain push; after a rebase, `--force-with-lease`).
- **A lane can be red because the gate tests a working tree polluted by ANOTHER task's
  uncommitted files — the repair path refuses to touch them, so the lane stays red until that
  task commits.** Repair declines with "not regenerating: <path> has uncommitted changes, and
  the repair may only commit bytes it wrote itself". This is not your diff. Prove it before
  bypassing: (1) your branch diff contains no file in the failing lane's scope
  (`git show --stat HEAD` / `git diff origin/main...HEAD`); (2) the lane's own output shows
  your files clean. Then the gate's failure block itself names the
  sanctioned escape: `VERIFY_SKIP=<lane> git push` — stage-scoped, NOT `--no-verify`
  (skips every lane) and NOT fixing the other task's files. The skipped lane still runs on
  GitHub CI only if CI has it; a docs lane is often local-only, so the PR is unaffected.
- **`git push --delete <branch>` also runs the pre-push gate** and can fail on it; bypass with
  `git -c core.hooksPath=/dev/null push origin --delete <branch>` — deleting a remote ref needs
  no verification.

## References

The gate script's own header + logs are the primary reference (usage, lane names, exit
codes, reproduce/bypass commands). Keep a repo-specific playbook beside the gate script
with the lane list and commands, log layout, AppHost command, E2E auth header + config,
manual repro steps, and any repo-local drift/state files — and update it when the gate
changes.