Octocode Scraping

---
name: octocode-scraping
description: "Use when extracting or mapping public web content into a local cited corpus: scrape or crawl a URL/docs site, pull tables/pricing/product fields, diagnose blocked or thin pages, or answer from saved pages. Phrases like scrape this URL, crawl the docs, build a corpus, extract pricing. Prefer keyless fetch; ask before hosted spend. Live clicks/HAR/perf → octocode-chrome-devtools."
---

# Octocode Scraping

Flow: `FRAME → POLICY → ROUTE → FETCH → CORPUS → SEARCH → CITE → RECOVER`.

FRAME before the first fetch: fix target URL/domain, goal, depth, and output shape — vague ask → `references/user-inputs.md`.

Defaults: one public URL, `--mode html`, omit `--provider` (keyless `cdp`→`direct`), session `.octocode/tmp/scrape/{sessionId}`, compact stdout. Search corpus before refetch. Live interaction → chrome-devtools on one port, then `har-ingest` + `corpus-run` into the same session. Ask before auth, hosted spend, crawl widen, CAPTCHA/MFA, destructive actions. Cite paths + URL metadata — not raw dumps.

Stop when: two same-class failures (report evidence, route tried, sanitized status, next approval); hosted `403` (wrong key or credits gone — status only, no retry); CAPTCHA/MFA, auth wall, or cookie/profile transfer needed; still blocked after one `cdp` try (ask before `--provider scrapingant`); personal data, form submits, purchases, sends, deletes, or account changes in scope; the saved corpus already proves the claim (cite it, do not refetch); crawl widen before `reports/summary.md` is useful. Recovery table: `references/failure-recovery.md`.

## Route (pick one)
| Need | Do | Skip |
|---|---|---|
| Vague scrape | `--mode html`, omit `--provider` | markdown / auto hosted |
| See auto pick | `scripts/provider-check.mjs` | guessing |
| Fetch/crawl | `scripts/fetch.mjs` | deprecated `scrapingant-fetch` |
| Corpus on disk | `corpus-inspect` → find helpers → `corpus-run` | blind refetch |
| Live click/DOM/auth | chrome-devtools | `--provider cdp` alone for interaction |
| Page health | chrome measure + `measure-query` | hosted scrape for scores |
| CDP → corpus | `scripts/har-ingest.mjs --from-cdp-dir` | new sessionId |
| Prove field | `scripts/corpus-run.mjs --regex\|--script` | reopen browser |
| Still blocked | evidence + ask → `--provider scrapingant` | silent spend |

## Scripts (Node, no install; every one takes `--help`)
| When | Run |
|---|---|
| fetch / crawl / extract a URL — the owner of every network call | `scripts/fetch.mjs --url <u> [--mode html] [--crawl --same-domain --max-pages <n>] [--no-raw]` |
| want the fetch plus an immediate corpus brief in one shot | `scripts/fetch-and-brief.mjs --url <u>` (wraps `fetch.mjs` → `corpus-inspect.mjs`) |
| before routing or hosted spend: which provider auto-wins, credits left | `scripts/provider-check.mjs [--provider <p>]`, `scripts/provider-usage.mjs` — sanitized, never prints the key |
| read a saved session first, then search it | `scripts/corpus-inspect.mjs --session-dir <d> [--page <n>]`, then `scripts/corpus-find.mjs --session-dir <d> --query <t>` |
| pull static DOM, assets, or graph paths out of the corpus (live DOM → chrome-devtools) | `scripts/dom-find.mjs --kind form/button/table`, `scripts/resource-list.mjs --kind asset/external`, `scripts/graph-navigate.mjs --from <nodeId>` — each with `--session-dir <d>` |
| prove one field locally instead of reopening a browser | `scripts/corpus-run.mjs --session-dir <d> --roots cdp,extracts --regex <re>` (or `--script <file>`, `--concat-parts`) |
| bridge chrome-devtools artifacts into this session, or hand a packet back | `scripts/har-ingest.mjs --session-dir <d> --from-cdp-dir <run>`; reverse with `--export-packet` |
| need field names before an extraction | `scripts/schema-helper.mjs --intent "extract pricing and features"` |
| an old transcript uses the legacy names | deprecated shims `scripts/scrapingant-fetch.mjs`, `scripts/scrapingant-check.mjs`, `scripts/scrapingant-usage.mjs` forward to `fetch.mjs` / `provider-check.mjs` / `provider-usage.mjs` — call the new names |
| editing or extending a script | shared modules in `scripts/lib/` (`providers` registry, `client` fetch, `corpus`, `analyzers`, `extractors`, `text`, `args`, `bridge`); env/key resolution vendored in `scripts/octocode-config.mjs`; JSON contracts in `scripts/schemas/` |

Full flags, roles, and the library map: read `scripts/README.md` before adding a flag or a vendor.

## References
| When | Load |
|---|---|
| frame a vague scope | `references/user-inputs.md` |
| legal/safety/privacy policy | `references/scraping-policy.md` |
| cost / keyless vs hosted route | `references/route-selection.md` |
| provider registry / add a vendor | `references/providers.md` |
| hosted API, after approval | `references/scrapingant.md` |
| search a saved session corpus | `references/session-corpus.md` |
| site graph / navigation / workflows | `references/website-analysis.md` |
| stdout and corpus file shapes | `references/data-contract.md` |
| extract quality and citations | `references/extraction-quality.md` |
| live browser bridge playbook | `references/browser-scraping.md` |
| blocked, thin, or oversized results | `references/failure-recovery.md` |