Seo Firecrawl

---
name: seo-firecrawl
description: Site crawling, mapping, and JS-rendered scraping via the Firecrawl MCP when connected; otherwise builds a URL inventory from the sitemap and does targeted page fetches with the built-in fetcher, naming what a full crawl would add. Trigger when the user says "crawl site", "map site", "full crawl", "find all pages", "broken links", "site structure", "discover pages", "JS rendering", "scrape with javascript".
---

# seo-firecrawl

**Family:** seo
**Status:** Stable

## Purpose

Site-level discovery and scraping. **Tool-aware:** with the Firecrawl MCP connected
it does true multi-page crawling and JS-rendered scraping (for sites where a plain
HTTP fetch returns an empty shell); without it, it builds a URL inventory from the
sitemap and fetches the pages you point it at — a real, useful map on the free path.

## Triggers

- "crawl site" / "full crawl"
- "map site" / "site structure"
- "find all pages" / "discover pages"
- "broken links"
- "JS rendering" / "scrape with javascript"

## Inputs

- Starting URL (and sitemap URL if known)
- Crawl depth / page cap (Tier 1) or the specific pages to fetch (Tier 2)
- Output format (URL inventory | per-page markdown | broken-link list)

## Steps

1. **Detect tooling.** Check whether the Firecrawl MCP is exposed this session
   (run `python3 "${CLAUDE_PLUGIN_ROOT}/scripts/workflow/capability_probe.py"` — it also reports `FIRECRAWL_API_KEY`).
2. **Crawl path (Tier 1, if connected).** Run the Firecrawl MCP crawl/map/scrape for
   the requested depth; return the URL inventory + per-page markdown + broken links.
3. **Built-in path (Tier 2, always available).** Fetch `sitemap.xml` (or the given
   sitemap) and run `python3 "${CLAUDE_PLUGIN_ROOT}/scripts/seo/sitemap_tools.py" --validate` to get a structured URL
   inventory; fetch the priority pages with WebFetch; flag fetch failures as likely
   broken links. State that JS-rendered-only content and large-scale crawls need Tier 1.
4. **Report which tier ran** and what a full Firecrawl crawl would add.

## Capability routing

This skill follows the plugin's capability-tier cascade
(`references/CAPABILITY-TIERS.md`):

1. **Tier 1 — Firecrawl MCP.** If connected, use it for multi-page crawl, JS
   rendering, and batch scrape.
2. **Tier 2 — built-in (default).** Otherwise `sitemap_tools.py` for the URL
   inventory + WebFetch for targeted pages. Produces a real site map for
   static/server-rendered sites.
3. **Tier 4 — guided.** If the site is JS-rendered and Firecrawl isn't connected,
   deliver the sitemap inventory and note that full rendering needs Firecrawl (or
   the Playwright extension for interaction-heavy pages).

Always state which tier ran and what a full crawl would add.

## Outputs

- URL inventory (from crawl on Tier 1; from sitemap on Tier 2)
- Per-page content as markdown for the pages retrieved
- Broken-link list (crawl-wide on Tier 1; from fetch failures on Tier 2)

## Dependencies

- Optional (Tier 1): Firecrawl MCP (`extensions/firecrawl/`) — adds full crawl + JS
  rendering; free path works without it
- Built-in (Tier 2): WebFetch; `scripts/seo/sitemap_tools.py`
- `scripts/workflow/capability_probe.py`

## Notes

Backbone of `seo-audit` and `design-research` when a full crawl is needed. For pages
that require interaction beyond rendering, pair with the Playwright extension.