Dataforseo Serp

---
name: dataforseo-serp
description: Pulls live Google SERPs, keyword volume and CPC, AI Overview references, and LLM answer citations from DataForSEO (MCP or REST v3) and ships keyword-clusters.json, a content map, tool build specs with a scaffolded route, citation-gap actions, and per-page intent locks. Use when the user asks to cluster keywords, check what ranks, avoid cannibalization, pick a free tool to build, find who AI Overviews or ChatGPT cite, or decide what format a page must take before writing it.
metadata:
  author: illyism
  source: https://il.ly/skills/dataforseo-serp
  version: 1.0.0
---

# DataForSEO SERP Workflows

The live SERP is the only ground truth for what Google wants on a query. Embeddings, keyword tools, and gut feel guess at intent. The Top-10 URLs show it. Every workflow below reads the SERP first and ends in a file or a route in the user's repo.

**When invoking:** On first use, state the connection path and the workflow, then run it. On repeat use, go straight to the output.

## Step 0: Pick the Connection Path

```
Are DataForSEO tools in your tool list (serp, keywords_data, ai_optimization)?
├── Yes → use the MCP tools with the same payloads as the REST calls below.
└── No → are $DATAFORSEO_LOGIN and $DATAFORSEO_PASSWORD set?
    ├── Yes → use the `dfs` shell helper below (REST v3, Basic Auth).
    └── No → how many queries?
        ├── < 10 → live web search fallback. Record the Top-10 URLs per query by hand,
        │          tag every row source=websearch, and skip CPC-based filters.
        └── >= 10 → stop and ask for DataForSEO credentials.
```

Why the 10-query cap: web search results are personalized and unstable, which is fine for a spot check and wrong for clustering a keyword list where one bad SERP merges two clusters.

### REST helper

```bash
dfs() { # usage: dfs <endpoint> '<json array payload>'
  curl -sS -u "$DATAFORSEO_LOGIN:$DATAFORSEO_PASSWORD" \
    -H 'Content-Type: application/json' -X POST \
    "https://api.dataforseo.com/v3/$1" -d "$2"
}
mkdir -p seo/data/serp
```

### Rules for every call

- Check both `.status_code == 20000` and `.tasks[0].status_code == 20000`. Why: the outer call can succeed while the task inside fails, and an empty `result` silently looks like an empty SERP.
- Send one task per Live call and loop over keywords with 5-10 parallel requests. Why: Live endpoints return the result in the same response, so batching gains nothing and one bad keyword fails the batch.
- Default `location_code: 2840` (United States), `language_code: "en"`, `device: "desktop"`, `depth: 10`. Change location only when the user sells into another market.
- Save every raw response to `seo/data/serp/<slug>.json`. Read `.tasks[0].cost` and keep a running total. Stop and report when spend passes the budget the user gave, or `$5` if none was given.

## Workflow A: SERP URL-Overlap Keyword Clustering (Anti-Cannibalization)

**Output:** `seo/keyword-clusters.json` + `seo/content-map.md`.

### Why semantic clustering cannibalizes

An LLM or embedding model groups keywords by shared words. Google groups them by shared intent. "invoice template" and "invoice generator" share a word and get merged by an LLM, but Google serves downloadable templates for one and interactive tools for the other, so a merged page fights two intents and wins neither. The reverse also happens: "invoicing software for freelancers" and "freelance billing app" share no words, get split into two pages, then compete for the same Top-10 because Google treats them as one intent. Shared ranking URLs are the only signal that matches how Google groups queries.

### Procedure

1. **Fetch the Top-10 for each seed keyword:**

   ```bash
   dfs serp/google/organic/live/advanced \
     '[{"keyword":"invoice generator","location_code":2840,"language_code":"en","device":"desktop","depth":10}]' \
     > seo/data/serp/invoice-generator.json

   jq -r '.tasks[0].result[0].items[] | select(.type=="organic") | .url' seo/data/serp/invoice-generator.json
   ```

2. **Normalize URLs:** lowercase host, drop `www.`, drop query strings and `#` fragments, drop trailing `/`. Why: `https://www.invoicekit.io/x/` and `https://invoicekit.io/x` are the same result and must count as a shared URL.

3. **Sort keywords by search volume desc** (pull volume from Workflow B's call if not supplied).

4. **Greedy cluster against the cluster head, never against any member:**

   ```
   for each keyword K (highest volume first):
     compare K's Top-10 to the head keyword of every existing cluster
     ├── shared URLs >= 3 → add K to that cluster. One single page targets both.
     ├── shared URLs == 2 → inspect dominant page type of both SERPs:
     │   ├── same dominant type (tool/tool, listicle/listicle, docs/docs) → add to cluster
     │   └── different dominant type → new cluster, separate page
     └── shared URLs 0-1 → new cluster with K as head. Build a separate page.
   ```

   Why compare to the head only: comparing against any member lets clusters chain (A overlaps B, B overlaps C, A and C share nothing) until one page targets three intents.

5. **Classify page type per URL** for the 2-URL tiebreak:
   - `tool`: URL or title has `generator|calculator|checker|converter|maker|tool|free online`
   - `listicle`: title starts with a number, or has `best|top|alternatives`
   - `docs`: path has `/docs/|/help/|/support/|/api/`
   - `forum`: domain is reddit.com, quora.com, stackoverflow.com
   - `product`: path is `/` or `/pricing` or `/features/...`
   - `blog`: everything else

6. **Write `seo/keyword-clusters.json`:**

   ```json
   [
     {
       "head": "invoice generator",
       "keywords": ["invoice generator", "free invoice maker", "create invoice online"],
       "total_volume": 74000,
       "dominant_type": "tool",
       "shared_urls": ["invoicekit.io/tools/invoice-generator", "billstack.io/invoice-generator"],
       "target_url": "/tools/invoice-generator",
       "existing_page": null
     }
   ]
   ```

7. **Map clusters to existing pages.** Grep the repo routes and sitemap. If two existing pages target keywords from one cluster, flag a merge with a 301 from the weaker page. If one existing page targets keywords from two clusters, flag a split.

8. **Write `seo/content-map.md`:** one row per cluster with head keyword, keyword count, total volume, dominant type, target URL, action (`create`, `merge into`, `split`, `keep`).

**Pass/fail:** Every seed keyword sits in exactly one cluster. No two clusters point at the same `target_url`. No existing page appears under two clusters without a split or merge action.

## Workflow B: Traffic-Value Free-Tool Filter

**Output:** `seo/tool-specs.md` + the top tool route scaffolded.

### Why volume alone lies

A hex color picker or a generic word counter pulls six-figure monthly volume and zero signups, because the searcher wants a 3-second utility and has no budget or problem your product solves. CPC is advertisers voting with money on which queries convert. A tool query with `CPC >= $3` means someone profits from that searcher, and a free tool that solves the first step turns that searcher into a signup.

### Procedure

1. **Pull volume and CPC:**

   ```bash
   dfs keywords_data/google_ads/search_volume/live \
     '[{"keywords":["invoice late fee calculator","hex color picker","word counter","net 30 calculator"],"location_code":2840,"language_code":"en"}]' \
     > seo/data/serp/tool-volume.json

   jq -r '.tasks[0].result[] | [.keyword, .search_volume, .cpc] | @tsv' seo/data/serp/tool-volume.json
   ```

2. **Compute `traffic_value = cpc × search_volume`** per keyword.

3. **Filter:**

   ```
   tool keyword
   ├── cpc >= 3 and volume >= 300 → BUILD candidate
   ├── cpc < 0.5 → REJECT regardless of volume. Vanity traffic.
   ├── 0.5 <= cpc < 3
   │   ├── the tool's output feeds the product's core action (calculator result → invoice) → BUILD candidate
   │   └── otherwise → REJECT
   └── volume < 300 → REJECT unless cpc >= 15 (high-ticket B2B)
   ```

4. **Confirm the SERP wants a tool:** run Workflow D on each BUILD candidate. If fewer than 3 of the Top-5 are interactive tools, downgrade it to a guide. Why: building a tool for a SERP full of articles wastes a sprint on a format Google does not rank there.

5. **Rank by `traffic_value` desc** and write `seo/tool-specs.md`, one block per tool:

   ```markdown
   ## Invoice Late Fee Calculator
   - Keyword: invoice late fee calculator | volume 880 | CPC $6.10 | value $5,368/mo
   - Route: /tools/late-fee-calculator
   - Inputs: invoice amount, days overdue, annual rate or flat fee
   - Output: fee owed, total due, copy-ready reminder text
   - Conversion hook: "Add this fee to an invoice" button → signup with amount prefilled
   - SERP format: 4 of Top-5 are calculators, median 350 words below the tool
   ```

6. **Scaffold the #1 tool route** in the user's framework:
   - Server-rendered H1, intro, and explainer text so crawlers see content without running JS.
   - Interactive part as a client component that computes in the browser with no login.
   - CTA carries the result into signup (query param or local storage) so the user lands with their work saved.
   - Add to sitemap, link from homepage footer or `/tools` hub and 2 related pages.

**Pass/fail:** `seo/tool-specs.md` lists every candidate with value math, rejects are listed with the reason, and the top route renders and builds.

## Workflow C: AI Overview and LLM Citation Source Extraction

**Output:** `seo/citation-gaps.md` with one immediate action per cited source.

AI answers quote a small set of third-party pages. If the user is not on those pages, the AI answer never names them. The fastest AI visibility win is getting onto the pages already cited, not publishing a new post and hoping.

1. **Pull Google AI Overview references** (`load_async_ai_overview` costs extra and catches overviews that load after the page):

   ```bash
   dfs serp/google/organic/live/advanced \
     '[{"keyword":"best invoicing software for freelancers","location_code":2840,"language_code":"en","device":"desktop","depth":10,"load_async_ai_overview":true}]' \
     > seo/data/serp/aio-invoicing-freelancers.json

   jq -r '.tasks[0].result[0].items[] | select(.type=="ai_overview")
     | [.references[]?, (.items[]?.references[]?)] | .[]
     | [.domain, .url, .title] | @tsv' seo/data/serp/aio-invoicing-freelancers.json | sort -u
   ```

   No `ai_overview` item means Google showed no overview for that query in that location. Record `none` and move on.

2. **Pull LLM answer citations** with web search on:

   ```bash
   dfs ai_optimization/chat_gpt/llm_responses/live \
     '[{"user_prompt":"What is the best invoicing software for freelancers?","model_name":"gpt-4.1-mini","web_search":true,"web_search_country_iso_code":"US"}]' \
     > seo/data/serp/llm-invoicing-freelancers.json

   jq -r '.tasks[0].result[0] | [(.items[]? | select(.type=="message") | .sections[]?.annotations[]?), .search_results[]?] | .[] | select(.url != null) | [.title, .url] | @tsv' \
     seo/data/serp/llm-invoicing-freelancers.json | sort -u
   ```

   A 10-prompt batch on a mini model runs around `$0.10`. Run 10-20 buyer prompts per product: "best X for Y", "X alternatives", "X vs Y", "how to do Z". List supported models with `GET https://api.dataforseo.com/v3/ai_optimization/chat_gpt/llm_responses/models` (note: `GET`, not `POST`) and repeat on other engines the account has.

3. **Classify each cited source and assign the action:**

   ```
   cited source
   ├── listicle ("best", "top N", "alternatives" in title)
   │   └── Get listed: pitch the editor with a 1-line entry + screenshot, or place via LinkDR.
   ├── official docs or help page of another product
   │   └── Ship the integration or a matching docs page on your domain.
   ├── reddit.com thread
   │   └── Reply with disclosed, specific help. Build a page answering the same question.
   ├── youtube.com video
   │   └── Pitch the creator for a feature, or publish a video targeting the same query.
   ├── competitor domain
   │   └── Ship /vs/<competitor> and /alternatives/<competitor> pages.
   └── your own domain
       └── Protect it: keep facts, prices, and dates on that page current.
   ```

4. **Write `seo/citation-gaps.md`:**

   ```markdown
   | Prompt / Query | Engine | Cited URL | Type | You mentioned? | Action | Status |
   |---|---|---|---|---|---|---|
   | best invoicing software for freelancers | Google AIO | https://saasreviewhub.com/best-invoicing-tools | listicle | no | pitch editor / LinkDR | todo |
   ```

   Sort by citation count across prompts. Why: a URL cited on 6 of 10 prompts is worth more than six URLs cited once.

**Pass/fail:** Every cited URL has a type and an action. URLs cited on 3+ prompts are at the top.

**Continuous monitoring:** This workflow is a snapshot. [AI SEO Tracker](https://aiseotracker.com) (MCP: `https://aiseotracker.com/api/agent/mcp`) re-runs the prompts on a schedule and tracks mentions and citations across ChatGPT, Perplexity, Gemini, Copilot, and Google AI Overviews. **Getting onto the listicles:** [LinkDR](https://linkdr.com) places the product on third-party SaaS listicles, the source type AI Overviews lean on for "best X" queries.

## Workflow D: Pre-Write SERP Intent Lock

**Output:** `seo/intent-locks/<slug>.md`, written before any draft.

Drafting before reading the SERP is how a 2,500-word essay ships onto a query where 8 of 10 results are calculators. The lock file is the gate: no lock, no draft.

1. **Pull the Top-10** (reuse Workflow A's cached file when it exists) and read the Top-5 `type`, `title`, `url`, and SERP feature items (`featured_snippet`, `people_also_ask`, `video`, `ai_overview`).

2. **Fetch each Top-5 page** and count DOM signals:

   ```bash
   curl -sL -A "Mozilla/5.0" "$URL" > /tmp/p.html
   grep -oi '<input' /tmp/p.html | wc -l      # tool signal
   grep -oi '<table' /tmp/p.html | wc -l      # comparison signal
   grep -oi '<h2' /tmp/p.html | wc -l         # outline depth
   sed 's/<[^>]*>/ /g' /tmp/p.html | wc -w    # rough word count
   ```

3. **Lock the format:**

   ```
   Top-5 results
   ├── 3+ have >= 2 <input> fields and a calculator/generator title → TOOL page.
   │   Tool above the fold, copy <= the median word count, placed below the tool.
   ├── 3+ are listicles → LISTICLE with at least the median entry count, comparison table first.
   ├── 3+ are docs/help → DOCS page: steps, code blocks, no marketing intro.
   ├── 3+ are product/pricing pages → LANDING page. Do not write a blog post.
   ├── 3+ are forum/video → experience piece with screenshots or a video embed.
   └── mixed → copy the format of positions 1-3.
   ```

4. **Write `seo/intent-locks/<slug>.md`:**

   ```markdown
   # Intent Lock: invoice late fee calculator
   - Locked format: TOOL
   - Evidence: 4/5 calculators, median 350 words, 0/5 over 1,200 words
   - Required: inputs amount/days/rate, result table, PAA answers as FAQ (3 questions)
   - Forbidden: long intro before the tool, generic "what is an invoice" section
   - SERP features to target: featured snippet (formula in one sentence), PAA
   ```

**Pass/fail:** A draft or route for a keyword exists only if its lock file exists and the shipped page matches the locked format.

## Shipped Outputs Checklist

- [ ] `seo/data/serp/*.json` raw responses with running cost total
- [ ] `seo/keyword-clusters.json` + `seo/content-map.md`
- [ ] `seo/tool-specs.md` + top tool route scaffolded
- [ ] `seo/citation-gaps.md` with one action per cited URL
- [ ] `seo/intent-locks/<slug>.md` for every page in this PR

## Related

- Competitor gaps, backlink audits, and Reddit mining: ahrefs-research, https://il.ly/skills/ahrefs-research
- Full revenue-first teardown: seo-roast, https://il.ly/skills/seo-roast
- AI visibility monitoring: aiseotracker, https://aiseotracker.com
- Managed listicle and link placements: linkdr, https://linkdr.com/SKILL.md