Data Scraper Agent

---
name: data-scraper-agent
description: Build a fully automated AI-powered data collection agent for any public source — job boards, prices, news, GitHub, sports, anything. Runs on a schedule, enriches data with a free LLM (Gemini Flash), stores results in Notion/Sheets/Supabase, and learns from user feedback. Runs 100% free on GitHub Actions. Use when the user wants to monitor, collect, or track any public data automatically.
metadata:
  origin: community
---

# Data Scraper Agent

Build a production-ready, AI-powered data collection agent for any public data source.
Runs on a schedule, enriches results with a free LLM, stores to a database, and improves over time.

**Stack: Python · Gemini Flash (free) · GitHub Actions (free) · Notion / Sheets / Supabase**

## When to Activate

- User wants to gather or monitor any public website or API
- User says "build a bot that checks...", "monitor X for me", "collect data from..."
- User wants to track jobs, prices, news, repos, sports scores, events, listings
- User asks how to automate data collection without paying for hosting
- User wants an agent that gets smarter over time based on their decisions

## When NOT to Use

- One-off scraping scripts (plain Python + requests/BeautifulSoup is enough)
- Paid SERP/API pipelines (use `deep-research`, `exa-search`)
- Video/transcript pipelines (use `claude-video`, `pipeline-video-agente`)

## Contents

| Topic | Reference |
|---|---|
| Layers, free stack, batching | `references/concepts.md` |
| Steps 1-2 (goal, architecture) | `references/workflow-design.md` |
| Steps 3-5 (connector, AI client, pipeline) | `references/connectors-ai.md` |
| Steps 6-8 (feedback, storage, main) | `references/feedback-storage.md` |
| Steps 9-10, patterns, limits | `references/ops-patterns.md` |

## Quality Checklist

Before marking the agent complete:

- [ ] `config.yaml` controls all user-facing settings — no hardcoded values
- [ ] `profile/context.md` holds user-specific context for AI matching
- [ ] Deduplication by URL before every storage push
- [ ] Gemini client has model fallback chain (4 models)
- [ ] Batch size ≤ 5 items per API call
- [ ] `maxOutputTokens` ≥ 2048
- [ ] `.env` is in `.gitignore`
- [ ] `.env.example` provided for onboarding
- [ ] `setup.py` creates DB schema on first run
- [ ] `enrich_existing.py` backfills AI scores on old rows
- [ ] GitHub Actions workflow commits `feedback.json` after each run
- [ ] README covers: setup in < 5 minutes, required secrets, customisation

---

## Real-World Examples

```
"Build me an agent that monitors Hacker News for AI startup funding news"
"Scrape product prices from 3 e-commerce sites and alert when they drop"
"Track new GitHub repos tagged with 'llm' or 'agents' — summarise each one"
"Collect Chief of Staff job listings from LinkedIn and Cutshort into Notion"
"Monitor a subreddit for posts mentioning my company — classify sentiment"

## Camada anti-bot + fallback (leva YouTube rodada 8)

Para alvos anti-bot (X/Reddit/YouTube 403, layout que muda): mantenha 2 rotas por fonte (primária + fallback) e detecte quebra por seletor ausente, não por erro genérico. Quando a plataforma muda o layout, só a rota afetada é reescrita — o pipeline continua. Para gating de tasks pagas/bloqueadas, use pause-gate (pausa antes de submit) e para tutoria, modo explica-passo + gera-similar + corrige. Ver rowser-qa para verificação visual.