Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations
Scanned 9/7/2026
Install to Claude Code
Skills Directorynpx -y skills add bytesagain/ai-skills --skill crawler --agent claude-codeInstalls into .claude/skills of the current project.
Are you the author of Crawler?
Add the live security badge to your README — it updates automatically with every re-scan.
[](https://www.skillsdirectory.com/skills/bytesagain-crawler)More formats (shields.io, HTML) on the badges page.
---
name: "crawler"
version: "3.0.0"
description: "Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations"
author: "BytesAgain"
homepage: "https://bytesagain.com"
source: "https://github.com/bytesagain/ai-skills"
tags: [web-scraping, scrapy, crawler, robots-txt, selenium]
category: "devtools"
---
# Crawler
Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations. No API keys or credentials required — outputs reference documentation only.
## Commands
| Command | Description |
|---------|-------------|
| `intro` | Crawling vs scraping, robots.txt, sitemap |
| `standards` | HTTP caching, structured data, meta tags |
| `troubleshooting` | Anti-bot detection, JS rendering, encoding |
| `performance` | Concurrency, dedup, incremental, distributed |
| `security` | Legal landscape, ethical guidelines, proxies |
| `migration` | BeautifulSoup to Scrapy, requests to Playwright |
| `cheatsheet` | Scrapy commands, CSS/XPath, curl, user-agents |
| `faq` | Legality, JS pages, blocking, storage |
## Output Format
All commands output plain-text reference documentation via heredoc. No external API calls, no credentials needed, no network access.
---
*Powered by BytesAgain | bytesagain.com | hello@bytesagain.com*
Is this your skill, or is something wrong with this listing? Request removal or report an issue. Author removals are honored within 72 hours.
No comments yet. Be the first to comment!
Create Mermaid diagrams for flowcharts, sequences, ERDs, and
Understand the components, mechanics, and constraints of context in agent systems. Use when designing agent architectures, debugging context-related failures, or optimizing context usage.
Draft release notes and changelog entries from git history or merged PRs between two refs (tags/SHAs/branches), including breaking changes, migrations, and upgrade steps. Use when the user asks for release notes, changelog updates, or a GitHub Release draft.
Documentation style guide enforcer by @planetabhi. Applies and reviews the writing style guide when authoring or editing product documentation and tutorials. Use to check prose for voice, tense, word choice, inclusive language, formatting, code block, UI, Markdown, and number/date conventions.
Quick-reference card for all caveman modes, skills, and commands. One-shot display, not a persistent mode. Trigger: /caveman-help, "caveman help", "what caveman commands", "how do I use caveman".
