Sub-skill de agent-ready-cloudflare: Implement robots.txt
Scanned 9/5/2026
Install to Claude Code
Skills Directorynpx -y skills add fabricioctelles/skills --skill robots-txt --agent claude-codeInstalls into .claude/skills of the current project.
Are you the author of Robots Txt?
Add the live security badge to your README — it updates automatically with every re-scan.
[](https://www.skillsdirectory.com/skills/fabricioctelles-robots-txt)More formats (shields.io, HTML) on the badges page.
---
name: agent-ready-robots-txt
description: >
Sub-skill de agent-ready-cloudflare: Implement robots.txt
---
# Implement robots.txt
Publish a valid robots.txt at your site root per
[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309).
## Requirements
- Serve `/robots.txt` as `text/plain` with HTTP 200
- Include `User-agent` directives with `Allow`/`Disallow` rules
- Reference your sitemap if one exists: `Sitemap: https://example.com/sitemap.xml`
## Cloudflare
[AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/)
can manage your robots.txt from the dashboard, including AI-specific bot rules.
## Validate
```
POST https://isitagentready.com/api/scan
Content-Type: application/json
{"url": "https://YOUR-SITE.com"}
```
Check that `checks.discoverability.robotsTxt.status` is `"pass"`.
Is this your skill, or is something wrong with this listing? Request removal or report an issue. Author removals are honored within 72 hours.
No comments yet. Be the first to comment!
