Manages Scrapy spider deployments via Scrapyd API with custom item pipelines for MongoDB ingestion, deduplication via MinHash LSH, and rotating proxy middleware configuration.
36 stars
0 votes
0 copies
2 views
Added June 2, 2026
ai-agentspythongobashgitapisecuritydocumentation
Works with
claude code
api
Security analysis
A92/100
mediumInstalls packages at runtime which could introduce malicious dependencies
Installs into .claude/skills of the current project.
Are you the author of Scrapy Pipeline Manager?
Add the live security badge to your README. It updates with every re-scan.
[](https://www.skillsdirectory.com/skills/agentskillexchange-scrapy-pipeline-manager)
---
name: "Scrapy Pipeline Manager"
slug: "scrapy-pipeline-manager"
description: "Manages Scrapy spider deployments via Scrapyd API with custom item pipelines for MongoDB ingestion, deduplication via MinHash LSH, and rotating proxy middleware configuration."
github_stars: 61314
verification: "security_reviewed"
source: "https://github.com/scrapy/scrapy"
author: "scrapy"
category: "Research & Scraping"
framework: "Claude Code"
tool_ecosystem:
github_repo: "scrapy/scrapy"
github_stars: 61314
---
# Scrapy Pipeline Manager
Manages Scrapy spider deployments via Scrapyd API with custom item pipelines for MongoDB ingestion, deduplication via MinHash LSH, and rotating proxy middleware configuration.
## Installation
Use the upstream install or setup path that matches your environment:
- pip install scrapy
Requirements and caveats from upstream:
- :alt: Supported Python Versions
- It is cross-platform, and requires Python 3.10+. It is maintained by Zyte_
Basic usage or getting-started notes:
- .. code:: bash
- And follow the documentation_ to learn how to use it.
- .. _documentation: https://docs.scrapy.org/en/latest/
- Source: https://github.com/scrapy/scrapy
- Extracted from upstream docs: https://raw.githubusercontent.com/scrapy/scrapy/HEAD/README.rst
## Source
- [Agent Skill Exchange](https://agentskillexchange.com/skills/scrapy-pipeline-manager/)
