Scraper

Name: Scraper
Author: raven-sourav

ASecurity

Scrape newsletter posts from Substack and Beehiiv publications. No API keys needed.

3 stars
0 votes
0 copies
1 view
Added May 26, 2026

developmentpythonbashreactapi

Works with

Security analysis

A92/100

mediumInstalls packages at runtime which could introduce malicious dependencies

Pro scans all 3 files and shows the line behind each finding

Scanned May 27, 2026

npx -y skills add raven-sourav/pmm-content-engine --skill scraper --agent claude-code

Installs into .claude/skills of the current project.

Are you the author of Scraper?

Add the live security badge to your README. It updates with every re-scan.

[![Security: A — Skills Directory](https://www.skillsdirectory.com/api/skills/raven-sourav-scraper/badge)](https://www.skillsdirectory.com/skills/raven-sourav-scraper)

More formats (shields.io, HTML) on the badges page. Keep it an A: scan every change in CI with Pro.

Download with Pro

SKILL.md

# Skill: Newsletter Scraper

Scrape newsletter posts from Substack and Beehiiv publications. No API keys needed.

## Trigger
User wants to scrape, research, or pull posts from a newsletter URL.

## Supported Platforms

### Substack
Any Substack publication (subdomain or custom domain).

```bash
python3 skills/scraper/scripts/scrape_substack.py --url "https://newsletter.substack.com" --days 30 --output summary
python3 skills/scraper/scripts/scrape_substack.py --url "https://www.lennysnewsletter.com" --keywords "AI" --output json
python3 skills/scraper/scripts/scrape_substack.py --url "https://newsletter.substack.com" --full-content --output json
```

### Beehiiv
Any Beehiiv publication (including custom domains).

```bash
python3 skills/scraper/scripts/scrape_beehiiv.py --url "https://www.growthunhinged.com" --days 30 --output summary
python3 skills/scraper/scripts/scrape_beehiiv.py --url "https://www.growthunhinged.com" --keywords "pricing" --output json
python3 skills/scraper/scripts/scrape_beehiiv.py --url "https://www.growthunhinged.com" --full-content --output json
```

## CLI Flags (both scripts)

| Flag | Default | Description |
|------|---------|-------------|
| `--url` | *required* | Newsletter URL |
| `--keywords` | none | Filter by keywords (comma-separated, OR logic) |
| `--days` | all | Only include posts from last N days |
| `--max-posts` | all | Max posts to return |
| `--full-content` | false | Fetch full post body (slower) |
| `--output` | summary | `summary` (table) or `json` |

## How It Works

- **Substack**: Uses public `/api/v1/archive` JSON endpoint. Returns title, subtitle, word count, reactions, comments, tags, author.
- **Beehiiv**: Uses hidden `/posts` JSON endpoint. Returns title, subtitle, tags, reading time, author.

## Dependencies

Only `requests` — install via `pip install requests`.