Research Crawl Rss

---
name: research-crawl-rss
description: Batched RSS/Atom/RDF feed crawl from N feed-URLs via Python feedparser; per-feed markdown + manifest.json to caller dir; rolled-up + per-source summary.
---

# Research Crawl RSS

**Argument:** $ARGUMENTS (required, see Runtime Inputs)

If `$ARGUMENTS` is empty or missing required fields, STOP and report which fields are absent.

---

## Preamble

### Runtime Inputs

Parse from `$ARGUMENTS`:

```
--feed-urls=[u1,u2,...]          [JSON-style array of absolute feed URLs (RSS / Atom / RDF), non-empty; each must be an http(s) URL. Singular invocation expressed as a single-element array.]
--raw-output-dir={PATH}          [absolute path to a directory where per-feed rendered markdown files + manifest.json will be written]
--max-items-per-feed={N}         [positive integer per-feed item cap; MUST be ≥ 1, see §1.0 Step 3 + §6.0 row max-items-zero]

# Optional power params (omit when not needed):
--days={N}                       [positive integer recency window in days; default 1]
--max-body-chars={N}             [positive integer per-item body truncation cap; default 8000]
```

Validation rules (before §1.0):

- `--feed-urls`, `--raw-output-dir`, and `--max-items-per-feed` mandatory. If any missing, STOP and report which.
- `--feed-urls` must parse as a non-empty JSON array of strings. Else reject (`feed-urls-malformed`).
- Each entry must be an absolute `http://` or `https://` URL. First failing entry → STOP and report (`feed-urls-malformed`) with the offending value named.
- `--raw-output-dir` must be absolute. Reject relative paths (caller owns path resolution).
- `--max-items-per-feed` must parse as integer ≥ 1. Zero rejected per §6.0 `max-items-zero`.
- `--days` and `--max-body-chars` must parse as positive integers when supplied; defaults 1 and 8000.

If any validation fails, STOP and report which field failed.

### Global References

```
RSS_INGEST_SCRIPT = scripts/rss-ingest.py    # feedparser-via-Bash leaf engine; one batched run over all feeds
```

Leaf does not read tool guides at runtime, the documented pitfalls are already encoded inside `RSS_INGEST_SCRIPT` (browser-UA default load-bearing, `Accept-Encoding: identity` + gzip fallback, HTML-strip before boundary-aware truncation, per-feed cap + newest-first sort-then-slice as the token governor, recency-window filtering on `published`/`updated` dates, dateless-item fallback + `items_dropped_dateless` surfacing, 403 gating classification `cloudflare-waf` vs `cloudflare-js-challenge`, `bozo` + content-type sniff + empty-body soft-failure surfacing, title-only `body_sources:{none}` provenance, `est_tokens` chars/4 budgeting heuristic). RSS fetch is free, $0 per run; the per-feed cap is a context/token governor, not a cost cap.

### Caller Isolation

Pure primitive, no agent spawn, no pipeline agents, no telemetry. The skill runs the feedparser fetch + parse + render directly in whatever context invokes it; the caller absorbs the raw-feed context cost (raw feed items land in the caller's window). Callers crawling many feeds, or composing this leaf alongside other work, wrap the invocation in their own isolation:

```
Agent({subagent_type: "general-purpose", model: "sonnet", prompt: "Invoke /research-crawl-rss with <args>"})
```

Consumer skills document their own isolation choice. The run-pulse composer wraps each source-type leaf in one classifier sub-agent per source-type.

---

## 1.0 Context Capture

### Step 1: Verify raw output directory parent exists

Check the parent directory of `--raw-output-dir` exists. If not, route to §6.0 (`raw-output-parent-missing`). Do not create parents, caller owns path resolution. The `--raw-output-dir` itself MAY or MAY NOT exist; §3.0 Step 1 creates it via `mkdir -p` before the run.

### Step 2: Validate feed URLs and derive per-feed slugs

For each entry in `--feed-urls`: confirm it is an absolute `http(s)` URL. Build `feeds[]` preserving caller order. First entry that fails → route to §6.0 (`feed-urls-malformed`) with the offending value named.

Derive a deterministic slug per feed: lowercase the URL, drop the `http://` / `https://` scheme, replace each run of non-alphanumeric characters with a single `-`, strip leading/trailing `-`, truncate to 50 characters. If two feeds produce the same slug, append `-{ordinal}` to the later one so each feed maps to a distinct file. The slug becomes the feed's TSV name (and the script's output filename).

### Step 3: Validate `--max-items-per-feed` lower bound

Confirm `--max-items-per-feed ≥ 1`. If `--max-items-per-feed == 0`, route to §6.0 (`max-items-zero`): pass a positive integer. The cap is the effective token governor, the script sorts in-window items newest-first then slices to this count, so a large feed backlog clamps to a bounded payload. RSS fetch is free; this is a context cap, not a cost cap.

### Step 4: Build the feeds TSV

Write a tab-separated feed list for the script, one line per feed, `{slug}<TAB>{url}<TAB>` (empty domain column; the leaf does not classify or tag). Write it to a temp path via `mktemp` (e.g., `RSS_FEEDS_TSV=$(mktemp /tmp/rss-feeds-XXXXXX)`, no suffix after the X's; BSD/macOS `mktemp` rejects a template-then-suffix form like `-XXXXXX.tsv`, and the script reads `--feeds-file` by path, not by extension). The window (`--days`), per-feed cap (`--max-items-per-feed`), and body cap (`--max-body-chars`) pass directly to the script; no per-feed query construction is needed (feedparser fetches the whole feed and the script windows + caps internally).

---

## 2.0 Plan & Confirm

Skill runs autonomously, no user pause. Pure primitive; executes directly in the caller's context. Composed leaves never gate parallelization; direct invocations log the execution config and proceed.


State the planned execution config in one structured log statement, then proceed directly to §3.0:

```
## /research-crawl-rss: executing crawl

feeds:                 {feeds[]}                       # N feed URLs in caller-supplied order
raw_output_dir:        {--raw-output-dir}
max_items_per_feed:    {--max-items-per-feed}          # per-feed cap (token governor)
days:                  {--days}                        # default 1
max_body_chars:        {--max-body-chars}              # default 8000
engine:                {RSS_INGEST_SCRIPT}             # feedparser via Bash; $0/run
execution:             direct (pure primitive, no sub-agent spawn); one batched script run over all feeds
```

---

## 3.0 Crawl

Execute the crawl directly, no sub-agent spawn. Run `RSS_INGEST_SCRIPT` once over all feeds (it fetches, windows, caps, and renders per-feed markdown), then parse its manifest to build the per-source return. This step writes raw bytes only, it does NOT classify (which domain, which author), summarize, template-fill, deduplicate across feeds, or follow item links to recover missing bodies; the composing router or domain orchestrator handles all downstream transformation. Cross-feed dedup belongs to the dispatcher / classifier layer, not this leaf; title-only feeds whose bodies are genuinely absent surface `body_sources:{none}` and a downstream layer decides whether to fetch the full body from the item link.

### Inputs (from §1.0)

```
Feeds:                 {feeds[]}                        # N feed URLs in caller-supplied order
Feed TSV:              {RSS_FEEDS_TSV}                  # {slug}<TAB>{url}<TAB> per feed
Raw output dir:        {--raw-output-dir}
Max items per feed:    {--max-items-per-feed}
Window:                last {--days} day(s)
Engine:                {RSS_INGEST_SCRIPT}
```

### Procedure

Execute these steps in order:

```
1. Ensure {--raw-output-dir} exists. Use Bash `mkdir -p {--raw-output-dir}`.

2. Write the feeds TSV from §1.0 Step 4 to {RSS_FEEDS_TSV}.

3. Run the engine once, capturing the manifest to disk:

     python3 {RSS_INGEST_SCRIPT} \
       --feeds-file={RSS_FEEDS_TSV} \
       --out-dir={--raw-output-dir} \
       --days={--days} \
       --max-items={--max-items-per-feed} \
       --max-body-chars={--max-body-chars} \
       > {--raw-output-dir}/manifest.json

   The script writes one markdown file per NON-EMPTY feed into {--raw-output-dir}
   (filename {slug}.md) and prints the JSON manifest to stdout, redirected to
   {--raw-output-dir}/manifest.json. The manifest lists ALL N feeds, including
   empty / gated / errored ones, and is the canonical per-source rollup (the
   dispatcher's volume-driven auto-scaler keys on manifest counts + est_tokens,
   never on file existence). Empty feeds deliberately produce no md file.

   Exit codes: 0 = ran (per-feed failures are captured in the manifest, not the
   exit code); 2 = bad invocation (neither --feeds-file nor --url) → §6.0
   `script-exit-nonzero`; non-zero otherwise (e.g. ModuleNotFoundError for
   feedparser) → §6.0 `script-exit-nonzero` / `feedparser-missing`.

4. Read {--raw-output-dir}/manifest.json. For each record in manifest.feeds[],
   derive the per-feed crawl_status:

     - failed, record.gated is non-null (use the gated label as diagnostic),
                 OR record.error is a hard failure: an HTTPError (404 etc.),
                 "not a feed (...)", "unparseable: ...", or "empty body (...)".
     - empty, record.error is "no items in window" or "feed has zero
                 entries" (feed fetched + parsed cleanly; just no recent items).
                 An empty window is a clean operational outcome, not a failure.
     - clean, record.error is null and record.out_file is present
                 (items_emitted ≥ 1).

5. Build per_source_outputs[], one entry per feed in caller order:

     per_source_outputs[i] = {
       feed_url:        {feeds[i]},
       raw_output_path: {record.out_file},          # null for empty / failed feeds
       item_count:      {record.items_emitted},
       char_count:      {wc -c on record.out_file, or 0 when no file},
       est_tokens:      {record.est_tokens},
       crawl_status:    {"clean" | "empty" | "failed"},   # per-feed
       diagnostic:      {record.gated or record.error when failed; otherwise omit},
     }

6. Compute aggregate fields:

     total_char_count     = sum(entry.char_count for entry in per_source_outputs)
     total_item_count     = sum(entry.item_count for entry in per_source_outputs)
     overall_crawl_status = (
       "clean"   if all entries have crawl_status in {clean, empty}
       else "failed"  if all entries have crawl_status == "failed"
       else "partial"
     )

   Note: `empty` entries do NOT degrade overall status, an empty window is a
   clean outcome. Only mixed clean+failed or all-failed produces
   `partial`/`failed` aggregate.

7. Read-back: confirm {--raw-output-dir}/manifest.json exists and lists N feed
   records. Confirm one md file exists for each entry with crawl_status: clean.
   Mismatch → set overall_crawl_status = "failed" with diagnostic =
   "post-write reconciliation failed: expected N manifest records / M clean files".
```

### Capture

After the read-back (step 7), capture the structured summary as `CRAWL_RETURN` and proceed to §4.0:

```
raw_output_dir:        {--raw-output-dir}
manifest_path:         {--raw-output-dir}/manifest.json
char_count:            <total across all per-feed files>     # aggregate
item_count:            <total emitted across all feeds>       # aggregate
crawl_status:          <clean | partial | failed>             # aggregate
window_days:           {--days}
per_source_outputs:    [<one entry per feed, see step 5>]
diagnostic:            <one-line note when crawl_status != clean; otherwise omit>
```

Do NOT surface raw feed content to the caller, the rendered per-feed markdown is on disk at `{--raw-output-dir}`. §5.0 returns the reference plus summary only.

---

## 4.0 Verify

### Step 1: File-state verification

Verify the raw output directory, manifest, and per-feed files:

- [ ] Directory exists at `--raw-output-dir` (use Bash: `test -d {DIR}`).
- [ ] `manifest.json` exists at `--raw-output-dir` and parses as JSON with `feeds[]` length N.
- [ ] One md file exists at each `CRAWL_RETURN.per_source_outputs[i].raw_output_path` whose `crawl_status == clean`, and that file is non-empty (`char_count > 200`, a single rendered item plus the metadata comment exceeds 200 bytes).
- [ ] Entries with `crawl_status` of `empty` or `failed` carry `raw_output_path: null` (no file), confirm no orphan expectation.

If any check fails (entry status `clean` but file empty/missing, manifest absent or wrong feed count), route to §6.0 (`agent-write-mismatch`).

---

## 5.0 Summary

Return this structured summary to the caller (typically the `/run-pulse` composer):

```
raw_output_dir:        {CRAWL_RETURN.raw_output_dir}
manifest_path:         {CRAWL_RETURN.manifest_path}
char_count:            {CRAWL_RETURN.char_count}            # aggregate
item_count:            {CRAWL_RETURN.item_count}            # aggregate
crawl_status:          {CRAWL_RETURN.crawl_status}          # clean | partial | failed
window_days:           {CRAWL_RETURN.window_days}
per_source_outputs:    {CRAWL_RETURN.per_source_outputs[]}
  # Each entry: { feed_url, raw_output_path, item_count, char_count,
  #              est_tokens, crawl_status (clean|empty|failed per-feed), diagnostic? }
diagnostic:            {CRAWL_RETURN.diagnostic if present}
```

Caller decides next action based on overall `crawl_status` (short-circuit on `failed`; proceed with extraction iteration on `clean` or `partial`) and per-feed `crawl_status` (skip extraction on entries with `crawl_status == failed`; treat `empty` as no-content-this-cycle). The manifest carries per-feed `est_tokens` + counts for the dispatcher's volume-driven auto-scaler. The leaf does not retry, the caller governs retry policy.

---

## 6.0 Error Handling

| Error | Response |
|---|---|
| `--feed-urls`, `--raw-output-dir`, or `--max-items-per-feed` missing from `$ARGUMENTS` | STOP. Report which field is absent. |
| `feed-urls-malformed`, `--feed-urls` does not parse as a JSON array, OR array is empty, OR any entry is not an absolute http(s) URL | STOP. Report the malformed value (the offending entry when per-entry). Caller passes absolute feed URLs. |
| `--raw-output-dir` not an absolute path | STOP. Report the relative path; caller must resolve. |
| `max-items-zero`, `--max-items-per-feed == 0` or not a positive integer | STOP. Report the malformed value; pass a positive integer. The cap is the token governor, an uncapped run is unbounded. |
| `raw-output-parent-missing`, parent directory of `--raw-output-dir` does not exist | STOP. Report the missing parent path. Do not create, caller scaffolds the directory tree before invocation. The `--raw-output-dir` itself MAY or MAY NOT pre-exist; §3.0 Step 1 creates it. |
| A single feed produces `crawl_status: failed` (HTTP error / 404, non-feed HTML, unparseable XML, empty body, or a `gated:cloudflare-waf` / `gated:cloudflare-js-challenge` classification) | The script captures the failure in that feed's manifest record and continues the batch, no md file is written for it. The failure is isolated; aggregate becomes `partial` (or `failed` only if all feeds failed). Per-feed diagnostic (the gated label or error string) surfaces in `per_source_outputs[]`. A `js-challenge` feed is unreachable via any static fetcher, route it to another lane, do not retry here. |
| Crawl produces overall `crawl_status: partial` (some feeds succeeded, others failed) | Proceed to §5.0 normally. Return surfaces overall `partial` + per-feed detail in `per_source_outputs[]`. Caller iterates the array, drops failed entries from downstream extraction, continues with clean entries. |
| Crawl produces `crawl_status: empty` for one or more feeds | Proceed normally. Per-feed entry surfaces `item_count: 0` with `raw_output_path: null` (valid no-recent-items result; the feed was reachable). Empty does NOT degrade aggregate status. |
| `script-exit-nonzero`, the script exits 2 (neither `--feeds-file` nor `--url` passed, indicates a leaf bug in TSV construction) or any non-zero code | Surface the script's stderr verbatim. Exit 2 means the TSV was empty or the `--feeds-file` path was wrong, verify §3.0 Step 2 wrote the TSV. Do not retry without correcting the invocation. |
| `feedparser-missing`, the script aborts with `ModuleNotFoundError: feedparser` | STOP. The runtime dependency is absent. Report the install command `pip install feedparser`; the leaf cannot run without it. |
| `agent-write-mismatch`, a record reports `crawl_status: clean` but the md file is empty/missing, OR `manifest.json` is absent / lists a feed count ≠ N | Surface the mismatch in the return diagnostic. Set the affected entry's `crawl_status: failed`; recompute overall status. File-state on disk (manifest + md files) is authoritative. |