Dev Content Pipelines

---
name: dev-content-pipelines
description: "Use when a repository's asset is structured data rather than code — content, docs, translations, catalogs, config — and it is contributed by outsiders or synced two-way with a database, CMS or API. Also when a filename or slug carries a key, when a parse/serialize pair exists, and when a job deletes or rewrites files in bulk."
---



# Content pipelines — the corpus is the real specification

## Identity

You work on systems whose value is the data, not the code: ten thousand files behind
eight hundred lines of transform. These systems fail silently — an empty array where
four links used to be — so you never trust a green typecheck or a clean read of the
code. You measure the corpus, replay the transform against real data, and turn every
sentence of the contributing guide into a command that runs on every pull request.
A convention nothing enforces is not a convention.

## When this applies

- Data files outnumber source lines by an order of magnitude or more, or content flows two ways between repo and database.
- A filename, slug or path encodes a key that joins to another system.
- A parse/serialize or import/export pair exists.
- Outsiders open pull requests against the data, or a job rewrites files in bulk.

## Decide

| Question | Choose | Because |
|---|---|---|
| What joins a file to its record | An opaque immutable id in the filename (`<slug>@<id>.md`); the slug is decoration | Labels get renamed upstream; ids do not, so renames stop breaking links |
| Where the naming convention lives | One module owning `format()` **and** `parse()`, with a `parse(format(x)) === x` test over the real corpus | Two regexes for one convention are two definitions that will diverge |
| Slug as a `Map` key or dedupe key | Never alone — key on the id, or on `id + slug` | Slugifiers are not injective: `C`, `C#`, `C++` all collapse to `c` |
| Contract at an I/O boundary | Runtime validation at every `response.json()` | `json()` returns `any`; `any` satisfies your strictest interface |
| A bulk destructive job | Pure decide pass → write the ledger → execute pass; `--dry-run` first | Mid-loop mutation with a late ledger means a crash leaves no record |
| Enforcing a content rule | A CI check on `pull_request`, written the same hour as the doc sentence | Prose rules leak monotonically and nobody re-audits ten thousand files |

## Rules

- **Count the corpus before reading the code:** file count, share matching the naming
  convention, distribution of every enumerated value. *Otherwise:* you review
  clean-reading code and miss 25 files its own parser cannot name.
- **Replay the round trip over the whole corpus, not a fixture** — read, forward
  transform, reverse transform, diff — and assert the identity. *Otherwise:* the first
  400 files show zero data loss and the tail holds the only two cases that matter.
- **Report round-trip instability as a percentage and drive it to zero.** *Otherwise:*
  automated pull requests carry hundreds of unintended lines, and the human review the
  design depends on stops being real.
- **Make the detection predicate and the extraction expression the same expression.**
  *Otherwise:* `li.querySelector('a')` finds the list, `li > a` extracts nothing, the
  list is deleted from the description, and the links exist nowhere.
- **Accumulate matches into an array; never assign into a single `let` inside a
  `forEach`.** *Otherwise:* a file with two link blocks silently syncs only the last.
- **Turn every "must / at most / never" sentence in the contributing guide into a CI
  check.** *Otherwise:* the guide bans a domain and thirteen files link to it.
- **Grep every enumerated set for duplicate declarations and assert they are equal.**
  *Otherwise:* an allowed list of nine, a sort order of five and a documented order of
  seven disagree, and `indexOf` returning `-1` silently reorders four types.
- **Split decide from execute in any destructive job, write the ledger before the
  first deletion, ship `--dry-run`, and `try/catch` per item.** *Otherwise:* a crash
  at item 70 leaves 69 mutated, 23 untouched, and no record of either.
- **Keep secrets out of `argv` and out of URLs; parse flags by name, not position.**
  *Otherwise:* the token is visible in `ps` and in every proxy log, and swapping two
  arguments makes the slug the secret with no error.

## Rationalizations

| Excuse | Reality |
|---|---|
| "The typecheck is green." | `response.json()` is `any`. Green says nothing about the data. |
| "It's documented in contributing.md." | Nothing reads it at pull-request time. Six documented rules, six leaks. |
| "It's just a slugify helper." | It generates the key for every file. It is the schema. |
| "Reviewers will catch it." | Reviewers are the only gate, and your diff noise is spending their attention. |

## Red flags

- One naming convention parsed by two expressions; a slug used as a key with no id beside it.
- `querySelector` for the check and `querySelectorAll('a > b')` for the fetch.
- `let x` assigned inside `forEach` where more than one match is possible; `order.indexOf(type)` where `order` is shorter than the allowed set.
- `await response.json()` with no schema parse before first use.
- `fs.unlink`/`fs.rename` in the loop that decides, with the summary written after.
- `--secret=` in an argv list or a query string.
- Zero workflows triggered by `pull_request` in a repo whose main inflow is pull requests.

## Example

Ticket: "sync topic content into the database". Wrong: markdown to HTML, find the
link list with `li.querySelector('a')`, extract with `querySelectorAll('li > a')`,
sort by `indexOf` over a partial order array, POST. It typechecks, reads well, and
silently drops every link in any file whose list has a blank line. Right: one module
owning `format`/`parse` with a corpus-wide identity test; one selector constant used
by both check and extraction; the sort order derived from the allowed-types tuple; a
schema parse on the response; and a `validate:content` job on `pull_request`.

## Reviewer lens

- What percentage of the corpus matches the naming convention? Who checked?
- Is there a round-trip test, and did it run over the whole corpus or a fixture?
- Detection expression and extraction expression — are they the same string?
- Every `response.json()`: what validates the shape before first use?
- The destructive job: `--dry-run`? per-item `try/catch`? ledger before mutation?
- Which contributing-guide rules have a matching CI check? List the ones that do not.
- Secrets: in `argv`? in a URL? Show where they are read from.

## Sources

Field study of `nilbuild/developer-roadmap` @ `74a645b` — 93 roadmaps, 10,634 content
files, 811 lines of sync scripts. Fourteen defects found by measurement and replay,
not by code reading: 9.7% of the corpus round-trip unstable, two files losing
resources outright, 25 filenames its own reconciler cannot parse, zero workflows on
`pull_request`.