Mnx Ingest

---
name: mnx-ingest
description: Bootstrap or update a Mnemex knowledge graph from an EXISTING code or documentation repository (a local folder or a git remote) — no live session required. Use this when the user wants to seed a graph from a repo, says "ingest this repo", "bootstrap the graph from <repo>", "import our docs/code into memory", "index this codebase into the graph", or wants to re-sync a previously ingested corpus. Walks the source, DISTILLS durable atoms (never transcribes), discovers a deduped entity catalog, wikifies links, and stages a labeled bulk batch — then /mnemex:mnx-promote --bulk merges it. Two gates only (scope up front, bulk summary at the end); no per-atom review. Idempotent on re-run (a deleted source file surfaces as an orphan candidate, never auto-death). Never writes the graph (staging only) and never mutates the source.
argument-hint: "<local-path|git-url> [--into <graph>] [--dry-run] [--resume <ingest-batch>]"
---

# mnx-ingest — bootstrap the graph from an existing repo (a *source adapter*, not a new subsystem)

> **Speak in the graph's labels.** Read/promote payloads carry a `naming` object
> (`{root_label, space_label}`) — the words THIS graph uses for its root and its spaces (e.g. `area`,
> `repo`, `client`). When the Gate #1 source-tree→cluster map or a routing echo refers to a space, say
> `naming.space_label` ("repo: checkout-svc"), never hard-code "team"/"org". No `naming` in the payload
> (older engine) → fall back to org/team. Folders stay `space-*` on disk — that's storage, not what you
> say.

A live session is one producer of staged atoms; **a corpus is a second.** Ingest walks a repo, *distills*
durable atoms, discovers the entity structure a lived transcript would have handed capture for free,
wikifies links, and stages a **labeled bulk batch** into the same staging tier. Everything downstream —
reconcile, MERGE/SUPERSEDE, contradiction HITL, the wiki mesh, consolidate, doctor, push — is the
**existing** `/mnemex:mnx-promote --bulk`, reused unchanged. **`mnx-promote` stays the only writer to the
graph; ingest only stages.**

The deterministic mechanics (walk · classify · chunk · hash · delta · ER blocking/clustering · bulk
staging) live in `mnx_*.py`; the **judgment** (is this a durable atom? which `[[link]]`? which merge?)
lives here in prose + sub-agents and cannot move into Python.

Background: `docs/corpus-ingestion.md` (the full model), `docs/staging-and-promotion.md` (the pipeline
this reuses), `docs/link-reconciliation.md` (the mesh), `docs/multi-graph-and-team-routing.md` (routing).
Helpers: `mnx_ingest` (walk/probe/delta/manifest), `mnx_glean` (the bounded recall loop), `mnx_er` (entity
resolution), `mnx_simindex` (the ER blocker), `mnx_stage` (bulk staging), `mnx_binding` (locate/clone),
`mnx_phonebook`/`mnx_mesh` (wikification catalog).

## The two invariants that shape everything
- **Distill, never transcribe (DP2).** No file body is copied wholesale into a node; **zero atoms from a
  file is valid and common** (boilerplate, generated code, changelogs, lockfiles). The graph is distilled
  durable memory, not a RAG index over the repo — but distilled is not thin: the body carries the
  *reasoning* behind the fact, the summary is the index row.
- **Single writer + read-only source (DP1, DP3).** Ingest stages only; it never writes the graph and never
  mutates the source (a remote is cloned to a read-only cache; a local path is read in place). Secrets are
  never read.

---

## Preflight — resolve the target graph
Run `python3 "${CLAUDE_PLUGIN_ROOT}/scripts/mnx_binding.py" status --session <sid>` (the session id
from session-start, if you have one — see mnx-init step 1; honors a mid-session graph switch), or
honor an explicit `--into <graph>`.
- If unresolved and no `--into` → **STOP**: *"No target graph. Pass `--into <graph>` or run `/mnemex:mnx-init`."*
- **Echo the resolved target graph** before anything stages, exactly like capture/promote — the user must
  see where the import will land. If `default_fallback` is true, flag it prominently and confirm.
- If `clone_present` is false → `mnx_binding.py sync` once. Note `graph_root`.

## SOURCE — acquire the corpus (read-only)
`python3 "${CLAUDE_PLUGIN_ROOT}/scripts/mnx_ingest.py" acquire --source <path|url>`.
- A **local path** is used in place; a **remote URL / `*.git`** is shallow-cloned into a read-only cache.
- Note `root` (what you walk), `commit` (the exact ref distilled from), and the source slug (derive it with
  `mnx_ingest.py source-slug --source <…>` — it keys the manifest). **Never** write under `root`.

## PROBE + DELTA — scope the run (feeds gate #1)
1. `mnx_ingest.py probe --root <root>` → `{units[], files{}, counts, est_atoms, bytes_total, skipped_secrets}`.
   Keep `files` (path → whole-file hash) for the manifest at DRAIN — a unit's `unit_hash` is per-chunk
   and must never go into one. Units are already classified (`doc | interface | code-doc | config | skip`) and chunked along structure
   (docs by heading, code by **exported** symbol — private symbols are never emitted).
   YAML/JSON are **shape-gated** (they are mostly generated data): an OpenAPI / JSON-Schema-shaped file
   (`openapi:` / `swagger:` / `$schema`) → `interface`, an authored (commented) YAML config → `config`,
   and everything else (data blobs, lockfiles, uncommented JSON) → `skip`. So a `.yaml` file showing up
   under `skip` is expected, not a miss.
2. On a **re-ingest**, diff against the prior manifest:
   `mnx_ingest.py delta --root <root> --manifest <graph_root>/.mnemex/ingest/<slug>.json --commit <sha>`
   → `{added[], changed[], unchanged, orphans[], warning?}`. **Extract only from `added` + `changed`** — unchanged
   files are skipped before any sub-agent runs (the dominant re-run cost saver). Hold the `orphans` list
   (deleted source files' node_ids) for the report — **never auto-tombstone** them (DP4). A `warning` means
   the delta is implausible (a bad prior manifest); show it and stop rather than re-importing the corpus.

## GATE #1 — scope + source-tree → cluster map  (STOP for the human)
Emit the scope preview and **wait**:

```
INGEST SCOPE  source: github.com/acme/payments-service @ 9f3c1a  → graph: payments-knowledge
  files: doc 42 · interface 30 · code-doc 18 · config 6 · skip 210 · secrets-skipped 3
  est. atoms ≈ 140   (cost ceiling ingest_max_atoms_per_run: 2000 — excess resumes next run)
  re-ingest delta: 12 added · 4 changed · 900 unchanged (skipped) · 2 orphan candidates
  SOURCE-TREE → CLUSTER MAP  (proposed from paths; editable — the bulk analog of per-atom domain:)
    settlement/**   → space-payments/settlement
    rails/**        → space-payments/rails
    docs/risk/**    → space-payments/finality    (ONE space per run — another space is its own promote)
    proto/**        → (classify per file — mixed)
  code_extract: gated   (public/documented/config-only; deep opt-in per subtree)
```

Every cluster this map creates gets a **one-line description** (proposed here, written into the
cluster's `index.md` header `> …` line after the bulk promote lands) — read routing decides on that
line and index regeneration preserves it; never leave the scaffold placeholder.
The **source-tree → cluster map** is the bulk analog of capture's per-atom `domain:` — approve/edit it
**once** here, and every atom under a subtree inherits that placement. Routing still flows through the
normal promote precedence (root→space match, `default_space` fallback); the map is a *default*, not a bypass.

**A map may target ONE new space (the create-space gate).** The map normally targets *existing* spaces
(a missing *cluster* inside one is cheap). If a subtree clearly deserves its own new space — a distinct
`repo`/`area`/`client` in the graph's words — the map may name it, but a new **space** is never silent:
**call it out in this gate** with its one-line charter ("new **repo** `checkout-svc` — Checkout service
knowledge?"), and on approval the bulk promote creates it FIRST, then lands its clusters:
`promote_begin(space=space-<name>, ingest_batch=<id>, create_space="<charter>")` (MCP) / `mnx_promote.py`
begin with the same. The engine scaffolds the space before taking the lock (immediately routable), then
the batch drains into it — honoring the graph's `space_creation` config (`off` refuses; add the space
via the Console/CLI first). Re-ingest deltas into the same space are the normal idempotent path.
Keep it to **one** new space per run; spawning a taxonomy of empty spaces is the H3 sprawl pattern one
level up. Speak the space in the graph's label, never hard-code "team".
Confirm scope (or trim it) and the `code_extract` policy. **`--dry-run` stops here** (probe only, nothing
staged).

## PASS 1 — extract + glean + entity-resolve (per subtree, bulk batches)
Mint one **ingest batch id** for the run (e.g. `ing-<date>-<rand>`). Work **per source subtree**, in
bounded batches, honoring `ingest_max_atoms_per_run` (excess resumes next run via `--resume`).

**1a — Distil candidate atoms + entities (kind-aware, LLM judgment).** For each unit, decide what durable
knowledge it holds — *"a fact (the *what*) or a prescriptive pattern (the *how*) a future agent would want
months from now without the source open?"* Kind-aware policy:
- **doc** → domain facts from sections; **patterns** from ADRs / decisions / "gotchas" / runbooks / CONTRIBUTING.
  **Every `type: pattern` atom MUST carry a `trigger`** — the one-line "when does this pattern fire?"
  (e.g. `"when sizing expiry windows in a connector"`). `mnx_stage.add` refuses a pattern without one,
  so distil the trigger from the section's context at extraction time, not later.
- **interface** → the *contract* (public API signature, `.proto`/GraphQL/OpenAPI message, exported type).
- **code-doc** → the intent/semantics the author wrote down (docstrings, module/dir headers).
- **config** → declared knobs + their meaning. **Secrets are never read.**
- **Code value-gate (non-negotiable noise control):** a code-derived atom is staged **only** if it is a
  public/exported contract, **or** carries an author-written docstring/comment explaining *why*, **or** is a
  config knob with a declared meaning. A bare private helper with no doc is **not** an atom. Function
  *bodies* are never transcribed — only the distilled semantics. Emit `[[wiki-links]]` inline by name.

**1b — Glean (checklist mode — coverage outranks importance).** A single extraction pass under-captures.
Unlike a session, a corpus unit is never skipped for being *unimportant* — only for holding nothing
durable, and then only on the record. Build the candidate-unit list as the coverage checklist and run the
shared *gleanings* loop:
```
mnx_glean.py coverage --units <units.json> --staged <staged-ledger.json> [--waived <waived.json>] --pass <k> --graph <graph_root>
```
It returns `uncovered` (the zero-atom units — your re-ask worklist), `waived`, `coverage_pct` and a `stop`
signal (`complete` = every unit covered or waived, `cap` = `ingest_glean_passes`, default 4). A unit is
*covered* when a staged atom carries its `anchor` in provenance. Re-examine each uncovered unit once —
*"what durable fact/entity did this unit contain that I did not extract?"* — then re-run. A unit that truly
holds nothing durable (lockfile, generated code, boilerplate) goes in `--waived` as `{"<unit-id>": "<why>"}`;
an unreasoned waiver is **refused** and the unit stays uncovered. **Aim for `complete`, not `cap`.**
Re-staging identical content stays an idempotent no-op (DP10) reporting `action: already-staged` — count
those separately from fresh `staged`. The judgment stays here; `mnx_glean` only bounds/bookkeeps.

**1c — Assemble the in-batch entity catalog + entity-resolve (dedup, DP5).** Collect the candidate entities
(canonical name + aliases + type) across the whole delta corpus, then run ER over `{new atoms ∪ existing
graph pages}`:
```
mnx_er.py resolve --graph <graph_root> --atoms <candidates.json> [--team <t>]
```
It blocks (via `mnx_simindex.pairs --with --intra`), scores, clusters, and proposes a disposition per
cluster: **CREATE** (no graph match) · **MERGE** (folds into an existing page, keeps its id) · **COLLAPSE**
(intra-batch duplicates → one CREATE). The `possible` band is the **only** place you (the LLM judge) rule
on a merge — everything at/above `match` or below `possible` is deterministic. The output is your deduped,
canonical entity set (aliases unioned). **One entity → one node**: many corroborating sources collapse into
one well-provenanced node, never duplicate nodes.

## PASS 2 — wikify (rewrite atom bodies against the catalog) + stage
Resolving each name greedily as you meet it fragments a large graph. Instead, link every atom against the
**one** catalog you just built (∪ the team phonebook — `mnx_phonebook.py resolve` / `resolve-batch`):
- For each mention of a catalog/graph entity in an atom body → emit `[[canonical-name]]` (piped display if
  the surface form differs). A mention with **no** catalog/graph entity → still `[[bracket]]` it → a
  **red-link** that heals the moment its page is created (later batches in this same run knit the mesh).
- **Precision discipline (DP6):** an **exact** catalog/phonebook match links deterministically; a
  **fuzzy/semantic** near-match is a `⚠ suggested` link surfaced at gate #2, **never** auto-written. A wrong
  link is a false edge. High recall in Pass 1 (bracket generously — red-links are cheap); high precision in
  Pass 2 (only *confident* links go live).

**Stage each atom under the bulk label** with source-anchored provenance so it is promotable **cold**:
```
mnx_stage.py add --json <<'JSON'
{ "type": "domain", "summary": "Settlement cut-off is 23:00 UTC; post-cutoff addenda ride in field 124",
  "aliases": ["field 124", "settlement cutoff"], "domain": ["settlement"], "score": "later",
  "ingest_batch": "ing-2026-07-11-a1b2",
  "provenance": { "source_repo": "github.com/acme/payments-service", "commit_sha": "9f3c1a…",
                  "source_path": "settlement/reconcile.md", "anchor": "Cut-off handling", "kind": "doc",
                  "rationale": "distilled from settlement design doc" },
  "body": "Settlement cut-off is 23:00 UTC. The batch is reconciled before any leg is posted, so a mismatch is caught while it is still cheap to fix.\n\nAddenda arriving after cut-off are not rejected — they ride in [[iso8583-field124]] and apply against the NEXT batch. That is why field 124 is populated on only a minority of messages yet must never be dropped by a parser: it is the sole carrier for a late correction." }
JSON
```
- **`provenance.anchor` = the unit's `anchor` from `probe` output, verbatim** (bare heading text, no
  leading `#`s) — the glean-coverage key. **`provenance.source_path` is what the manifest ties node ids
  back to**, so an atom missing it can never be surfaced when its source file is deleted. A `pattern`
  atom additionally carries the top-level `"trigger"` field (see the kind-aware policy above).

**Hub atoms — make the mesh knit (do this, or the import lands ~all-red).** Fact atoms link to entity
*names* ([[ilp-address]]), but no fact atom IS the entity page, so without hubs nearly every link stays
a red-link and the whole import scores in-degree ≈ 0 (inert tiers, orphan-flood). For each **catalog
entity** (Pass 1c output) that (a) is mentioned by **≥3** staged atoms and (b) has **no** graph match
(ER said CREATE, not MERGE), stage **one hub atom**: `type: domain`, the entity's canonical name +
aliases, `volatility: timeless` when definitional, a 1–3 sentence body that says what the entity *is*
and `[[links]]` to its closest siblings. Red-links to it heal deterministically at promote (which
re-derives every node in the space), knitting the mesh in the same run. Below the mention threshold, leave the
red-link latent — a hub nobody points at is noise.
The `--ingest-batch` label sets `bulk: true` and partitions these atoms from any hand-captures (DP8) — the
per-session nag never fires, and the batch has its own large cap. Re-staging identical content is a no-op.

## DRAIN — hand off to bulk promote
Ingest **never writes the graph.** Drain the staged batch with the existing writer:
`/mnemex:mnx-promote --bulk --ingest-batch <id>` (gate #2 = the bulk summary; it auto-accepts plain
CREATE/MERGE and stops only on contradictions + new-cluster creation). Hand it probe's `files`: its step 5
records the manifest (`source_path@commit → node_ids`) from those plus apply's `dispositions`, so the next
ingest diffs correctly and a deleted file can name its orphans.

## REPORT
Summarize by cluster: created / merged / superseded / dropped-dup / held, atoms staged this run, the
final **coverage** (`coverage_pct`, waived-with-reason, and any unit still uncovered at `cap`), the
**orphan candidates** (deleted source files — surfaced for the human, never auto-tombstoned), and whether
the cost ceiling forced a `--resume`. If `--resume <ingest-batch>` was given, continue from the manifest +
remaining staged atoms rather than restarting.

## Never
- Never write into `graph_root` — ingest **only stages**; `/mnemex:mnx-promote` is the sole writer.
- Never mutate the source corpus, and never read a secret (the walk skips + counts them; you never open them).
- Never transcribe a file body into a node — distill; zero atoms from a file is valid.
- Never stage 5 atoms for one entity — ER collapses intra-batch duplicates **before** staging (one entity → one node).
- Never auto-write a fuzzy link (⚠ suggested → gate #2) and never auto-tombstone a deleted file's nodes (orphan candidate).
- Never do per-atom review — two gates only (scope up front, bulk summary at the end).
- Never let community detection mint structure — path-based routing is the default; Leiden may only *propose* at gate #1.