Wiki Ingest

---
name: wiki-ingest
description: "Process a source — a file, a URL, a video, or dictation in chat — into structured wiki pages. Use when the user says 'ingest [file/url]', 'process this', 'let's ingest [subject]', or drops a file in raw/ and wants it turned into wiki pages. Also triggers when the user pastes a URL, PDF path, or image path directly. Orchestrates the full INGEST workflow: detect input, extract content, propose domain, load domain rules, discuss takeaways, run domain dispatch, then update index, overview and log. If no source is given, offers a chat-dictation source page instead."
maturity: tested
outputs: wiki/sources/*.md, wiki/*/entities/*.md, wiki/*/references/*.md
license: MIT
metadata:
  version: 1.0.0
---

# Wiki Ingest

## Purpose

Thin orchestrator for the INGEST operation defined in `agents-core.md § 3`. Adds trigger
phrases and progressive disclosure. **Does not duplicate domain logic** — dispatch is delegated
to `domains/<name>/agents-domain-<name>.md`.

---

## Step 1 — Detect input type and extract content

| Input | Detection | Action |
|---|---|---|
| Video URL | a video host in the argument | Transcribe with whatever capability your tool has. Capture the channel/author name — it drives the 3-source entity trigger |
| Other web URL | `http://` or `https://` | Fetch and convert to readable text |
| File path | `.pdf`, `.jpg`, `.png`, `.md`, `.txt`, audio | Read it. Images are read visually |
| Path in `raw/` | text content on disk | Read it |
| Nothing | no argument given | Ask: *"No source found — create a source page from chat dictation instead?"* If yes, skip to Step 5 and take content from the conversation |

**If extraction yields under ~100 words**, don't write pages from nothing. Say what you found
and stop. A thin result usually means the source is visual or the fetch was blocked — if your
tool can extract frames or screenshots, try that first and note in `## Sources` that
lower-fidelity extraction was used.

### Step 1b — Re-ingest guard

Once you know the `source_url`, check whether it's already been ingested:

```bash
grep -rl "source_url: \"<url>\"" wiki/sources/
```

If a match exists: report the existing page path and its `created:` date, then **stop**. The
user must explicitly pass `--force` to overwrite.

---

## Step 2 — Propose domain, and wait

Propose `domain` (and `subdomain` where the domain defines them). **Do not create any page
before the human confirms.**

This is the highest-leverage checkpoint in the whole workflow — nearly every downstream filing
mistake traces back to a wrong call here.

Default proposal for general or cross-cutting material is `knowledge`. For something the user
owns and operates, propose `home`.

Only installed domains are valid — see the dispatch table in `agents-core.md § 3`. If the
material clearly belongs to an uninstalled pack, say so and offer to install it.

---

## Step 3 — Load domain rules

Read `domains/<confirmed-domain>/agents-domain-<confirmed-domain>.md`. It holds the dispatch
flow and every convention for this domain.

---

## Step 4 — Extract `source_url`

- URL inputs: the URL itself.
- Raw files: scan for a primary URL (clipper metadata, first line). If absent, leave it empty —
  `raw_file:` is the provenance pointer. **Do not web-search for one; do not ask.**
- Never use a local file path as `source_url`.

---

## Step 5 — Discuss key takeaways, before writing anything

Summarise what the source actually says, and ask what to emphasise.

| Depth | Takeaways | Extras |
|---|---|---|
| `--depth deep` *(default)* | 8–12 | timestamps, counterarguments |
| `--depth standard` | 5–7 | — |
| `--depth light` | 3–5 | — |

Nothing is on disk yet. This is where a human corrects a misreading cheaply.

---

## Step 5a — Load page conventions

Read `agents-page-conventions.md` before writing a single page. It carries the frontmatter
tables and the per-page-type section templates.

---

## Step 5b — Read the relation neighbourhood before proposing anything

```bash
./scripts/relations.py <subject-slug>        # and each obvious candidate entity
```

Run this **before** dispatch proposes entity pages. It reports both directions of the
`## Relations` graph — necessary because each fact is stored once, on whichever page reads
naturally, so an existing entity related to this subject may say nothing about it on its own
page.

Two decisions depend on what comes back:

- **The Entity Placement Rule** — an entity already in the neighbourhood tells you which folder
  its siblings chose, which is usually the right answer for this one too.
- **Whether to create a page at all.** If the useful part of this source folds into a page that
  already exists, do that instead. The neighbourhood is how you find that page, and this is the
  main defence against quietly creating a second entity for something already covered under a
  different name.

An empty result is common and is **not** a signal to create a page — it means nothing related is
recorded yet.

---

## Step 6 — Domain dispatch

Run the ordered steps from the domain rules file. Universally:

- **Propose entity pages; don't create them.** Get approval for each.
- **Apply the Entity Placement Rule** per entity — humans to `people/`, owned instances to
  `home/entities/` (or a better-fitting domain), general things to `knowledge/entities/`.
- **Two-way link** — the entity's `sources:` frontmatter gets the source slug; the source body
  links to the entity.
- **A reference page needs 4+ compared entities and human approval.** For 1–3, use entity pages
  with `## Compared To` cross-links.
- **Channel trigger:** at the third source from one origin, propose a channel entity.
  ```bash
  grep -h "^channel:" wiki/sources/*.md | sort | uniq -c
  ```

---

## Step 7 — Post-flight

1. **Flag contradictions** with existing content — never silently overwrite.
2. **Update `wiki/index-<domain>.md`.** The master `index.md` changes only if a domain or meta
   page appeared. Write the one-line summary properly: say what the page is *for*, not what
   it's called — that line is the retrieval signal.
3. **Update `wiki/overview.md`** only if the big picture actually shifted. Usually it hasn't.
4. **Append to `wiki/log/log.md`** — `## [YYYY-MM-DD] ingest | <Subject>` plus 1–3 lines,
   ~40 words. Only if a page was created.
5. **Report:** created, updated, contradictions flagged, open questions.

---

## Flags

- `--depth deep` (default) · `--depth standard` · `--depth light`
- `--force` — overwrite an existing source page, bypassing the re-ingest guard

## Error handling

- **Under ~100 words extracted:** say what was found and stop. Never write pages from nothing.
- **Existing source page, no `--force`:** report the path and date; stop.
- **Raw file not found:** report the exact path checked; offer chat dictation.
- **Domain not confirmed:** do not proceed past Step 2.
- **Domain rules file missing:** name the missing file and stop.
- **Contradiction found:** surface it explicitly. Never resolve it silently.