wikidoc

---
name: wikidoc
description: Sort your documents into a corpus you can query. Use when the user asks to sort, file, rename or dedupe documents, asks what a file is or where it belongs, or when a durable fact about the user, their entities, dossiers or correspondents surfaces and must be written to the wiki.
---

No `config.yaml` in the workspace (`$WIKIDOC_HOME`, default `~/.wikidoc`)? Read [`SETUP.md`](SETUP.md), run it, come back here.

Filing is the means; a corpus that answers questions is the end. Every pass ends with each file carrying what it is, and `memory.jsonl` carrying why that was decided. Query it anytime: `memory.py stats` · `show <path|md5>` · `find <term>` — memory answers, it never selects.

Scripts live in `scripts/`, next to this file. Resolve this file's directory once and invoke every script by its **absolute path** (`python3 <skill-dir>/scripts/route.py`) — a relative invocation from the wrong cwd fails halfway through a pass. Long output goes to `bench/logs/`; read it paginated rather than re-running a script to see it again.

## Invariants

- **Evidence** decides. Strength 3 = a validated identifier read in the content, 2 = a name read in the content, 1 = path or filename only — hearsay. A rule matching only strength-1 conditions is capped at `propose`.
- Read a file before the gesture that touches it. The `propose` and `residual` triages exist for exactly that.
- Removal goes to the OS bin — every gesture is reversible. A sensitive document is never routed automatically and never trashed unread: `apply.py` re-reads the file itself and refuses the entry. It can still be filed and renamed, which is the point of sorting it. The one refusal that lifts is deduplication: name a `keeper` and apply re-hashes both files at the moment of the gesture — same bytes, keeper present, keeper not itself being binned, or the refusal stands. The guard protects the content, not the copy count.
- Duplicates are byte-identical or they are not duplicates. Zero-byte files group with nothing — emptiness is not identity. Same text, different bytes is a re-download: diff in full before keeping one.
- A move counts once it is re-stat'ed at its destination.
- **The scripts are optional; the record is not.** They hash without mistakes, walk without forgetting, and keep hundreds of files out of this conversation — that is their whole value. What a document is, and where it belongs, you decide by reading it; when a script contradicts the file, the file wins. So filing done by hand is filing all the same, and it owes the same `memory.jsonl` line — `Memory.record(...)` then append, `triage: "propose"`, a `reason` that says why, written after the destination is re-stat'ed. Provenance says WHO made the call: `provenance: "human-decision"` when the user's own word set the destination, `provenance: "llm-decision"` when you read the file and decided alone, unasked. Don't default to `"human-decision"` because it sounds more authoritative — it means the user actually said so; most hand-filing outside a pass is `"llm-decision"`. A file moved without its line is worse than one left alone: it looks sorted, and the next collect takes it for new. The only file that may have no line is one whose destination is undecided — that absence IS the open question, and it goes to `wiki/state.md`.

## The working file: `bench/routing.json`

One entry per selected file; each actor writes only its columns; empty columns are the remaining work. `bench/` is the pass's working directory — archived to `logs/<pass>/` at Learn, never deleted.

| columns | writer | step |
|---|---|---|
| `path size mtime md5 ext pages text truncated prose needs_vision render ids dates doc_year duplicate_of known_as known_desc opaque error` | collect.py | ① |
| `text lu` (needs_vision entries only) | vision agent | ② |
| `triage why guards rule entity strength destination rule_tags shadow` | route.py | ③ |
| `decision dst desc tags date_doc reviewed keeper` | decide agent | ④ |
| `result final` | apply.py | ⑤ |

## ① Collect

`collect.py` selects the pass — `memory.jsonl` is the seen-set. A file is a candidate when it has no memory line, its (size, mtime) changed with new content, or its last decision was `unanswered` or `refused`. Order: those two first, then `inboxes:` files, then the rest; take `batch_size` (default 600). A subtree behind a `type: "dir"` line (a compacted homogeneous payload — DICOM slices, exports; written by `scripts/compact.py`, full decision in the wiki) is checked as a unit against its fingerprint and pruned when it holds; on real drift its files come back as candidates and the log says so.

The walk starts at `root` **and at every inbox that lives outside it** — Desktop and Downloads are outside the document tree on most machines, and an inbox is a walk root, not merely a priority band. Their files are keyed `~/Desktop/…` in memory; everything under `root` stays root-relative. What could not be walked is named — a refused directory is reported `IGNORED`, and an unreadable walk root, or a scan of zero files against a non-empty memory, ends the pass with an error. "I could not look" is never reported as "there was nothing". Extraction is page 1 only, ~4000 chars; no text and size > 1 KiB → `needs_vision`, with a page-1 PNG in `bench/renders/` for PDFs, images, and the office formats Quick Look draws (`.doc`, `.xls`, `.ppt`, `.pages`, `.numbers`…) — other formats reach ② renderless and climb its ladder. Text that comes back as glyph names or mojibake is not text: it is dropped, and the file goes to vision like any unread one. An extractor's debris filed as content is a document nobody ever reads. Byte-identical duplicates are grouped with no size threshold. A symlink is a pointer, not a document: `opaque: "symlink"` with its `link_to`, never hashed, never grouped as a duplicate of its own target, never read — a corpus may file deliberately in links, and dedup would bin them. Its memory key is its own name, never the target's: keys resolve the parent directory, never the final component, or a link and its target would share one line and one of the two would be lost.

Done when the counts in `bench/logs/collect.log` are numbers you can explain from the corpus, and every `needs_vision` PDF and image has a render.

## ② Vision — in a subagent, always

**Delegate this step. You never open a render yourself.** A pass carries hundreds of `needs_vision` entries; opening their images in the conversation that also holds the config, the ledger and the decisions is how a pass dies of its own context halfway through. Spawn a subagent per file, or per small batch of files from the same folder, and give it exactly one job: look, and come back with words.

What the subagent is handed: the paths, their renders, and the ladder below. What it returns: for each entry, the `text` it read and the `lu` that says how — nothing else. No image, no base64, no "here is what I saw" gallery. You write the columns from its report.

Each entry gets `text` and `lu`: `"text"` (extracted layer sufficed) · `"render"` (read from the PNG) · `"pages N-M"` (escalated into the original). Escalation goes upward through readers, never around them: page-1 render → read the original (`Read`, targeted pages) → `collect.py --render <path> --pages A-B` on other pages → convert (`sips`, `qlmanage -t`, or whatever the platform offers). No script ever interprets content.

A file that survives every reader is **withdrawn**, and withdrawal is torn from you, not chosen: set `decision: "unanswered"` with a `reason` that names each attempt — `"tried: render p1, Read p1-2, sips — all failed"`. A withdrawal whose reason lists no attempts is laziness with a paper trail; the file comes back first next pass either way, so skipping the ladder buys nothing. `lu` is NEVER set on a file that was not read — it is a witness, not a checkbox.

**Answering a withdrawn file**: when the user says what an unreadable file is, write the real decision (`move`, `trash`, …) straight onto the entry — a human judgement is the strongest evidence there is, and it counts as triage `propose`. Re-run `route.py` and it stamps that triage instead of blocking; and even if `apply` already ran, `--learn` accepts a stamped `result` without a triage (reported under `answered_withdrawals`). Either order closes the pass.

Containers (`.zip`, `.tar`, …) are never `needs_vision` — nothing renders them: collect marks them `opaque: "container"`, they reach ④ as residuals, and an agent that opens one to read its listing records `lu: "container"`.

Done when every `needs_vision` entry has `text` filled, or a withdrawal whose `reason` lists the attempts — and when not one render reached this conversation.

## ③ Route

`route.py` gives every entry a triage: `route` (a rule recognised it), `propose` (needs eyes — **already settled by hand**, sensitive, duplicated, tied entities, inbox, weak evidence, unresolved destination), `residual` (nothing matched), `skip` (content already in memory). It exits 2, paths listed, while any entry is unread — no judgement on unread bytes. A `known_as` entry is exempt: its md5 is already in memory, `skip` is decided on identity and never reads the text. Recognising bytes read in an earlier pass is not judging them — and binning one still requires reading it this pass. The one way past the barrier is a withdrawal from ②: those entries are counted `withdrawn`, get no triage columns, and wait for the next pass. Shadow rules predict before guards, so guards never blind learning.

**The ledger is an input here, not only a seen-set.** When the last line for a path was written by hand (`provenance: "human-decision"` or `"llm-decision"`) and left the file where it now sits, the `settled` guard fires and its `reason` becomes the entry's `why`: an arbitration already made outranks anything rules or entities can re-derive from the same bytes — whether the user said so or the agent read the file and judged alone, both are a reading of the content a rule cannot reconstruct from the bytes alone. Without it a judgement went in and was forgotten, so the next pass reasoned from scratch and could land elsewhere — which is how an exception dies, out-reasoned in silence. The case that forced it: an invoice a company sent to its own owner as a private person carries that company's IBAN, so the entity claims it at strength 3 and beats the folder it belongs in. The guard does not decide; it forbids deciding alone. Only hand-written lines count — a line apply wrote for a routine gesture would make every re-selected file propose, and a guard that fires on everything is a guard nobody reads.

A document names several entities at once; the strongest evidence decides, config order decides nothing, and a tie goes to `propose`. An `inboxes:` file is always proposed, never silently routed. A destination variable that does not resolve ({doc_year} with no date) means the rule does not fire — never an `undated/` folder. All matching is accent- and case-insensitive.

Done when every entry has a triage and no count surprises you. This verb only writes columns in `bench/` — there is no `--dry-run` here; dry-run belongs to Apply alone.

## ④ Decide

The `route` triage needs confirming, `propose` opening, `residual` eyes. Fill `decision` (`move|trash|tag|rename|none`), `dst` (trailing `/` files into that folder; anything else is the full path, which is how a rename is written), a `desc` that says something the filename does not, `tags`, `date_doc` (never invented) — a rule that proposed tags left them in `rule_tags`, and a rule carrying tags but no destination is a `tag` decision, not a move, and `reviewed: "vision"` when you read the render this pass — the sensitive probe requires it before trashing a file whose text will not extract. On a duplicate, `keeper` names the copy that stays; with a byte-identical keeper — or one matching the same `sensitive: supersedable:` family in config — it is the only way a sensitive file is ever binned, and apply proves the survivor from disk before acting.

Blocking questions go to the user during this step, at the moment they arise — never batched to the end. What stays unsettled is left undecided; Learn will record it.

Done when every entry has a decision, or a reason it does not that you can say out loud.

## ⑤ Apply

`apply.py` is dry-run by default: it prints every gesture and touches nothing. `--execute` once the printout matches intent — and the execution report must equal the dry-run report. Each successful action appends its memory line immediately and stamps `result`; a crash leaves both exactly at the interruption point, and `apply.py --resume --execute` continues where `result` is missing (`--resume` alone prints the dry-run of the remainder — dry-run stays the default, always). Before replaying an entry whose source is gone, resume reconciles: a gesture the killed run already performed (memory line from this pass, or the file at its decided destination with matching size and md5) is stamped `result`, reported as `reconcile` — never a FAIL, never a ghost `unanswered`. Trash goes to the OS bin (fallback `<workspace>/.trash/<pass>/`), collisions get `(2)`, nothing is clobbered. The sensitive probe trusts nothing from the bench: text and ids are re-extracted from the file itself, and extractor debris does not count as a reading — a non-trivial file whose re-read yields no language is kept until `reviewed: "vision"`. No `sensitive:` block in config → every trash is refused.

A refusal writes its own memory line — `decision: "refused"`, naming the decision it refused and the guard that stopped it — and stamps `result`. The file is unfiled, so the next collect re-selects it first; but the record says a judgement was made and held back, which is not the same sentence as "nobody could read it".

Done when dry-run and execution report the same counts and `failed` is 0. `refused` is a separate count and is not a defect: a guard that keeps a sensitive or unread file is the tool working. Never chase it to zero — read each refusal, and answer it with evidence (`reviewed: "vision"` after actually looking) or leave the file alone.

## ⑥ Learn

`route.py --learn` closes the pass: scores every shadow rule against each file's final path, mines new candidates (the simplest form with zero counterexamples, born `status: shadow`), reports `ripe` rules, reports every `unanswered` file BY NAME and writes its memory line so the next pass selects it first, lists the `refused` files separately (their lines are already written, by apply), warns on dead `anchors:` pointers, archives `bench/` → `logs/<pass>/`. You then write the leftovers and their open questions into `wiki/state.md`.

**This whole step runs in a subagent, like vision — and that includes `--learn` itself.** A rule is instructed by replaying it over thousands of re-extracted documents to count its true and false positives; done inline, that volume lands in the very conversation that holds the config, the ledger and the pass's decisions. Spawn the agent, let it run `--learn`, read the report, replay every rule that moved **against the disk and the memory**, and come back with a figures report: diagnosis, proposed YAML, matched / true / false, plus the `unanswered` and `refused` files named. It never hand-edits `config.yaml`; you apply the decision, write `wiki/state.md`, and put the questions to the user.

Two traps it must be told about, or it will mistake bookkeeping for evidence. **A rule mined in the pass's own context matches the pass, not the corpus** — the miner's counter-sample is the pass's textual entries, and unique-among-250 is not discriminating-among-thousands. And **`--audit` runs text conditions against the recorded `desc`, not the document** — it says so in its output, and the note is easy to skip past: a rule showing `disagreed: 0` there has been compared to the human's own summary, not to the bytes. Measured once: a rule at `disagreed: 0` in `--audit` was wrong on 55 of the 64 documents it actually touched on disk.

Done when `bench/` is gone from the workspace and every leftover is named in `wiki/state.md`.

## Questions — three moments, never a fourth

- **Blocking** — during Decide, at the moment they arise.
- **Rules** — at promotion, one at a time, counters and full-audit list attached.
- **Leftovers** — written to `wiki/state.md` at Learn, asked at the next session start; their files are re-selected first.

## Rules: born shadow, promoted by the user

A rule earns its way in; it is never handed over on the strength of looking right. Born `status: shadow` from `--learn` only, evaluated on every pass, **never applied — a shadow rule fills its own column and nothing else, and route.py stops the pass if a non-active rule ever reaches the routing columns** — passes, hits, agreed, **disagreed** accumulate in the rule itself. After ≥5 passes with zero disagreement:

1. `route.py --audit <rule-id>` — replay against memory: retroactive precision on files already judged, diverging files listed. Cheap, read-only, runs anytime. **For a text condition it is indicative only**: the bytes left the bench, so the condition is tested against each record's `desc` — the human's summary, not the document. Its own output says so. Never promote a text rule on `--audit` alone; `--full-audit`, which reads the disk, is the one that decides.
2. `route.py --full-audit <rule-id>` — confront the whole disk, including what no pass ever judged. Beyond 500 candidates it says so and you sample. No ground truth here — the list is for human judgement.
3. The user promotes, on that evidence: `status: active` plus a dated `history:` line. Refusal → history line, rewrite, `cycle: N+1`, counters reset.

Retired after 3 failed cycles, or 10 passes at 0 hits. A rule that keeps diverging is wrong about these documents — rewrite it or drop it.

## The wiki

`wiki/` holds what the corpus cannot say about itself: who is who, which entity was live in which period, which arbitration was made and why. `context.md` carries the durable facts; `state.md` carries what the next session must pick up. Start there.

It is also written outside a pass, and outside this skill: a durable fact established while answering any question belongs in `wiki/` at that moment — `context.md`, `decisions.md`, `filing-patterns.md`, `trash-criteria.md`, `state.md` — with its line in `wiki/index.md` updated, surgically, section by section. A fact re-derived from the corpus for the third time is a wiki line that was never written. That writing is not a pass: it appends nothing to `memory.jsonl`, which records gestures on files alone.

`index.md` is a table of contents, not a digest. One line per file — `- [name](file.md) — what it answers` — and the hook names the question the file settles, never the answer to it. **Never restate in the index a fact the target file carries**: two places then own one fact, the index costs the price of both, and neither defers to the other. Past ~150 characters a line has stopped pointing and started summarising; that is the bound, and it is the whole discipline.

**A replaced fact is deleted where it lived.** What replaces it takes its place, and the arbitration that moved it goes to `decisions.md` — that is the trace it leaves. A superseded fact left standing is a second answer to a settled question with nothing marking it stale, and the reader cannot tell which of the two is current. Only a contradiction you cannot settle stays: recorded WITH its contradiction and with what would settle it, in the file, never in the index.