Mem Ingest

---
name: wicked-garden-mem-ingest
description: "Ingest source files into the wicked-estate knowledge store as cited chunks. Handles text files (md, txt, csv, html, json, code) with deterministic chunking and binary documents (pdf, docx, pptx, xlsx, png, jpg, gif, webp) via LLM vision — the agent extracts and chunks, estate persists and ranks. Use when: dispatched by the wicked-garden-mem skill's ingest action, or directly for 'ingest this file', 'add this document to the knowledge base', 'index this directory', 'learn from this pdf'."
metadata:
  role: worker
---

# Mem Ingest Worker

This skill is designed to run as an isolated worker; when your harness cannot fork, run it inline and keep its output separate from the caller's.

You ingest source files into the **wicked-estate knowledge store** so later
`knowledge.recall` / cited-answer calls can return them with `source`
citations. This ports the brain-era ingest pipeline (FOLD-2, Phase 5-S7)
onto estate: **you** read, extract, and chunk (DEC-R: the agent is the
reasoner); the **engine** writes and ranks (`knowledge.ingest` /
`knowledge.write` — deterministic, never calls a model). Estate's embedded
method card `skill://knowledge-ingest/SKILL.md` is the chunking doctrine
this pipeline follows.

## Runtime
Script-backed steps use the `wicked-garden` launcher: `wicked-garden run <plugin-root-relative path, e.g. scripts/…> [args]`. It is on PATH after `npm i -g wicked-garden`; otherwise use `npx wicked-garden run …`; inside a wicked-crew run it is `"$WICKED_GARDEN_ROOT/scripts/wicked-garden"`.
If none of these is available, or Python 3 is missing, skip the script-backed step, say so, and follow the manual alternative where one is given next to it — never invent the script's output.
Relative paths in this skill are relative to the directory that contains this SKILL.md.

## Parameters

- **source** (required): path to a file or directory
- **scope** (optional): estate scope, default `project:<cwd-basename>`
- **title** (optional): document title, default derived from the filename

## The write seam

All writes go through the mem backend (stdio shim → estate MCP):

```bash
wicked-garden run \
  scripts/mem/estate_memory.py ingest -
```

with a JSON body on stdin (chunks are long — never inline them in argv):

```json
{"title": "<doc title>", "chunks": ["<chunk 1>", "<chunk 2>"],
 "scope": "<scope>", "source": "<absolute-or-repo-relative source path>"}
```

Rules that make chunks worth storing:

- **Self-contained** — one idea per chunk (a section, a slide, a sheet, a
  table row group). Recall returns chunks verbatim; a fragment that needs
  its neighbours is a bad chunk.
- **Source provenance always** — `source` is what a cited answer cites. An
  uncited chunk cannot back `wicked-garden-search answer`. Never omit it.
- For a single standalone fact, use the `write` action instead
  (`{"content": ..., "class": "concept", "source": ...}`).

## Step 1 — assess the source

- Single file → Step 2 (text) or Step 3 (binary) by extension.
- Directory → Step 4 (batch).
- Health check when in doubt: `... estate_memory.py health '{}'` — on
  `"ok": false`, stop and report the degrade (estate binary missing or
  store unreachable). Do not fake success.

Text extensions: `.md .txt .csv .html .htm .json .py .js .jsx .ts .tsx .sh
.rs .go .java .yaml .yml .toml`. Binary (vision) extensions: `.pdf .docx
.pptx .xlsx .png .jpg .jpeg .gif .webp`.

## Step 2 — TEXT files (deterministic chunking)

1. Read the file.
2. Split:
   - **Markdown**: on H1/H2 headings; sub-split any section over ~800 words
     at paragraph breaks.
   - **Code**: one chunk per file (small files) or per top-level
     class/function group; prepend a one-line `path — what it is` header so
     the chunk stands alone.
   - **CSV/other text**: paragraph groups of ~800 words; for CSV keep the
     header row with every chunk.
3. Send title + chunks + scope + source through the write seam (one `ingest`
   call per file).

## Step 3 — BINARY files (LLM vision)

You receive the document natively via `Read` — examine it visually and
extract content as markdown. This is the one genuinely agent-only stage of
the pipeline; everything around it stays deterministic.

- **PDF**: one chunk per logical section (or every 3–5 pages). `Read` PDFs
  with the `pages` parameter for long documents.
- **PPTX**: one chunk per slide or tight slide group; capture speaker-note
  intent, not just visible text.
- **DOCX**: one chunk per section heading.
- **XLSX**: one chunk per sheet; render data as markdown tables (downsample
  huge sheets: header + representative rows + totals).
- **Images**: one chunk describing the visual content, plus any legible
  text verbatim.

Preserve numbers, names, and tables exactly — a cited answer will quote
them. Then write through the same seam as Step 2, `source` = the original
binary's path.

## Step 4 — directories (batch)

Do NOT ingest files one-by-one in conversation. Write a small Python batch
script to the session scratch dir that walks the tree (skip dotdirs,
`node_modules`, `__pycache__`, lockfiles), applies the Step-2 chunking to
each text file, and pipes one `ingest` JSON per file into
`wicked-garden run scripts/mem/estate_memory.py ingest -`. Run it, then:

- Report `files → chunks` counts from its output.
- List the binary files it found and vision-ingest the important ones
  yourself (Step 3), or hand the list back for the caller to choose.

## Step 5 — verify (seeded recall)

Pick 1–2 distinctive phrases from the ingested content and confirm they
come back cited:

```bash
wicked-garden run \
  scripts/mem/estate_memory.py sources '{"query":"<distinctive phrase>"}'
```

The matching chunk must appear with the right `source`. If it doesn't,
say so — never report an unverified ingest as done.

## Step 6 — report

- `{N} files ingested, {M} chunks written, scope {scope}`
- doc_ids returned by the backend
- binary files pending vision ingest (if any were deferred)
- verification: which seeded phrases came back, with their `source`