Walnut Core Internals

---
name: walnut-core-internals
description: Deep implementation details for src/core/ — task model (5 text fields, phase lifecycle, dependencies, session slots, child tasks), session monitoring (3-layer), message queue & FIFO/JSONL delivery patterns, daemon-uniform file access rule, chat history compaction. Use when implementing or debugging anything in src/core/ or the session data layer.
---

# Core Layer — Implementation Details

Detailed internals for `src/core/`. For architecture overview, see project `CLAUDE.md`.

## Task Model Details

### Task text fields (5-field model)

Each task has `description` (what & why — pre-action context, user-set), `summary` (TL;DR of progress — AI-maintained), `note` (detailed markdown blob — evolving, append via `addNote()` or replace via `updateNote()`), and `conversation_log` (append-only markdown log of user↔agent interactions, auto-managed by the chat handler via `appendConversationLog()`).

The conversation_log is automatically appended server-side after every chat turn where a task is focused — each entry has a `### YYYY-MM-DD HH:MM` timestamp heading followed by `**User:**`/`**AI:**` summaries (truncated to 150/200 chars). The field is optional, starts as `undefined`, and is created on first interaction.

Migration from the old `notes: string[]` is handled by `migrateNotesToFields()` in `task-manager.ts`.

MS To-Do sync: all fields combine into a structured body (`description\n\n---\n\n## Summary\n...\n\n## Notes\n...\n\n## Conversation Log\n...`) via `composeMsTodoBody()`/`parseMsTodoBody()` in `src/integrations/microsoft-todo.ts`.

External sync plugins: each plugin decides how to map these fields to its platform (e.g., description field, comment body, etc.). See the plugin's own documentation for details.

Conversation_log is tail-truncated (recent entries preserved) in Task Context (400 chars), session context (300 tokens), and task_get tool response (1500 chars).

### Task source routing

`TaskSource = 'ms-todo' | 'local' | string` — extensible via plugins. Source is auto-determined at creation by `addTask()` using the `IntegrationRegistry` category claim chain. The agent does not set source manually. `autoPushIfConfigured()` routes push by source.

Migration: legacy `source: 'local'` → `'ms-todo'` with a one-time `scheduleUnsyncedPush()` to push formerly-local tasks.

**Category-source consistency**: `validateCategorySource()` enforces that each category contains only one source type — `addTask()`, `updateTask()` (on category change), and `renameCategory()` throw `CategorySourceConflictError` (HTTP 409) if a cross-source conflict is detected. Plugins may reserve categories via their claim priority. Sync pull paths (`addTaskFull`, `updateTaskRaw`) bypass validation since they are the source of truth.

### Task phase system

Tasks have a 7-state `phase` lifecycle (`TODO` → `IN_PROGRESS` → `AGENT_COMPLETE` → `AWAIT_HUMAN_ACTION` → `HUMAN_VERIFIED` → `POST_WORK_COMPLETED` → `COMPLETE`) that replaces the old 3-state `status`. Status is auto-derived from phase via `PHASE_TO_STATUS` for backward compatibility.

Key mappings in `src/core/phase.ts`:
- `PHASE_TO_WORKFLOW` / `WORKFLOW_TO_PHASE` — external plugin workflow sync (includes both canonical names like `'Implementation'` and common aliases like `'In Progress'`)
- `PHASE_TO_MS_STATUS` / `MS_STATUS_TO_DEFAULT_PHASE` — MS To-Do sync
- Plugin-specific phase groups — preserving local phase during pulls

Phase is the source of truth — `applyPhase()` mutates both `task.phase` and `task.status`. The `VALID_PHASES` set is used for input validation.

### Task dependencies (`depends_on`)

Tasks can declare dependencies on other tasks via `task.depends_on: string[]` (array of task IDs). Dependencies are validated (no self-refs, no circular chains via BFS) and managed through mutation helpers: `add_depends_on`, `remove_depends_on`, `set_depends_on` in `updateTask()`. `isTaskBlocked(task, allTasks)` checks if any dependency is non-COMPLETE.

**Sync**: `autoPushIfConfigured()` calls `sync.updateDependencies(task, task.depends_on ?? [])` after the parallel field batch. Each plugin decides how to represent dependencies on its platform:
- **MS To-Do**: `DependsOn: id1,id2` header in comment body (informational only, roundtripped via `composeMsTodoBody()`/`parseMsTodoBody()`).
- **Other plugins**: Can implement native dependency APIs if the platform supports them. The plugin computes a delta against previous state and pushes add/remove operations.

Store version: v4 (`migrateToV4DependsOn`). The `depends_on` field is optional.

### Session slots (2-slot model)

Each task has typed session fields instead of an unbounded array: `plan_session_id?: string` (current plan session) and `exec_session_id?: string` (current execution session). The old `active_session_ids: string[]` is removed; `session_ids: string[]` is retained as a historical log.

`linkSessionSlot(taskId, sessionId, 'plan'|'exec')` and `clearSessionSlot(taskId, sessionId?, slotType?)` in `task-manager.ts` manage these fields. `session_start` in `tools.ts` checks slots pre-flight and returns a `blocked` response with the existing session info when a slot is occupied. `applyPhase('COMPLETE')` and `toggle-complete` clear both slots.

### Project metadata (`.metadata` tasks)

Each project can have a hidden `.metadata` task whose `description` contains YAML config (e.g., `default_host: remote-dev\ndefault_cwd: /home/user/project`). Retrieved via `getProjectMetadata(category, project)` in `task-manager.ts`. `.metadata` tasks are filtered from `task_query`, REST endpoints, and the TodoPanel UI. Used by `session_start` for host/cwd resolution.

**Session CWD resolution chain** (5 priorities, in `resolveSessionContext()` in `tools.ts`): ① explicit `working_directory` param → ② `task.cwd` → ③ parent task chain walk → ④ project metadata `default_cwd` → ⑤ project memory directory (`~/.open-walnut/memory/projects/{category}/{project}/`). The same chain runs in `handleStart()`/`handleStartSdk()` in `claude-code-session.ts` as defense-in-depth for the RPC/bus path. If all 5 priorities fail, the agent tool returns an actionable error message.

### Subtask model: two systems (child tasks vs. embedded subtasks)

Walnut has two subtask mechanisms. **Child tasks are the canonical model; embedded subtasks are legacy.**

#### Child tasks (the REAL subtask model)

Full Task objects with `parent_task_id` pointing to a parent. They have ALL Task fields (description, summary, note, phase, sessions, dependencies, etc.) and go through the normal task lifecycle (`addTask`, `updateTask`, `completeTask`).

- Inherit `category`, `project`, `source` from parent (unless overridden).
- `getChildTasks(taskIdPrefix)` returns all children.
- `guardActiveChildren()` blocks completing a parent with non-COMPLETE children (`ActiveChildrenError`, HTTP 409).
- UI: indented with `└ child` badge; parent shows `N sub` count.
- Sync: `Parent: <shortId>` header in MS To-Do body and plugin comment fields.

#### Embedded subtasks (LEGACY — will be removed)

`task.subtasks?: Subtask[]` — lightweight embedded checkbox (just `title` + `done`). Modeled after MS To-Do's "checklist items" which are NOT real tasks.

**This is deprecated.** Cannot have descriptions, priorities, phases, sessions, or dependencies. When the integration plugin system lands, this will be removed. All subtasks will be child tasks. Each plugin handles platform mapping internally:
- **MS To-Do**: plugin maps child tasks to checklist items
- External plugins handle platform-specific child mapping (e.g., sub-issues, child tasks)

Legacy code (do not extend):
- Types: `Subtask` interface, `task.subtasks` field in `types.ts`
- Functions: `addSubtask()`, `toggleSubtask()`, `removeSubtask()`, `updateSubtask()` in `task-manager.ts`
- Routes: `POST /api/tasks/:id/subtasks`, `PATCH/DELETE/POST .../:sid/toggle`
- UI: `SubtaskList.tsx`

### Needs-attention notification

`needs_attention?: boolean` — a synced flag that signals "this task needs human input." Set deterministically by `applySessionPhase('triage-sync')` (AGENT_COMPLETE → AWAIT_HUMAN_ACTION sets it) from the turn-complete summary flow in `session-hooks/builtins.ts`. The triage subagent that used to set it via `task_update` was deleted 2026-07 — the session self-report's PHASE_SIGNAL + the `decideNotify()` lookup replaced its judgment.

In the UI, flagged tasks show a red dot (`.task-attention-dot`). Clicking/focusing a task in MainPage auto-clears the flag via `PATCH { needs_attention: false }`. `applyPhase('COMPLETE')` also clears it.

MS To-Do sync: roundtripped via `Attention: true` header line in body. External plugins may include it in their sync payload (typically push-only). The `task_query` tool includes it when truthy and supports `where.needs_attention` filter.

## Session Monitoring — 3-Layer System

Sessions are monitored at three levels, from fastest to slowest:

| Layer | Component | Interval | Scope | Triggers |
|---|---|---|---|---|
| **Per-session** | `ClaudeCodeSession` liveness timer | 3s | Single session | PID death → flush file → emit result/error |
| **Health monitor** | `SessionHealthMonitor` | 30s | All non-terminal sessions | PID death missed by per-session → update DB + emit status |
| **Reconciler** | `reconcileSessions()` | On startup | All non-terminal sessions | Server restart → reconnect alive, mark dead completed |

**Per-session liveness** (`ClaudeCodeSession`): Each session has a 3-second interval that calls `isProcessAlive(pid, 'claude')`. On death, flushes the tailer, reads stderr, and emits `SESSION_RESULT` or `SESSION_ERROR`. This is the fast path — catches most normal exits.

**Health monitor** (`src/core/session-health-monitor.ts`): A 30-second interval that scans all non-terminal sessions from `sessions.json`. If a PID died while `work_status === 'in_progress'`, checks the output file for a result event line to determine `agent_complete` vs `error`. Clears task session slots via `clearSessionSlot()`. Started/stopped with the web server lifecycle.

**Reconciler** (`src/core/session-reconciler.ts`): Runs once on server startup. For each non-terminal session: if `pid + outputFile` exist and the process is alive → reconnectable (SessionRunner attaches and tails from byte 0). Otherwise → mark `completed`, clear task references, emit status change.

## Authoritative Status Reconcile (`src/core/session-reconcile.ts`) — READ BEFORE TOUCHING STATUS LOGIC

NOT the same as `session-reconciler.ts` above (startup-only zombie sweep). This is the **runtime convergence primitive**: every status write in the system is one-directional and event-driven, so losing a single `result` event (tailer freeze, server-restart window, WS drop, replay-guard swallow) wedges `process_status` at 'running' — or the task at IN_PROGRESS — forever. `reconcileProcessStatus()` is the only path that re-checks records against ground truth. Called from `attachToExisting()` (restart recovery) and `SessionHealthMonitor.reconcileStuckRunningSessions()` (periodic, 30s tick with 5-min per-session backoff + 5-per-tick cap).

### Hard-won evidence rules (each cost a real incident — do NOT regress)

1. **The daemon STREAM file is the ONLY authoritative source for turn-ends.** The CLI writes `result` / `session_state_changed` / `task_*` events ONLY to stdout (captured into `STREAMS_DIR/<sid>.jsonl`), NEVER to the canonical `~/.claude/projects/<enc-cwd>/<sid>.jsonl`. Verified empirically across 80+ real sessions: canonical files contain ZERO top-level result/state lines. A v1 of this module read the canonical JSONL via `recoverStateFromJsonl()` — its workStatus was ALWAYS undefined on real data and the convergence never fired, while unit tests passed because fixtures unrealistically wrote result lines into the canonical path. **Lesson: fixtures must mirror the REAL file layout (result events in the stream file, not canonical), and any evidence rule must be validated against live session files before shipping.**
2. **The stream file is blind to turn STARTS.** The CLI never echoes stdin user messages to stdout, so a turn started by a plain-text FIFO send leaves no anchor line. Without compensation, the fold's backward anchor scan lands on a PREVIOUS turn's user line and adopts that turn's stale result as the current turn's verdict (inc-1783644415695: a working session force-converged to a day-old error, 3× in one day). Compensations, both required: (a) the daemon appends a `walnut-injected` marker line at every FIFO delivery (`appendUserMarker` RPC) which `isRealUserLine` accepts as an anchor; (b) positional veto — a result whose byte offset ≤ `record.consumedOffset` was already consumed by the live path ('result-already-consumed'), and a fold that sees init / state:running AFTER a result discards that result.
3. **Missing evidence = no-op, never manufacture completion.** Converge only when ALL hold: real result after the last real user message + trailing idle after it (skipped for error results) + `gatingBgCount === 0` + `!teamActive`. Every string return from `fetchStreamTailFold` means "do not converge".
4. **Clear the PID only when the process is confirmed DEAD.** An error verdict with a live CLI + FIFO keeps its pid — clearing it forced the next send onto the cold `--resume` path, spawning a SECOND CLI against the same session (split-brain).
5. **Terminal converge (stopped/error) must `clearSessionSlot`** — mirrors the live result path; otherwise the reconciled session holds the task's session slot forever.
6. **After a DB converge, sync the live CCS instance** via `sessionRunner.findSessionByClaudeId(sid)?.setProcessStatusFromReconciler(to)` — `writeMessage()` decides mid-turn injection from the in-memory `_processStatus`, which otherwise desyncs from the record.
7. **Health-monitor loop hygiene:** `reconcileStuckRunningSessions` returns the converged-ID Set and the main loop skips those IDs (stale-snapshot overwrite race — same pattern as `checkIdleTimeout`/`checkHungSessions`); backoff timestamp + MAX_PER_TICK slot are recorded INSIDE the try, after the attempt actually starts (a fast refusal must not burn a slot + 5-min backoff).

### Connectivity loss ≠ process death (`reconcileTaskPhases` Rule A guard)

An SSH-tunnel flap makes `isAlive()` return false for every remote session past the 5-min disconnect grace, and the health monitor marks them `error` / `status_reason='remote_unreachable'`. In the SAME tick, `reconcileTaskPhases` Rule A used to see `aliveSessions: 0` and force IN_PROGRESS → AWAIT_HUMAN_ACTION — the "session is running but task shows complete" incident (inc-311a517d, clouddev tunnel flap; self-healed 4 min later when the stream reconnected, but the task badge lied the whole time). The guard: when ALL dead primary sessions are remote with `status_reason === 'remote_unreachable'`, liveness is UNKNOWN — skip the phase force. `isProcessAlive()` is a hard OS fact for local sessions only; for remote sessions during a disconnect it is a guess.

## Ending a CLI process: only the owning daemon may (`src/core/sessions/owner-stop.ts`)

**The server never signals a CLI pid.** A CLI belongs to the daemon that spawned it: that daemon
holds its FIFO, the `.pgid` file in its streams dir (written at spawn, by it or by the earlier daemon
on the same dir it adopted the session from), its registry entry and its start-time identity, and
every death funnels into its `reapSession()`. A pid in a server database row proves nothing about
which daemon spawned the process, or whether the pid still names that process.

Incident (2026-09-26 and 09-27): a test run started an ephemeral server over a hand-copied production
`sessions.sqlite`. Its orphan sweep found `stopped` rows whose pids were alive and looked like claude,
and SIGTERM'd 11, then 9, of the user's live sessions, all owned by the production daemon. The sweep's
only veto ("JSONL written in the last 2 minutes") also fails for any live CLI idling between turns.

How each path ends a process now:

| Path | Asks the owner with | Notes |
|---|---|---|
| Orphan sweep (runner on every start; health monitor on its first tick, then every 120th) | `stop` reason `'orphan'` + `expectPid` + `home` | Capability `orphan-stop-v1`; an old daemon is not asked, so nothing ends. A tick is 30 s while any session is running or idle and 5 min after 3 empty ticks, so the monitor's sweep comes every hour when busy and up to every 10 hours when quiet |
| Health monitor idle timeout, no manager | `stop` reason `'idle'` | Was a SIGINT, SIGTERM, SIGKILL chain on the record's pid; now up to 4 stops in flight per tick |
| Capacity eviction (`checkSessionLimit`) | `stop` reason `'idle'` | Was `process.kill(victim.pid)` from the LOCAL machine, remote pids included; a refusal keeps the session and undoes the teardown mark |
| Task completion (`completeTaskSessions`, also at boot) | `stop` reason `'maintenance'` | Same; ran over every completed task at startup |
| Terminate, ACP engine, no live session or manager | `acpStop` by `acpRuntimeId` (`stopAcpThroughOwner`) | Was a SIGTERM to the record's pid group; a test server never sends it to a shared host (CLI engines stop through the stop coordinator) |
| User stop, live manager stop / kill / stopForIdle | `stop` with `home` + `initiator` | `session-stop.ts`, `remote-session-manager.ts` |

- **Every `stop` names the asking Walnut** (`stopProvenance`, capability `owner-home-v1`): `home` (its
  data dir), `initiator` (`'automatic'` for a reaper or sweep deciding from records, `'human'` for a
  person or for the server driving a live session it holds), and `strict` from an ephemeral server.
  The daemon (`stopOwnerRefusal`, both twins, pinned identical) reads the spawn journal's FIRST line
  for the sid. When it names a Walnut, only that one may stop the session, for any reason. When it
  names none, `strict` is refused, a backfilled line keeps today's behaviour, and an unjournaled sid may
  only be stopped by a human. A session one Walnut started stays that Walnut's after another resumes
  it: the second is refused (fail closed) and the daemon's own idle reaper still ends the process.
  Sessions older than the journal's `home` field have only a backfilled line with no home, and a
  long-lived install holds many of them, which is why production does not refuse those. A daemon without `owner-home-v1` gets today's
  unlabelled stop from production and NO stop from an ephemeral server. The scenario this closes: an
  ephemeral server with `WALNUT_EPHEMERAL_REMOTE_HOSTS=1` over copied data, whose reconnect and health
  recovery write the shared remote daemon's live pids back onto copied rows; its eviction or task
  completion then asked that daemon to stop the user's production CLI.

- **Orphan candidates** (`orphanCandidateSkip`) are records that say someone DECIDED the session ends:
  `status_reason` in `user_stopped`, `user_terminated`, `expected_teardown`, `idle_timeout`,
  `idle_eviction`, with a usable pid, past a 2-minute grace, no pending user stop. An observation
  (`normal_completion`, `liveness_check_failed`, no reason) is a claim a live process refutes: the
  recovery loop relabels those rows and the daemon's idle reaper ends a CLI that really is idle.
- **The daemon's orphan proof** (`orphanStopRefusal` in both twins): the sid is in ITS registry with
  exactly `expectPid`, the `.pgid` file in its streams dir names the same group, the spawn journal's
  first line for the sid names the asking Walnut (`home`, its data dir: a remote host's daemon is shared
  by every Walnut that reaches it), and it is not a previous boot's process; then no output in 5
  minutes, no turn in flight, and no idle-scanner protection (cron, team, running background task,
  turn retry, recent scheduler firing). The check and `stopSessionProcess` (which re-checks the start
  time) run under the per-sid gate a start takes, synchronously up to the first SIGINT, so no daemon
  command can land in between. A refusal answers
  `{stopped:false, reason:'not_owned'|'not_running'|'recent_output'|'protected', detail}` and the
  server logs it at info (`orphan sweep: left alone`).
- **Ephemeral servers** clear every inherited pid at boot, from the column and from the payload JSON
  (`scrubInheritedSessionPids`, called from `startServer` when `IS_EPHEMERAL`), and refuse to start if
  that fails or if their data dir is the production one (`sessions/ephemeral-guard.ts`). The launcher's
  `*.sqlite` skip is not enough on its own: the incident's store was copied by hand around the launcher.
  A fixture that needs a live pid on a row sets it AFTER `startServer` (`tests/e2e/browser/test-server.ts`).
- **Ratchet**: `tests/core/signal-call-ratchet.test.ts` (AST, outside the daemon twins). A raw signal
  (`process.kill` with a non-zero signal, a bare `kill(`, the process-group helpers) may only appear in
  the files that signal a process they spawned themselves, within a per-file budget; session-store code
  (`session-*.ts`, `sessions/`, the runner, the session manager, the sessions routes) has a budget for
  every kill-shaped call, `.kill(` on an object included. A new site fails the quick tier.
- Tests: `tests/core/sessions/owner-stop.test.ts`, `tests/core/sessions/owner-stop-callers.test.ts`,
  `tests/core/sessions/stop-provenance.test.ts`, `tests/core/sessions/recovered-pid-stop.test.ts` (a pid
  recovery wrote back reaches the daemon's own decision code and is refused on every stop path),
  `tests/core/sessions/terminate-detached-acp.test.ts`, `tests/core/sessions/ephemeral-guard.test.ts`,
  `tests/providers/runner-orphan-sweep.test.ts`, `tests/providers/daemon-orphan-stop-twins.test.ts`,
  `tests/core/ephemeral-pid-scrub.test.ts`, `tests/e2e/ephemeral-inherited-pids.test.ts` and the health
  monitor's orphan cases. All stub `process.kill`; none sends a real signal. The one real-process
  proof, `tests/providers/daemon-orphan-stop-real-e2e.test.ts`, only runs with
  `WALNUT_REAL_SIGNAL_TEST=1`: a source-twin daemon ends a dummy it spawned, and a look-alike dummy it
  never spawned survives.

## Session Start Steps

1. Agent tool `session_start` validates the task and calls `sessionRunner.startSession()` directly (awaits the Claude session ID so the response can include a `<session-ref>` tag). Other callers (REST, CLI) still emit `SESSION_START` via the bus.
2. `SessionRunner.handleStart()` builds session context via `buildSessionContext()` (task info, project memory)
3. Creates `ClaudeCodeSession` instance and calls `.send(message, cwd, ...)`
4. Detached child process is spawned, stdout redirected to a JSONL file in `SESSION_STREAMS_DIR`
5. `JsonlTailer` tails the output file, parsing lines and emitting bus events in real-time
6. A per-session liveness timer (3s interval) checks PID — on death, flushes remaining file data and emits `SESSION_RESULT` or `SESSION_ERROR`
7. `startSession()` returns `{ claudeSessionId, title }` after `sessionReady` resolves (30s timeout)

## Session Message Queue (`session-message-queue.ts`)

Messages sent to sessions are persisted to disk before processing. This prevents message loss on server crash. `markProcessing()` atomically moves pending messages to a processing state; `removeProcessed()` cleans up after the session turn completes. Multiple messages arriving while a session is busy are batched and sent as one combined `--resume` call.

## Session Message Delivery — FIFO & JSONL Patterns

Messages reach Claude Code via two paths, which produce different JSONL patterns:

**1. FIFO write (mid-turn delivery):** `processNext()` calls `writeMessage()` → writes JSON to a named FIFO pipe that Claude CLI reads as stdin. Claude logs `queue-operation` entries in its JSONL output:

  - **Pattern A** (consumed): `enqueue` → `dequeue` → normal `human_turn_start`. The session-history parser matches enqueue→dequeue pairs and uses the user message that follows (skips the enqueue to avoid duplicates).
  - **Pattern B** (not consumed before turn ends): `enqueue` only (no matching dequeue). The parser synthesizes a user message from the enqueue content at its chronological position. This happens when the CLI finishes its turn before reading the FIFO.

**2. --resume spawn (between-turn delivery):** When no running process exists (or FIFO write fails), `processNext()` spawns `claude --resume <id> -p "message"`. This produces a normal `human_turn_start` entry (Pattern C). Always reliable.

**Frontend implications:** See `web/src/AGENTS.md` "Session Chat" section for how the frontend deduplicates optimistic messages against these JSONL patterns. Key issue: FIFO-delivered messages may or may not appear in JSONL depending on timing, so the frontend must handle both cases (message in persisted history vs. not).

## Session JSONL Files

Each session has two JSONL files:
- **Canonical JSONL** (`~/.claude/projects/<cwd>/<id>.jsonl`): owned by Claude Code, source of truth for `--resume` (uuid/parentUuid chain) — Walnut must never write to it.
- **Output-stream JSONL** (`~/.open-walnut/sessions/streams/<id>.jsonl`): owned by Walnut (`_outputFile`), used for real-time tailing and `writeSyntheticUserEvent()`. **Local sessions only** — remote sessions have no local output file (`RemoteSessionManager.outputFile` returns null).

## Daemon-Uniform File Access (the ONE rule for reading session data)

**Every read of a Claude Code session-data file goes through `DaemonFileReader(host ?? '__local__')` — local AND remote, no exceptions.** There is no separate "local" transport: `__local__` connects to the in-process local daemon over WebSocket exactly like a remote host connects over an SSH-tunneled one. This is the single most important invariant in the session layer — it makes local and remote one code path, so a bug fixed for one is fixed for both, and the whole surface is trivially decoupled from where the files physically live.

**Session-data files** = Claude Code's per-session artifacts (NOT Walnut's own tasks.json/sessions.json/memory/):
- main transcript `~/.claude/projects/<enc-cwd>/<sid>.jsonl`
- subagent transcripts `…/<sid>/subagents/agent-*.jsonl`
- dynamic-workflow manifests `…/<sid>/workflows/*.json`
- plan files `~/.claude/plans/*.md`
- the Walnut stream copy `<streams-dir>/<sid>.jsonl`

**Reader API** (`src/core/daemon-file-reader.ts`): `readFile(path)` (ENOENT→null, transport-fail→throw), `stat(path)` → `{mtimeMs,size}`, `listDir(path)`, `findSession(sid)` (daemon `fs.find` over `~/.claude/projects`, returns `{content,path}`), `batchReadSubagents(dir)`. Paths may be tilde-prefixed (`~/.claude/...`) — the daemon expands `~` against **its own** `process.env.HOME` (override with `WALNUT_HOME_OVERRIDE`, used only by tests to align the daemon with a mocked `CLAUDE_HOME`).

**Conventions:**
- The sanctioned read helpers live in `session-file-reader.ts` (`readSessionJsonlContent`, `readSubagentContents`, `readSingleSubagentContent`, `readWorkflowManifest`) and `session-history.ts` / `session-changes.ts`. New code reads through these, not raw `fs`.
- Path **builders** (`canonicalJsonlPath`, `remoteJsonlPath`, `encodeProjectPath`) only construct strings — building a path is fine; only direct `fs`/`ssh` **reads** of the files above are violations.
- mtime cache: `stat()` via the daemon before a full read; a `{mtimeMs}` match returns cached parse (`session-history.ts`). `host ?? '__local__'` is the cache key.
- Anti-pattern that WILL break: a `DaemonFileReader('__local__')` read racing ahead of any session `attach()` on a **cold server start** — `connect()` must self-warm the `__local__` pool (it does; mirrors `reconnect()`). Do not assume the pool is warmed by session creation order.

**Known remaining bypasses (local-only fast paths, migrate when touched):** `team-reader.ts`, `subagent-poller.ts`, and `claude-code-session.ts:_areTeammatesStillActive()` still hit the fs directly. They use *partial* reads — `readFileHead` (first N bytes) and byte-offset tailing — which the daemon has no command for yet. Routing them to the daemon requires adding an `fs.readRange`/`fs.head` command first (else every poll full-reads a large JSONL — a perf regression). Until then they are the documented exception, not a template to copy.

**Daemon socket writes (the OTHER one rule):** every daemon→client WebSocket write in `daemon-standalone.ts` MUST go through `safeSend` (never raw `ws.send`). Bun's `ServerWebSocket.send()` returns `0` and **silently drops the message forever** when the socket's backpressure buffer is saturated — one ~28MB `fs.read` response is enough, and everything sent while saturated (concurrent RPC replies, other sessions' `jsonl` stream events on the shared per-host connection) vanishes. `safeSend` queues dropped payloads per-socket and flushes them on Bun's `drain` callback, preserving FIFO order. The plain-Node `daemon-source.ts` does NOT need this (the `ws` package buffers internally and never drops) — see the PARITY NOTE there. Regression: `tests/providers/daemon-ws-backpressure.test.ts` (real binary, 6 concurrent large reads).

**Parser dedup edge case (mid-turn sends):** `parseSessionMessages` skips an `enqueue` when a real `type:"user"` twin with identical text follows within 50 lines (Pattern A). The twin's `content` is USUALLY a string but ~0.3% of the time an **array** of blocks (image refs / long text) — the matcher must extract the first `text` block for the array case, or those enqueues render twice. Content-matching (not enqueue/dequeue/remove counting) is the true invariant: counts don't balance across `--resume`/compact boundaries. Validated on a 2517-enqueue real corpus. Tests: `tests/core/session-parser-dedup.test.ts` (unit), `tests/e2e/mid-stream-message-position.test.ts` (e2e through the real server with a mocked daemon transport).

## Chat History & Compaction Details

The main agent chat persists via `~/.open-walnut/chat-history.json`. Unified `entries[]` array (v2 schema):
- **`tag: 'ai'`**: Conversation messages (Anthropic `ContentBlock[]` format). `getModelContext()` projects them for compaction and context statistics; CLI sessions own their execution history.
- **`tag: 'ui'`**: Display-only notifications. Never sent to the model.

Key API: `addAIMessages()`, `addNotification()`, `getModelContext()`, `getDisplayEntries()`. Auto-migrates from v1 on first read. Also runs one-time migration to mark orphan `tool_result` entries as compacted.

### Compaction process (when full payload exceeds 80% of context window)

Threshold is dynamic: 80% of the model's context window (160K for 200K models, 800K for `[1m]` 1M models). Reads `agent.main_model` from config. See `getContextWindowSize()` in `src/model/model.ts`.

1. **Summarize** (`buildCompactionInstruction()`): one `sendMessage` call receives the message history as `MessageParam[]` and produces a structured checkpoint summary. No tools or memory-flush agent turn run before pruning.
2. **Turn-boundary cutting**: `findTurnBoundaryIndex()` scans backward to keep the last 10 turns without splitting `tool_use`/`tool_result` pairs.
3. **Prune**: old AI entries and UI notifications are deleted together; kept entries are slimmed. At least four old messages are required.
4. **Persist**: the summary is stored as `compactionSummary`. This store-level compaction does not replace the CLI session's own context management.

**Why delete instead of marking `compacted: true`?** The model never reads old entries (they're filtered out), and the UI notification source data always lives in each subagent's own JSONL file (`~/.open-walnut/sessions/streams/<runId>.jsonl`). Retaining them only grew `chat-history.json` unboundedly (35 MB observed before the switch) and froze the Node event loop on every turn. The `compacted: true` marker is still checked defensively in `getModelContext()` for backward compatibility with older data, but new compactions never produce it.

**Defense layer**: `getModelContext()` strips any user message whose `tool_result` blocks have no matching `tool_use` in the preceding assistant message.

The REST `/compact` endpoint calls `triggerBackgroundCompaction()` in `src/web/background-compaction.ts`, which uses the one-shot summarizer from `createCompactionCallbacks()` in `src/web/routes/chat.ts`. No in-process chat loop triggers automatic compaction.