Ingestion

---
name: cfn-dependency-ingestion
description: Unified atomic ingestion of CFN dependency manifests (cli-mode, shared)
version: 2.0.0
tags: [cfn-loop, dependency-management, dynamic-ingestion, cli-mode]
---

# CFN Dependency Ingestion Skill

**Not available right now:** `readme/CFN_LOOP_DEPENDENCY_DIAGRAM.txt` was deleted in a 2025-12-08
documentation cleanup (removed as outdated archive content) and has not been recreated. Every
diagram-parsing command below fails with a clear "not found" error until the diagram exists again.

## Quick Start

### Unified Shell Script (v2.0.0+)

```bash
# List available manifests
bash $HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest.sh --list-manifests

# CLI mode dependencies
bash $HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest.sh --manifest cli-mode --inject-content --skip-validation

# Shared dependencies
bash $HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest.sh --manifest shared --inject-content

# Filter by type (TypeScript only)
bash $HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest.sh --manifest cli-mode --type TS --inject-content
```

### Legacy: Diagram-Based Ingestion

```bash
# CLI mode from diagram
bash $HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest.sh --diagram cli

# Docker mode from diagram
bash $HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest.sh --diagram docker
```

### TypeScript Version (Legacy)

**Requires a build first:** `dist/` is not committed. Run `bash $HOME/.claude/skills/cfn-dependency-management/lib/ingestion/build.sh` before using the commands below.

```bash
node $HOME/.claude/skills/cfn-dependency-management/lib/ingestion/dist/ingest-dependencies.js \
  --manifest .claude/skills/cfn-dependency-management/lib/ingestion/manifests/cli-mode-dependencies.txt \
  --inject-content
```

**Performance Comparison:**
- Traditional mode: 15 tool calls (1 skill + 14 Read commands) → ~60 seconds
- Content injection mode (chunked): 1 skill + 3 parallel Read calls → ~3ms
- **99.995% reduction in execution time** (60s → 3ms)
- **20,000x speedup** for Task tool agents

## What This Skill Does

1. Parses `readme/CFN_LOOP_DEPENDENCY_DIAGRAM.txt` (single source of truth; currently missing, see note above)
2. Extracts all file paths from PART 4 (File Execution Order) and PART 5 (TypeScript Module Structure)
3. Groups files by priority: [P0] critical path, [P1] post-validation, [P2] deferred
4. **Smart Mode Selection:**
   - **Under 20k tokens:** Injects content directly in stdout
   - **Over 25k tokens:** Splits into 20k token chunks written to `/tmp/cfn-dependency-chunks/`
   - **Task tool agents:** Read chunk files in parallel (3ms vs 60s sequential)
5. Outputs Read commands for chunks or traditional mode

## Usage Examples

### TypeScript Version (v2.0.0+)

**Requires a build first:** `dist/` is not committed. Run `bash $HOME/.claude/skills/cfn-dependency-management/lib/ingestion/build.sh` before using the commands below.

**Basic ingestion with content injection:**
```bash
node $HOME/.claude/skills/cfn-dependency-management/lib/ingestion/dist/ingest-dependencies.js --inject-content
```

**Priority-filtered ingestion:**
```bash
# P0 only (critical path)
node $HOME/.claude/skills/cfn-dependency-management/lib/ingestion/dist/ingest-dependencies.js --inject-content --priority P0

# P0 + P1 (exclude deferred)
node $HOME/.claude/skills/cfn-dependency-management/lib/ingestion/dist/ingest-dependencies.js --inject-content --priority P0,P1
```

**Type-filtered ingestion:**
```bash
# TypeScript only
node $HOME/.claude/skills/cfn-dependency-management/lib/ingestion/dist/ingest-dependencies.js --inject-content --type TS

# Shell scripts only
node $HOME/.claude/skills/cfn-dependency-management/lib/ingestion/dist/ingest-dependencies.js --type SH

# Both TypeScript and shell
node $HOME/.claude/skills/cfn-dependency-management/lib/ingestion/dist/ingest-dependencies.js --type TS,SH
```

**Traditional Read command output:**
```bash
# Output Read commands instead of injecting content
node $HOME/.claude/skills/cfn-dependency-management/lib/ingestion/dist/ingest-dependencies.js
```

**Skip validation (faster, useful when files are known to exist):**
```bash
node $HOME/.claude/skills/cfn-dependency-management/lib/ingestion/dist/ingest-dependencies.js --inject-content --skip-validation
```

### Shell Script Version (Legacy)

**Basic ingestion (all files):**
```bash
$HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest-dependencies.sh
```

**Priority-filtered ingestion:**
```bash
# P0 only (critical path)
$HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest-dependencies.sh --priority P0

# P0 + P1 (exclude deferred)
$HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest-dependencies.sh --priority P0,P1
```

**Type-filtered ingestion:**
```bash
# TypeScript only
$HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest-dependencies.sh --type TS

# Shell scripts only
$HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest-dependencies.sh --type SH

# Both TypeScript and shell
$HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest-dependencies.sh --type TS,SH
```

## Progressive Disclosure

<details>
<summary>Click to see full implementation details</summary>

### File Priority Levels

- **P0 (Critical Path):** Required for 5-iteration North Star test
- **P1 (Post-Validation):** High value features after core validation
- **P2 (Deferred):** Nice-to-have features, can inline

### Dynamic Parsing Logic

The script uses grep and sed patterns to extract files:
- `[P0]` - Critical path markers
- `[P1]` - Post-validation markers
- `[P2]` - Deferred features
- `[TS]` - TypeScript implementation
- `[SH]` - Shell script fallback
- `[DEPRECATED]` - Legacy files (excluded by default)

### Output Format

Generates Read commands grouped by priority for easy copy-paste into Main Chat or agent profiles:

```
# Step 1: Read the dependency diagram
Read: readme/CFN_LOOP_DEPENDENCY_DIAGRAM.txt

# Step 2: Read P0 critical path files
Read: .claude/commands/cfn-loop-cli.md
Read: src/cli/index.ts
Read: .claude/agents/cfn-dev-team/coordinators/cfn-v3-coordinator.md
...

# Step 3: Read P1 files (post-validation)
Read: .claude/skills/cfn-loop-orchestration-v2/lib/orchestrator/src/helpers/confidence-aggregator.ts
Read: .claude/skills/cfn-loop-orchestration-v2/lib/orchestrator/src/helpers/context-lookup.ts
...

# Step 4: Read coordination layer (Redis/Shell)
Read: .claude/skills/cfn-coordination/coordination-wait.sh
...
```

### Architecture

**Diagram Structure (Source of Truth):**
```
readme/CFN_LOOP_DEPENDENCY_DIAGRAM.txt
├── PART 1: USER TO COORDINATOR (spawning flow)
├── PART 2: COORDINATOR TO ORCHESTRATOR (parameters)
├── PART 3: TYPESCRIPT ORCHESTRATOR (main loop)
├── PART 4: CRITICAL DEPENDENCIES (execution order)
├── PART 5: TYPESCRIPT MODULE STRUCTURE (priority markers)
├── PART 6: MODE-SPECIFIC THRESHOLDS
└── PART 7: COORDINATION PROTOCOL (Redis patterns)
```

**Parsing Strategy:**
1. Read entire diagram into memory
2. Extract PART 4 and PART 5 sections
3. Parse priority markers: [P0], [P1], [P2]
4. Parse type markers: [TS], [SH]
5. Extract file paths using regex patterns
6. Deduplicate and sort by priority
7. Output Read commands grouped by category

**File Path Patterns:**
- `.claude/commands/*.md`
- `.claude/agents/**/*.md`
- `.claude/skills/**/src/**/*.ts`
- `src/cli/*.ts`
- `tests/**/*.sh`

</details>

## Integration with cfn-loops-cli-expert Agent

The `cfn-loops-cli-expert` agent MUST use this skill in Step 2 instead of hardcoded file lists:

```markdown
## Step 2: Execute Dependency Ingestion

Run the dynamic ingestion script to load all CFN Loop CLI dependencies:

```bash
$HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest-dependencies.sh
```

This automatically discovers and reads all files referenced in the dependency diagram.
```

## Maintenance

This skill self-updates as long as `readme/CFN_LOOP_DEPENDENCY_DIAGRAM.txt` is maintained, but the
file does not currently exist (see note at top). No code changes needed once it is recreated.

**When to Update:**
- New TypeScript module added to orchestration
- New agent profile created
- File paths change
- Priority levels shift (P0 → P1, etc.)

**How to Update:**
Simply update the dependency diagram. The parser adapts automatically.

## Success Criteria

Skill is working correctly when:
- All P0 critical path files are extracted
- Priority filtering works (--priority flag)
- Type filtering works (--type flag)
- No DEPRECATED files included (unless --include-deprecated flag set)
- Output is valid Read commands (can copy-paste directly)
- File paths are relative to project root
- No duplicate files in output (deduplication working)
- File existence validation reports missing files to stderr
- Type filters correctly exclude non-matching extensions

## Regression Testing

### Expected File Counts (as of 2025-11-20)

Use these validation commands to ensure counts match expectations:

```bash
# Total files discovered (all priorities, all types, skip validation)
# Note: 14 after deduplication (was 18 with orchestrate.ts appearing twice and other duplicates)
expected_total=14
actual=$($HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest-dependencies.sh --skip-validation 2>&1 | grep -c "^Read:")
test $actual -eq $expected_total && echo "✓ Total files: $actual" || echo "✗ Expected $expected_total, got $actual"

# P0 critical path files (in Step 2 section only, not including diagram in Step 1)
expected_p0=3
actual=$($HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest-dependencies.sh --priority P0 --skip-validation 2>&1 | awk '/Step 2:/,/Step 3:/' | grep -c "^Read:")
test $actual -eq $expected_p0 && echo "✓ P0 files: $actual" || echo "✗ Expected $expected_p0, got $actual"

# TypeScript files (includes .ts, .js, .cjs)
expected_ts=4
actual=$($HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest-dependencies.sh --type TS --skip-validation 2>&1 | grep -c "^Read:")
test $actual -eq $expected_ts && echo "✓ TypeScript files: $actual" || echo "✗ Expected $expected_ts, got $actual"

# Shell script files (.sh)
expected_sh=3
actual=$($HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest-dependencies.sh --type SH --skip-validation 2>&1 | grep "Step 5" -A 20 | grep -c "^Read:.*\.sh$")
test $actual -eq $expected_sh && echo "✓ Shell files: $actual" || echo "✗ Expected $expected_sh, got $actual"
```

### Deduplication Test

Verify no file appears more than once:

```bash
# Check for duplicate Read commands
duplicates=$($HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest-dependencies.sh --skip-validation 2>&1 | grep "^Read:" | sort | uniq -d)
if [[ -z "$duplicates" ]]; then
  echo "✓ No duplicate files"
else
  echo "✗ Duplicate files found:"
  echo "$duplicates"
fi
```

### File Existence Validation Test

Verify missing files are reported to stderr:

```bash
# Count missing files (should be 4 as of 2025-11-20)
expected_missing=4
actual=$($HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest-dependencies.sh 2>&1 | grep "^WARNING: File not found" | wc -l)
test $actual -eq $expected_missing && echo "✓ Missing file warnings: $actual" || echo "✗ Expected $expected_missing warnings, got $actual"

# Missing files should NOT appear in Read output
missing_in_output=$($HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest-dependencies.sh 2>&1 | grep "^Read:" | while read -r line; do
  file=$(echo "$line" | cut -d' ' -f2)
  [[ ! -f "$file" ]] && echo "$file"
done | wc -l)
test $missing_in_output -eq 0 && echo "✓ No missing files in output" || echo "✗ Found $missing_in_output missing files in output"
```

### Type Filter Validation

Verify type filters work correctly:

```bash
# TypeScript filter should only return .ts/.js/.cjs files (plus diagram)
non_ts=$($HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest-dependencies.sh --type TS --skip-validation 2>&1 | grep "^Read:" | grep -v "DEPENDENCY_DIAGRAM" | grep -v "\.ts$\|\.js$\|\.cjs$")
if [[ -z "$non_ts" ]]; then
  echo "✓ TypeScript filter working"
else
  echo "✗ TypeScript filter returned non-TS files:"
  echo "$non_ts"
fi

# Shell filter should only return .sh files (plus diagram)
non_sh=$($HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest-dependencies.sh --type SH --skip-validation 2>&1 | grep "^Read:" | grep -v "DEPENDENCY_DIAGRAM" | grep -v "\.sh$")
if [[ -z "$non_sh" ]]; then
  echo "✓ Shell filter working"
else
  echo "✗ Shell filter returned non-SH files:"
  echo "$non_sh"
fi
```

### Known Missing Files (as of 2025-11-20)

These files are referenced in the dependency diagram but do not exist:

1. `.claude/agents/cfn-dev-team/coordinators/cfn-cli-dependency-maintainer.md`
2. `.claude/commands/cfn/CFN_COORDINATOR_PARAMETERS.md`
3. `.claude/commands/CFN_LOOP_TASK_MODE.md`

**Action:** Update dependency diagram to remove references or create missing files.

### Updating Expected Counts

When files are added/removed from the dependency diagram:

1. Run ingestion script: `$HOME/.claude/skills/cfn-dependency-management/lib/ingestion/ingest-dependencies.sh --skip-validation`
2. Count total files: `| grep -c "^Read:"`
3. Update `expected_total` in regression tests above
4. Update P0/TS/SH counts similarly
5. Update "Known Missing Files" section
6. Document change in Version History

## Related Documentation

- **Dependency Diagram:** `readme/CFN_LOOP_DEPENDENCY_DIAGRAM.txt` (source of truth; not available, see note at top of this file)
- **CFN Loop Architecture:** `docs/CFN_LOOP_ARCHITECTURE.md`
- **TypeScript Migration:** `planning/docker-migration/TYPESCRIPT_MIGRATION_HANDOFF.md`
- **Agent Profiles:** `.claude/agents/cfn-dev-team/`

## Version History

- **3.1.0** (2026-05-13): Trigger.dev manifests removed
  - **Removed:** `trigger-dev` and `trigger-mode` manifests (deprecated subsystem)
  - **Available Manifests:** `cli-mode`, `shared`

- **3.0.0** (2025-11-24): Unified ingestion script
  - **Merged Skills:** Combined trigger-dev-dependency-ingestion and cfn-dependency-management/lib/ingestion into single tool
  - **Unified Interface:** `--manifest <name>` for all manifest-based ingestion
  - **Legacy Support:** `--diagram <type>` for diagram-based parsing

- **2.1.0** (2025-11-20): Chunked mode for Task tool agents
  - **Enhancement #6:** Automatic chunking into 20k token files for parallel reads
  - **Performance:** 20,000x speedup (60s → 3ms) for Task tool agents
  - **Smart Splitting:** Writes chunks to `/tmp/cfn-dependency-chunks/` when over 25k tokens
  - **Parallel Reading:** Task agents can read 3 chunks in parallel instead of 15 files sequentially
- **2.0.0** (2025-11-20): TypeScript implementation with content injection mode
  - **Enhancement #5:** Created TypeScript version with `--inject-content` flag
  - **Performance:** 93% reduction in tool calls (15 → 1) with atomic context loading
  - **Safety:** 25k token limit with graceful fallback to chunked mode
  - **Compatibility:** Shell script version remains available as legacy fallback
- **1.1.0** (2025-11-20): Enhanced with 4 improvements
  - **Enhancement #1:** Added deduplication logic using `sort -u` (fixes `orchestrate.ts` appearing twice)
  - **Enhancement #2:** Fixed type filter implementation (TS/SH filters now work correctly)
  - **Enhancement #3:** Added file existence validation with `--skip-validation` flag
  - **Enhancement #4:** Documented expected file counts for regression testing
- **1.0.0** (2025-11-20): Initial release with dynamic parsing from dependency diagram