Build Automated Component Pipelines

---
name: build-automated-component-pipelines
description: "Build automated pipelines with reusable components, data flow between nodes, and state management."
category: "Data & Analytics"
author: community
version: "1.0.0"
icon: chart-bar
---

## Architecture

Workflows live in `workflows/` with components and flows:

```
workflows/
├── index.md                 # Inventory with tags
├── components/
│   ├── connections/         # Auth configs
│   ├── nodes/               # Reusable operations
│   └── triggers/            # Event sources
└── flows/{name}/
    ├── flow.md              # Definition
    ├── config.yaml          # Parameters
    ├── run.sh               # Executable
    ├── state/               # Persistent between runs
    └── logs/
```

## Quick Reference

| Topic | File |
|-------|------|
| Directory layout, naming, formats | `structure.md` |
| Data passing between nodes | `data-flow.md` |
| Cursor, seen set, checkpoint | `state.md` |
| Retry, rollback, idempotency | `errors.md` |
| Connections, nodes, triggers | `components.md` |
| Create, test, update, archive | `lifecycle.md` |

## Requirements

- **jq** — JSON processing
- **yq** — YAML config parsing
- **curl** — HTTP requests
- **flock** — Lock files to prevent concurrent runs
- Secrets in macOS Keychain (`security find-generic-password`)

## Data Storage

- **Location:** `workflows/` in workspace root
- **State:** `flows/{name}/state/` — cursor.json, seen.json, checkpoint.json
- **Logs:** `flows/{name}/logs/` — JSONL per run
- **Data:** `flows/{name}/data/` — intermediate files between nodes

## Core Rules

### 1. Data Flow Pattern
Each node writes output to `data/{NN}-{name}.json`. Next node reads it.
```bash
curl ... > data/01-fetch.json
jq '...' data/01-fetch.json > data/02-filter.json
```
Breaking this pattern = nodes can't communicate.

### 2. Error Declaration
Every node in flow.md MUST declare:
- **On error:** retry(N) | fail | continue | alert
- **On empty:** skip | continue | fail

Undefined behavior = unpredictable workflow.

### 3. Lock Files
Prevent concurrent runs:
```bash
LOCKFILE="/tmp/workflow-${NAME}.lock"
exec 200>"$LOCKFILE"
flock -n 200 || exit 0
```

### 4. State Files
| File | Purpose |
|------|---------|
| cursor.json | "Where did I leave off?" |
| seen.json | "What have I processed?" |
| checkpoint.json | "Multi-step recovery" |

### 5. Component Reuse
Before creating new connections/nodes/triggers:
```bash
ls workflows/components/connections/
ls workflows/components/nodes/
```
Use existing. Update "Workflows Using This" when adding.

---

**Related:** For LLM-driven multi-phase processes, see the `cycle` skill.