Pipeline Foundations

---
name: pipeline-foundations
description: "Build and debug DataFlow operator, pipeline, storage, prompt, and
  wrapper foundations."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# pipeline-foundations

Use this sub-skill when a task needs the DataFlow core mechanics: custom `OperatorABC` classes, `PipelineABC` subclasses, compile-time key validation, storage/cache steps, prompt-template restrictions, `draw_graph`, or offline non-LLM smoke checks.

## Load order

1. Read `references/api-reference.md` for the public classes, constructor parameters, and operator/pipeline contracts.
2. Read `references/storage-and-data-formats.md` before choosing `FileStorage`, `LazyFileStorage`, batched storage, or database-backed storage.
3. Read `references/pipeline-patterns.md` when implementing or debugging a pipeline skeleton.
4. Read `references/troubleshooting.md` when compile errors, missing keys, storage step errors, prompt type errors, graph rendering, wrappers, or non-TTY diagnostics fail.
5. Use `scripts/smoke_pipeline_foundations.py` to prove a tiny offline custom-operator pipeline.
6. Use `scripts/validate_tabular_input.py` before running operators that require specific columns.

## Scope

This sub-skill covers:

- `PipelineABC`, `BatchedPipelineABC`, and `StreamBatchedPipelineABC` compile and forward behavior.
- `OperatorABC.run(storage, input_*, output_*)` conventions and compile-time key matching.
- `FileStorage`, `LazyFileStorage`, `DummyStorage`, `BatchedFileStorage`, `StreamBatchedFileStorage`, and `MyScaleDBStorage` selection.
- `prompt_restrict`, `PromptABC`, and `DIYPromptABC` prompt-template guardrails.
- `BatchWrapper` and non-LLM foundation smoke patterns.
- `draw_graph(port=0, hide_no_changed_keys=True)` usage and dependency caveats.

## Routing boundaries

- Route API keys, local model serving, OpenAI-compatible endpoints, WebUI, and CLI command choices to the `serving-cli` sub-skill.
- Route text-specific, document-specific, vision, RAG, and operator-catalog workflow assembly to the `text-workflows` or `document-vision-rag` sub-skills.
- Route Ray actor wrapping, distributed execution, and GPU resource allocation to the `rayorch-acceleration` sub-skill.
- Stay here for generic pipeline shape, storage semantics, column validation, and offline reproducibility.

## Fast operating rules

- Always call `pipeline.compile()` once before `pipeline.forward()` on a `PipelineABC` subclass.
- In a pipeline `forward`, call `self.some_operator.run(storage=self.storage.step(), input_key="existing_column", output_key="new_column")` or the equivalent positional storage form.
- Treat `storage.step()` as the step boundary: step 0 reads the first input file; operator output is written to the next step cache file.
- Use `input_*` parameters only for columns that already exist in the input data or were produced by earlier operators. Use `output_*` for columns the operator writes.
- Compile failures mentioning `Key Matching Error` are usually schema or `input_*` typos, not model failures.
- Prefer `FileStorage` for deterministic offline smoke tests; avoid `hf:` and `ms:` sources unless the workflow is allowed to use network access.