Fabric Pipeline

---
name: fabric-pipeline
description: Create, deploy, and test a Fabric Data Factory pipeline that chains all topic notebooks in order (download → bronze → dq_bronze → silver → dq_silver → gold → dq_gold). Use after all notebooks for a topic have been individually deployed and smoke-tested.
---

# fabric-pipeline

## MUST

- All notebooks for the topic must be deployed and individually smoke-tested before creating the pipeline
- Pipeline names follow the convention `pipeline_<topic>`
- Activities execute in strict order — each depends on its predecessor with `Succeeded`; failures never cascade
- For a **new** pipeline, always use `test` (create + run + monitor) — never report completion after `create` alone; activity wiring is only validated by a successful end-to-end run
- Use `create` alone only when the human explicitly says to skip the run (e.g. they will schedule the first run manually)
- Report STATUS: Completed or STATUS: Failed to orchestrator and stop — never re-run autonomously
- Each pipeline run consumes Fabric capacity; always await human approval before re-running a failed pipeline

## PREFER

- `test` command over separate `create` + `run` + `status` when running an existing pipeline
- Auto-discover over `--notebooks` — auto-discover reads naming conventions directly from the workspace
- Check the Fabric portal Run History for per-activity logs on failure

## AVOID

- Creating a pipeline before all its notebooks are deployed in Fabric
- Re-running on failure without human approval

## Commands

```bash
# 0. Confirm active workspace (every session, before any pipeline action)
fabric-vibe workspace switch list
# Stop. Ask: "Active workspace is '<displayName>'. Confirm to proceed?"
# Do not run pipeline manage until the human confirms.
# To deploy to a different workspace instead, use workspace transfer:
#   fabric-vibe workspace transfer --pipeline <topic> --to <displayName>

# Full cycle: auto-discover notebooks, create/update pipeline, run, and monitor
fabric-vibe pipeline manage test --topic lux_energy_price

# Create or update only (no run)
fabric-vibe pipeline manage create --topic lux_energy_price

# Explicit ordered list (use when auto-discover picks up the wrong notebooks)
fabric-vibe pipeline manage create --topic lux_energy_price \
    --notebooks download_lux_energy_price,bronze_lux_energy_price,dq_bronze_lux_energy_price

# Trigger an already-deployed pipeline (no monitoring)
fabric-vibe pipeline manage run --pipeline pipeline_lux_energy_price

# Monitor an existing run to completion
fabric-vibe pipeline manage status \
    --pipeline pipeline_lux_energy_price --instance <job-instance-id>

# List all data pipelines in the workspace
fabric-vibe pipeline manage list
```

## Activity Ordering

The tool sorts discovered notebooks by naming prefix. Use `--notebooks` to override.

| Prefix | Order | Layer responsibility |
|---|---|---|
| `download_` | 1 | Fetch source data to sandbox |
| `bronze_` | 2 | Ingest raw files to Bronze Delta |
| `dq_bronze_` | 3 | Validate Bronze layer |
| `silver_` | 4 | Clean and transform to Silver |
| `dq_silver_` | 5 | Validate Silver layer |
| `gold_` | 6 | Aggregate to Gold |
| `dq_gold_` | 7 | Validate Gold layer |

Notebooks that do not match a known prefix are appended at the end in alphabetical order.

## STATUS Handling

```
STATUS: Completed  → PASS — report to orchestrator
STATUS: Failed     → FAIL — report pipeline name, job instance ID, and failureReason.
                     STOP. Await human approval before any re-run.
STATUS: Cancelled  → report to orchestrator. Do not re-run without human approval.
```

For per-activity failure details: Fabric portal → Workspace → pipeline item → Run history → select the failed run.