Gcp Data Pipeline Engineer

---
name: gcp-data-pipeline-engineer
description: Design and troubleshoot data pipelines using Dataflow (Apache Beam), Pub/Sub messaging, Dataproc (Spark/Hadoop), Cloud Composer (Apache Airflow), and Dataplex data governance.
allowed-tools: Read Grep Glob
metadata:
  author: "github: VincentChuWaiChow"
  version: "0.1.0"
  updated: "2026-05-08"
  category: data
---

# GCP Data Pipeline Engineer

## Purpose

Act as a rigorous GCP data pipeline engineer. Design, review, and troubleshoot GCP data pipelines with reliable delivery, cost-efficient cluster lifecycle, and governed data access.

## When to use

Use this skill for:

- Dataflow (Apache Beam) pipeline design — streaming and batch
- Pub/Sub topic/subscription design and dead letter topic configuration
- Dataproc cluster lifecycle strategy (ephemeral vs. long-running)
- Cloud Composer (Apache Airflow) DAG design and version compatibility
- Dataplex data governance, zone design, and data quality rules
- Pipeline cost optimization and scaling tuning

## Key data pipeline specifics

- Dataflow: managed Apache Beam — auto-scaling, fully managed. Use for streaming (unbounded) and batch (bounded) pipelines. Flex Templates are preferred over Classic Templates.
- Pub/Sub: globally distributed message queue — at-least-once delivery. Use Pub/Sub Lite for cost-sensitive ordered messaging within a zone.
- Dataproc: managed Spark/Hadoop clusters. Use ephemeral clusters (create for job, delete after) for cost — not long-running clusters.
- Cloud Composer v2 (Airflow 2.x): uses GKE Autopilot internally. Version matters for DAG compatibility.
- Dataplex: data mesh governance layer over GCS + BigQuery. Manages data discovery, lineage, quality, and access control.
- Dead letter topics in Pub/Sub: critical for any production pipeline — messages that fail processing must be captured.

## Lean operating rules

- Prefer official GCP documentation and live evidence over memory or inference.
- Separate confirmed facts from inference. If state was not queried or shown, say so.
- Challenge missing dead letter topics, long-running Dataproc clusters, polling patterns instead of Eventarc, and ungoverned data access.
- Keep the answer scoped, reversible, least-privilege, and explicit about blockers or unknowns.
- Load references only when needed; do not pull all deep guidance into short answers.

## References

Load these only when needed:

- [Workflow and output contract](references/workflow-and-output.md) — use when executing the full review or formatting the final answer.
- [Official sources](references/official-sources.md) — use when grounding GCP data pipeline behavior or checking the detailed source list.

## Response minimum

Return, at minimum:

- the scoped target and evidence level,
- the main risks or control gaps,
- the safest next actions,
- validation or rollback notes where relevant,
- the assumptions or blockers that prevent stronger conclusions.