Big Data Pipeline

---
name: big-data-pipeline
description: "Use when building big data pipelines."
version: 1.0.0
author: Hermes Agent
license: MIT
platforms: [linux, macos, windows]
metadata:
  hermes:
    tags: [data-science, big-data, pipeline, etl]
    related_skills: ['etl-pipeline-design']
---

## Overview
Build scalable ETL/ELT pipelines for processing large datasets.

## When to Use
- "Big Data Pipeline design and architecture"
- "Best practices for Big Data Pipeline"
- "Big Data Pipeline implementation and deployment"
- "Big Data Pipeline optimization and monitoring"
- "Big Data Pipeline troubleshooting and scaling"

## Key Concepts
1. Foundational concepts
2. Implementation approaches
3. Testing and validation

## Implementation Patterns
1. Define clear requirements and specifications
2. Choose appropriate tools and frameworks
3. Implement with modular, maintainable code
4. Write tests and automate verification
5. Document architecture and decisions
6. Monitor performance and iterate

## Common Pitfalls
1. **Not accounting for constraints** — resource or timeline limitations
2. **Ignoring industry standards** — not following established best practices
3. **Poor stakeholder alignment** — conflicting requirements
4. **Inadequate testing** — no validation of critical functions
5. **Not documenting decisions** — lost knowledge transfer

## Verification Checklist
- [ ] Requirements documented
- [ ] Standards reviewed
- [ ] Design validated
- [ ] Testing established
- [ ] Documentation complete