Cicd Pipelines

---
name: cicd-pipelines
description: CI/CD pipeline design and DevOps automation — use when the user mentions GitHub Actions, GitLab CI, Jenkins, Terraform, infrastructure as code, DevSecOps, ArgoCD, Kubernetes deployment automation, or pipeline configuration YAML. NOT for release orchestration or semantic-release workflows (use git-workflow), NOT for Docker containers or Dockerfiles (use docker-containerization), NOT for git branching or commits (use git-workflow).
license: MIT
---

# CI/CD Pipelines - Comprehensive DevOps Skill

A unified skill for CI/CD pipeline design, DevOps automation, infrastructure as code, GitOps deployment automation, security scanning, and enterprise pipeline readiness across major platforms.

## When to Use This Skill

Use this skill when:

### CI/CD Pipeline Development
- Creating new CI/CD workflows (GitHub Actions, GitLab CI, Jenkins)
- Debugging pipeline failures or flaky tests
- Optimizing slow builds or test suites
- Implementing caching strategies
- Setting up deployment workflows
- Implementing matrix builds or test sharding
- Configuring multi-environment deployments

### DevSecOps & Security
- Securing pipelines (secrets, OIDC, supply chain)
- Implementing security scanning (SAST, DAST, SCA)
- Container image vulnerability scanning inside a pipeline
- Secret detection and management
- Enterprise readiness assessment (OpenSSF compliance)

### Infrastructure as Code
- Terraform module development
- CloudFormation/CDK templates
- Deploying Kubernetes manifests and Helm charts from CI/CD
- GitOps workflows (ArgoCD, Flux)

### Container Pipeline Integration
- Building and signing container images from an existing Dockerfile
- Publishing to registries from CI/CD
- Scanning container images with tools such as Trivy or Snyk
- Deploying containerized services through GitOps

For Dockerfile design, Docker Compose, local container environments, or container runtime architecture, switch to `docker-containerization`.

### Release Management
- Semantic versioning automation
- Changelog generation
- GitHub/GitLab release creation
- Artifact signing and provenance

## Quick Start

### 1. Creating a New Pipeline

**Decision tree:**
```
What are you building?
+-- Node.js/Frontend --> templates/github-actions/node-ci.yml | templates/gitlab-ci/node-ci.yml
+-- Python --> templates/github-actions/python-ci.yml | templates/gitlab-ci/python-ci.yml
+-- Go --> templates/github-actions/go-ci.yml | templates/gitlab-ci/go-ci.yml
+-- Container image pipeline --> templates/github-actions/docker-build.yml | templates/gitlab-ci/docker-build.yml
+-- Security Scanning --> templates/github-actions/security-scan.yml | templates/gitlab-ci/security-scan.yml
```

**Basic pipeline structure:**
```yaml
# 1. Fast feedback (lint, format) - <1 min
# 2. Unit tests - 1-5 min
# 3. Integration tests - 5-15 min
# 4. Build artifacts
# 5. E2E tests (optional, main branch only) - 15-30 min
# 6. Deploy (with approval gates)
```

### 2. Optimizing Pipeline Performance

**Quick wins checklist:**
- [ ] Add dependency caching (50-90% faster builds)
- [ ] Remove unnecessary `needs` dependencies
- [ ] Add path filters to skip unnecessary runs
- [ ] Use `npm ci` instead of `npm install`
- [ ] Add job timeouts to prevent hung builds
- [ ] Enable concurrency cancellation for duplicate runs

**Analyze existing pipeline:**
```bash
# Use the pipeline analyzer script
python3 scripts/pipeline_analyzer.py --platform github --workflow .github/workflows/ci.yml
```

### 3. Securing Your Pipeline

**Essential security checklist:**
- [ ] Use OIDC instead of static credentials
- [ ] Pin actions/includes to commit SHAs
- [ ] Use minimal permissions
- [ ] Enable secret scanning
- [ ] Add vulnerability scanning (dependencies, containers)
- [ ] Implement branch protection
- [ ] Separate test from deploy workflows

**OIDC Authentication (GitHub Actions to AWS):**
```yaml
permissions:
  id-token: write
  contents: read

steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::123456789:role/GitHubActionsRole
      aws-region: us-east-1
```

## Core Capabilities

| Domain | Tools & Technologies |
|--------|---------------------|
| **CI/CD Platforms** | GitHub Actions, GitLab CI, Jenkins |
| **Infrastructure as Code** | Terraform, AWS CDK, CloudFormation, Pulumi |
| **Container Pipeline Integration** | Image build/publish steps, Trivy/Snyk scanning, Cosign signing, Kubernetes/Helm deploy jobs |
| **GitOps** | ArgoCD, Flux |
| **Security Scanning** | CodeQL, Semgrep, Trivy, Snyk, TruffleHog |
| **Cloud Platforms** | AWS, Azure, GCP, Cloudflare |
| **Release Artifacts & Provenance** | Cosign, SLSA, signed tags, reproducible build checks |

## Architecture Patterns

### CI/CD Pipeline Flow
```
Code Commit --> Build --> Test --> Security Scan --> Package
                                                       |
Monitor <-- Release Staging <-- Smoke Tests <-- Deploy Dev
                 |
         Manual Approval
                 |
         Deploy Production
```

### GitOps Architecture
```
App Repo --CI--> Config Repo --ArgoCD--> K8s Cluster
                     ^                        |
                     +----Continuous Sync-----+
```

## Reference Documentation

### CI/CD & Pipeline Design
- `references/best_practices.md` - Pipeline design patterns, testing strategies, deployment workflows
- `references/optimization.md` - Caching strategies, parallelization, build performance
- `references/troubleshooting.md` - Common issues, debugging, platform-specific problems
- `references/cicd-github-actions.md` - GitHub Actions workflows, runners, secrets

### Security & DevSecOps
- `references/security.md` - Secrets management, OIDC, supply chain security
- `references/devsecops.md` - SAST, DAST, SCA, container scanning guide
- `references/devsecops-basics.md` - Security best practices, shift-left security
- `references/security-hardening.md` - TLS enforcement, input validation, headers

### Infrastructure & Cloud Platforms
- `references/terraform-eks-module.tf` - Production EKS cluster Terraform
- `references/kubernetes-deployment.yaml` - Example manifest deployed by pipelines
- `references/kubernetes-basics.md` - Kubernetes concepts needed to understand deployment jobs
- `references/docker-basics.md` - Pipeline-adjacent container concepts; use `docker-containerization` for Dockerfile design
- `references/docker-compose.md` - Pipeline-adjacent compose references; use `docker-containerization` for local container environments
- `references/aws-overview.md` - AWS fundamentals, IAM, services
- `references/gcloud-platform.md` - GCP overview, gcloud CLI
- `references/cloudflare-workers-basics.md` - Edge computing, Workers

### Enterprise Readiness
- `references/general.md` - Universal enterprise readiness checks
- `references/github.md` - GitHub-specific enterprise requirements
- `references/openssf-badge-silver.md` - Silver badge criteria
- `references/openssf-badge-gold.md` - Gold badge criteria
- `references/signed-releases.md` - Artifact and tag signing
- `references/reproducible-builds.md` - Deterministic build patterns

### Release Artifact References
- `references/signed-releases.md` - Artifact and tag signing
- `references/reproducible-builds.md` - Deterministic build patterns

## Templates

### GitHub Actions
| Template | Description |
|----------|-------------|
| `templates/github-actions/node-ci.yml` | Complete Node.js CI/CD with security scanning |
| `templates/github-actions/python-ci.yml` | Python pipeline with pytest, coverage, PyPI |
| `templates/github-actions/go-ci.yml` | Go pipeline with multi-platform builds |
| `templates/github-actions/docker-build.yml` | Docker build with multi-platform, signing |
| `templates/github-actions/security-scan.yml` | Comprehensive DevSecOps pipeline |
| `templates/github-actions/dco-check.yml` | DCO sign-off enforcement |

### GitLab CI
| Template | Description |
|----------|-------------|
| `templates/gitlab-ci/node-ci.yml` | GitLab CI Node.js pipeline |
| `templates/gitlab-ci/python-ci.yml` | Python pipeline with parallel testing |
| `templates/gitlab-ci/go-ci.yml` | Go pipeline with Kubernetes deployment |
| `templates/gitlab-ci/docker-build.yml` | Docker build with DinD, multi-arch |
| `templates/gitlab-ci/security-scan.yml` | DevSecOps with GitLab security templates |

### Enterprise Templates
| Template | Description |
|----------|-------------|
| `templates/GOVERNANCE.md` | Project governance documentation |
| `templates/ARCHITECTURE.md` | Technical architecture template |
| `templates/CODE_OF_CONDUCT.md` | Contributor Covenant v2.1 |
| `templates/SECURITY_AUDIT.md` | Security self-audit template |

## Scripts

### Pipeline Analysis
| Script | Description |
|--------|-------------|
| `scripts/pipeline_analyzer.py` | Analyze workflows for optimization opportunities |
| `scripts/ci_health.py` | Check pipeline status and identify issues |
| `scripts/validate-devops-skill.sh` | Validate DevOps configurations |

### Security & Compliance
| Script | Description |
|--------|-------------|
| `scripts/verify-badge-criteria.sh` | OpenSSF Badge verification |
| `scripts/check-coverage-threshold.sh` | Statement coverage validation |
| `scripts/check-branch-coverage.sh` | Branch coverage analysis |
| `scripts/verify-signed-tags.sh` | Git tag signature verification |
| `scripts/verify-review-requirements.sh` | PR review requirements check |
| `scripts/check-tls-minimum.sh` | TLS 1.2+ enforcement check |
| `scripts/verify-spdx-headers.sh` | SPDX license header verification |
| `scripts/add-spdx-headers.sh` | Add SPDX headers to files |

### Infrastructure
| Script | Description |
|--------|-------------|
| `scripts/cloudflare_deploy.py` | Cloudflare Worker deployments |
| `scripts/docker_optimize.py` | Dockerfile analysis and optimization |

## Anti-Patterns

| Anti-Pattern | Symptom | Fix |
|-------------|---------|-----|
| YAML copy-paste proliferation | Identical workflows duplicated across repos | Reusable workflows, Helm charts, Kustomize bases, Terraform modules |
| Hardcoded secrets in code | API keys/passwords committed to git | Secret managers (Vault, AWS SM), sealed secrets, env vars from secure sources |
| No rollback strategy | No plan for deployment failure | Blue/green, canary with automated rollback, ArgoCD auto-revert |
| Monolithic CI pipeline | Single 45-minute pipeline on every commit | Parallel jobs, caching, incremental builds, path-based triggers |
| Running as root in containers | No USER instruction, privileged pods | Add USER instruction, set `securityContext.runAsNonRoot: true` |
| Using :latest tags | `FROM node:latest` in production | Pin specific versions, use immutable tags with SHA digests |
| Script injection vulnerability | `${{ github.event.* }}` directly in `run:` blocks | Use environment variables instead (see below) |
| Missing resource limits | Pods consume unbounded resources | Set requests and limits for CPU/memory in all deployments |
| Unpinned GitHub Actions | `uses: actions/checkout@v4` without SHA | Pin to commit SHA: `uses: actions/checkout@b4ffde6` |

**Script injection fix:**
```yaml
# DANGEROUS
- run: echo "Title: ${{ github.event.issue.title }}"

# SAFE
- name: Process issue
  env:
    TITLE: ${{ github.event.issue.title }}
  run: echo "Title: $TITLE"
```

## Quick Reference Commands

### GitHub Actions
```bash
gh workflow list                    # List workflows
gh run list --limit 20              # View recent runs
gh run view <run-id>                # View specific run
gh run rerun <run-id> --failed      # Re-run failed jobs
gh run view <run-id> --log > logs.txt  # Download logs
gh workflow run ci.yml              # Trigger workflow manually
```

### GitLab CI
```bash
gl project-pipelines list           # View pipelines
gl project-pipeline get <id>        # Pipeline status
gl project-pipeline retry <id>      # Retry failed jobs
gl project-pipeline cancel <id>     # Cancel pipeline
```

### Docker
```bash
docker build -t myapp .             # Build image
docker run -p 3000:3000 myapp       # Run container
docker compose up -d                # Start multi-container app
docker scout cves myapp             # Scan for vulnerabilities
```

### Kubernetes
```bash
kubectl apply -f deployment.yaml    # Apply manifest
kubectl get pods,services           # Check status
kubectl logs -f <pod>               # Stream logs
kubectl rollout status deployment/app  # Check rollout
```

### Terraform
```bash
terraform init                      # Initialize
terraform plan                      # Preview changes
terraform apply                     # Apply changes
terraform state list                # List resources
```

## Quality Checklist

```
[ ] All secrets in secret management (not in code)
[ ] Resource limits defined for all containers
[ ] Health checks configured (liveness, readiness)
[ ] Horizontal pod autoscaling enabled
[ ] Security contexts set (non-root, read-only)
[ ] Monitoring and alerting configured
[ ] Rollback strategy documented
[ ] Multi-environment support (dev, staging, prod)
[ ] Concurrency controls in CI pipelines
[ ] Remote state backend for Terraform
[ ] Vulnerability scanning in pipeline
[ ] Version pinning for all dependencies
[ ] Branch protection enabled
[ ] Code review required before merge
```

## Platform Selection Guide

| Need | Choose |
|------|--------|
| **Sub-50ms latency globally** | Cloudflare Workers |
| **Serverless functions (AWS)** | AWS Lambda |
| **Containerized workloads** | AWS ECS/Fargate, GKE, AKS |
| **Kubernetes at scale** | AWS EKS, Azure AKS, GCP GKE |
| **Object storage (zero egress)** | Cloudflare R2 |
| **Managed SQL** | AWS RDS, Azure SQL, Cloud SQL |
| **GitHub-integrated CI/CD** | GitHub Actions |
| **Self-hosted CI/CD** | GitLab CI, Jenkins |
| **Kubernetes GitOps** | ArgoCD, Flux |
| **Predictable workloads** | Reserved Instances, Savings Plans |
| **Fault-tolerant workloads** | Spot Instances, Preemptible VMs |

## Getting Started

1. **New pipeline**: Start with a template from `templates/`
2. **Add security scanning**: Use DevSecOps templates or add security stages
3. **Optimize existing**: Run `scripts/pipeline_analyzer.py`
4. **Debug issues**: Check `references/troubleshooting.md`
5. **Improve security**: Review `references/security.md` and `references/devsecops.md`
6. **Enterprise readiness**: Follow `references/general.md` checklist
7. **Release notes/versioning**: switch to `git-workflow` for changelog, semantic version, and commit convention workflows

## Source Skills

This curated skill combines content from the following legacy skills (now part of `cicd-pipelines`):
- Pipeline design, DevSecOps, optimization
- IaC, Kubernetes, deployment automation
- Cloud platforms, Docker, Cloudflare
- Multi-cloud, FinOps, comprehensive DevOps
- OpenSSF compliance, security assessment
- Artifact signing, provenance, and pipeline gates

## Resources

- **GitHub Actions**: https://docs.github.com/actions
- **GitLab CI**: https://docs.gitlab.com/ee/ci/
- **Terraform**: https://developer.hashicorp.com/terraform
- **Kubernetes**: https://kubernetes.io/docs
- **ArgoCD**: https://argo-cd.readthedocs.io
- **OpenSSF Scorecard**: https://securityscorecards.dev/
- **SLSA Framework**: https://slsa.dev/