Security Ci Security Testing

# CI Security Testing

> Run SAST, DAST, SCA, and secrets scanning on every commit -- automated security gates that
> catch vulnerabilities before they reach production

## When to Use

- Setting up a new CI/CD pipeline and need to integrate security scanning from the start
- Adding automated security testing to an existing pipeline that currently has none
- Evaluating SAST, DAST, and SCA tools for your technology stack
- Reducing the time between vulnerability introduction and detection
- Meeting compliance requirements for automated security testing (SOC2, PCI-DSS, ISO 27001)
- Managing false positives from security scanners without disabling them entirely

## Threat Context

Automated security testing in CI catches vulnerabilities at the earliest possible point after
code is written. Without CI security testing, vulnerabilities accumulate silently until a
periodic security review or penetration test discovers them -- by which time hundreds of
vulnerable commits have been deployed, the original developers have moved to other features,
and the fix requires understanding code written weeks or months ago.

The 2021 Codecov supply chain attack exploited a vulnerability in Codecov's Bash Uploader
script to exfiltrate environment variables (including CI secrets) from thousands of CI/CD
pipelines. The attack went undetected for two months because most organizations did not scan
their CI pipeline components for tampering. Pipeline security -- securing the CI/CD system
itself, not just the code it builds -- is as critical as the application security testing the
pipeline performs.

The 2023 CircleCI breach exposed customer secrets stored in CircleCI's environment. The 2024
xz-utils backdoor (CVE-2024-3094) was injected through a compromised maintainer who modified
build scripts. These incidents demonstrate that CI/CD pipelines are high-value targets: they
have access to production credentials, signing keys, and deployment permissions.

## Instructions

1. **Implement Static Application Security Testing (SAST) on every pull request.** SAST
   analyzes source code for vulnerability patterns without executing it. Tool options: CodeQL
   (GitHub, semantic analysis, supports C/C++, Java, JavaScript, Python, Go, Ruby, C#),
   Semgrep (pattern-based, fast, supports 30+ languages, community rules plus custom rules),
   SonarQube (broad language support, code quality plus security). Run SAST on pull requests,
   not just the main branch, so developers see findings before merging. Configure findings as
   PR comments or check annotations, not just dashboard entries.

2. **Implement Software Composition Analysis (SCA) for dependency vulnerabilities.** SCA
   identifies known vulnerabilities in third-party dependencies. Tools: Dependabot (GitHub
   native, auto-creates PRs for vulnerable dependencies), Snyk (broad ecosystem support,
   license compliance), Trivy (container and filesystem scanning, open-source), Grype (Anchore,
   SBOM-based vulnerability matching). Run SCA on every PR and on a daily schedule against the
   main branch (new CVEs are published daily against existing dependencies). Block merges for
   critical and high severity vulnerabilities with known exploits.

3. **Implement secrets scanning to prevent credential leaks.** Secrets scanning detects API
   keys, passwords, tokens, and private keys in source code. Tools: gitleaks (fast, regex-based,
   open-source), truffleHog (entropy-based plus regex, scans git history), GitGuardian
   (commercial, real-time monitoring, broad secret type coverage). Run secrets scanning as a
   pre-commit hook (catches secrets before they enter git history) and in CI (catches secrets
   that bypass the pre-commit hook). Scan the full git history periodically -- a secret committed
   and then deleted is still in the git history and still compromised.

4. **Implement Dynamic Application Security Testing (DAST) against staging environments.** DAST
   tests the running application by sending malicious requests and observing responses. Tools:
   OWASP ZAP (open-source, scriptable, API and web scanning), Burp Suite (commercial, more
   comprehensive, better for complex applications), Nuclei (template-based, fast, good for
   known vulnerability checks). Run DAST against a staging environment deployed from the PR
   branch. Configure authenticated scanning so the scanner tests authenticated endpoints, not
   just the login page. DAST finds runtime vulnerabilities that SAST misses: misconfigured
   headers, exposed error pages, authentication bypasses.

5. **Harden the CI/CD pipeline itself.** The pipeline has access to production secrets,
   deployment credentials, and signing keys -- it is a high-value target. Runner isolation: use
   ephemeral runners that are destroyed after each job (GitHub Actions hosted runners, ephemeral
   self-hosted runners). Secret masking: ensure CI secrets are never printed in logs. Pin
   dependencies: pin GitHub Actions and other CI plugins by SHA, not by tag (a tag can be
   reassigned to point to a malicious commit). Least privilege: CI service accounts should have
   the minimum permissions needed for their specific job, not admin access to the entire
   infrastructure.

6. **Establish a triage and suppression workflow for findings.** Every security scanner produces
   false positives. Without a triage workflow, developers either ignore all findings (alert
   fatigue) or waste time on non-issues. Implement a triage process: new findings are reviewed
   within a defined SLA (24 hours for critical, 1 week for medium). Confirmed findings become
   backlog items with severity-based SLAs. False positives are suppressed with a documented
   justification (code comment, scanner configuration, or triage tool entry). Track the
   false-positive rate per scanner and per rule to identify rules that need tuning or removal.

7. **Measure and report on CI security metrics.** Track: mean time to detect (from commit to
   finding), mean time to remediate (from finding to fix), false-positive rate by scanner and
   rule, findings by severity and category, percentage of PRs with security findings. Report
   these metrics monthly to engineering leadership. The goal is not zero findings -- it is a
   decreasing trend in time-to-detect and time-to-remediate, with a manageable false-positive
   rate that keeps developers engaged rather than fatigued.

## Details

- **SAST: pattern-based vs semantic analysis**: Pattern-based tools (Semgrep) match code against
  known vulnerability patterns using AST matching or regex. They are fast, easy to write custom
  rules for, and have low setup cost. Semantic analysis tools (CodeQL) build a queryable database
  of the code's data flow and control flow, enabling taint tracking (tracing untrusted input
  from source to sink). CodeQL can find vulnerabilities that require understanding data flow
  across multiple functions and files, which pattern matching cannot. Use both: Semgrep for
  fast, broad coverage with custom rules; CodeQL for deep taint analysis on critical code paths.

- **False positive management strategies**: Accept that false positives are inherent in static
  analysis. Strategies to manage them: configure scanner sensitivity (start strict, loosen rules
  with high FP rates), use inline suppression annotations with mandatory justification comments,
  create a central suppressions file reviewed in security champion meetings, track FP rates per
  rule and disable rules with >50% FP rate after investigation, deduplicate findings across
  scanners to avoid reviewing the same issue twice.

- **Pipeline-as-code security**: If the CI configuration (GitHub Actions YAML, Jenkinsfile,
  GitLab CI YAML) is stored in the repository, any developer with commit access can modify the
  pipeline to skip security checks. Mitigations: use CODEOWNERS to require security team
  approval for CI configuration changes, use reusable workflows (GitHub) or templates (GitLab)
  that projects cannot modify, use branch protection rules that require specific checks to pass
  before merging.

- **Scanning schedule strategy**: PR-time scanning (SAST, secrets, SCA): fast feedback, catches
  new introductions. Daily scheduled scanning (SCA against main): catches newly disclosed CVEs
  in existing dependencies. Weekly full DAST scans: comprehensive runtime testing. Monthly
  historical secrets scan: catches secrets committed and deleted in past history. Each cadence
  serves a different detection objective.

## Anti-Patterns

1. **Alert fatigue from untuned scanners.** Deploying a scanner with default rules and no tuning
   generates hundreds of findings, most of which are false positives or low-severity issues.
   Developers learn to ignore the scanner. Start with a small set of high-confidence rules,
   verify that findings are actionable, and gradually expand coverage. A scanner that produces
   10 accurate findings is more valuable than one that produces 500 findings of which 450 are
   false positives.

2. **Security scanning as a separate pipeline stage.** Running security scans in a separate,
   optional pipeline stage that developers can skip or ignore. Security scanning should be
   integrated into the required CI checks -- the same checks that must pass before a PR can be
   merged. If security scanning is optional, it will be skipped when it is inconvenient, which
   is exactly when it is most needed.

3. **Scanning only the main branch instead of pull requests.** Scanning after merge means the
   vulnerability is already in the production codebase when it is detected. The developer who
   introduced it may have moved to another task. Scan pull requests so that findings appear in
   the context of the change that introduced them, when the developer can fix them immediately.

4. **No triage process for findings.** Scanner output dumped into a dashboard that nobody
   reviews. Findings accumulate, the dashboard becomes a graveyard of unresolved issues, and
   the scanner provides no security value. Every finding must have an owner, a severity-based
   SLA, and a tracked resolution (fixed, suppressed with justification, or accepted risk).

5. **Treating scanner output as a compliance checkbox.** Running scanners to satisfy an audit
   requirement without acting on the findings. The audit asks "do you run SAST?" and the answer
   is "yes" -- but the 200 open critical findings tell a different story. Scanners provide value
   only when their findings drive remediation. Track and report the percentage of findings
   remediated within SLA, not just whether the scanner ran.

6. **Hardcoding suppression of findings without review.** Developers adding blanket suppression
   annotations to silence scanner warnings without understanding the finding. Require that all
   suppressions include a justification comment and are reviewed by a security champion or the
   security team. Periodically audit suppressed findings to verify the justification is still
   valid.