Your CI/CD Security Wiring Diagram: SAST, SCA, IaC, SBOM, and Signatures Without Killing Throughput

The problem isn’t tools; it’s wiring and incentives

If you’ve ever dropped a scanner into CI and watched lead time double, you know the movie. Two sprints later, the team quietly flips it to “warn.” I’ve seen this at a fintech on GitLab, a B2B SaaS on Actions, and a Fortune 500 on Jenkins. Same smell: security bolted on, not integrated.

What actually works is a clear wiring diagram, hard gates only where they’re cheap and credible, and metrics that keep leadership honest. Below is the wiring I deploy at GitPlumbers when we’re asked to “add security without killing velocity.”

• Fast on PR: secrets, SAST, IaC (under 90s total)
• Heavier on merge to main: SCA, container scan, SBOM, sign
• Gate at deploy: policy + signature verification
• Post-deploy/nightly: DAST, repo-wide baseline jobs

Keep PRs fast, put heavy lifting after merge, and enforce at deploy like you enforce tests.

The reference flow: where each scan belongs

Think checkpoints, not a monolith.

1. Commit/PR

   • Secrets scanning: gitleaks or native SCM (GitHub secret scanning) on diff
   • SAST: semgrep or codeql with incremental scope
   • IaC: checkov/tfsec on changed *.tf, k8s/*.yaml, Helm charts
   • Gate: fail on new Critical/High; warn on Medium until baseline is clean

2. Build and containerize (on merge)

   • SCA: lockfile scan (e.g., trivy fs or Snyk) plus docker image layers
   • Container scan: trivy image or grype on the built image
   • SBOM: syft to CycloneDX or SPDX; attach to artifact
   • Sign: cosign sign with keyless or KMS; push provenance (SLSA provenance attestation)

3. Pre-deploy

   • Policy check: conftest (OPA) against SBOM and k8s manifests (e.g., block :latest, CVSS >= 9)
   • Signature verification: cosign verify in the CD controller (e.g., Kyverno, Policy Controller)

4. Post-deploy / nightly

   • DAST: OWASP ZAP baseline on staging or review env
   • Repo-wide baseline: full SAST/SCA to ratchet debt without blocking PRs

This gets you credible protection where it matters, and speed where devs live.

A concrete GitHub Actions pipeline (fast PRs, heavier merges)

Below is a minimal but real Actions setup. Same logic ports to GitLab or Jenkins.

ame: ci-security
on:
  pull_request:
    branches: ["*"]
    paths-ignore: ["**/*.md", "docs/**"]
  push:
    branches: ["main"]

jobs:
  pr-fast-scans:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    permissions: { security-events: write, contents: read }
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }

      - name: Cache tools
        uses: actions/cache@v4
        with:
          path: ~/.cache/semgrep
          key: ${{ runner.os }}-semgrep-${{ hashFiles('**/*.yml') }}

      - name: Secret scan (gitleaks)
        uses: zricethezav/gitleaks-action@v2
        with:
          args: "detect --source . --no-git --redact --baseline-path .gitleaks.baseline.json --exit-code 1"

      - name: SAST (Semgrep incremental)
        uses: returntocorp/semgrep-action@v1
        with:
          config: p/owasp-top-ten
          generateSarif: true
          auditOn: pull_request
          baselineRef: origin/main

      - name: IaC scan (Checkov)
        uses: bridgecrewio/checkov-action@v12
        with:
          directory: .
          soft_fail: false
          framework: terraform,kubernetes,helm

  build-and-scan:
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    permissions: { contents: read, packages: write, id-token: write }
    steps:
      - uses: actions/checkout@v4
      - name: Build image
        run: |
          docker build -t ghcr.io/${{ github.repository }}:${{ github.sha }} .
          echo "IMAGE=ghcr.io/${{ github.repository }}:${{ github.sha }}" >> $GITHUB_ENV

      - name: SBOM (Syft CycloneDX)
        uses: anchore/sbom-action@v0
        with:
          image: ${{ env.IMAGE }}
          format: cyclonedx-json
          artifact-name: sbom.cdx.json

      - name: Container scan (Trivy)
        uses: aquasecurity/trivy-action@0.20.0
        with:
          image-ref: ${{ env.IMAGE }}
          format: table
          severity: CRITICAL,HIGH
          ignore-unfixed: true
          exit-code: 1

      - name: Produce provenance (SLSA)
        uses: slsa-framework/slsa-github-generator@v2
        with:
          artifact_path: .

      - name: Sign image (Cosign keyless)
        env:
          COSIGN_EXPERIMENTAL: "true"
        run: |
          cosign sign ${{ env.IMAGE }}

      - name: Push image
        run: docker push ${{ env.IMAGE }}

  policy-gate:
    needs: build-and-scan
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Policy (Conftest on k8s manifests)
        run: |
          conftest test deploy/k8s --policy policy/ --input yaml

Add verification in your CD path. For Argo CD or Flux, enforce signature verification and basic policies.

# kyverno ClusterPolicy example
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-signed-images
spec:
  validationFailureAction: Enforce
  rules:
    - name: require-sigstore
      match:
        any:
          - resources:
              kinds: [Deployment]
      verifyImages:
        - imageReferences: ["ghcr.io/your-org/*"]
          attestors:
            - entries:
                - keyless:
                    issuer: https://token.actions.githubusercontent.com
                    subject: "repo:your-org/your-repo:ref:refs/heads/main"

GitLab and Jenkins snippets (because not everyone’s on Actions)

GitLab CI: SAST/Secrets/IaC on PR, heavy on default branch.

stages: ["test","build","security","deploy"]

variables:
  TRIVY_EXIT_CODE: 1

pr_scans:
  stage: test
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  script:
    - gitleaks detect --no-git --exit-code 1
    - semgrep --config p/owasp-top-ten --error --baseline-ref origin/main
    - checkov -d . --framework terraform,kubernetes,helm

container_scan:
  stage: security
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - trivy image --severity CRITICAL,HIGH --ignore-unfixed --exit-code $TRIVY_EXIT_CODE $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - syft packages -o cyclonedx-json $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA > sbom.cdx.json
    - cosign sign $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  artifacts:
    paths: [sbom.cdx.json]

Jenkinsfile: keep it parallel and cache aggressively.

pipeline {
  agent any
  stages {
    stage('PR Scans') {
      when { changeRequest() }
      parallel {
        stage('Secrets') { steps { sh 'gitleaks detect --no-git --exit-code 1' } }
        stage('SAST') { steps { sh 'semgrep --config p/owasp-top-ten --error --baseline-ref origin/main' } }
        stage('IaC') { steps { sh 'checkov -d . --framework terraform,kubernetes,helm' } }
      }
    }
    stage('Build+Scan') {
      when { branch 'main' }
      steps {
        sh 'docker build -t $IMAGE .'
        sh 'trivy image --severity CRITICAL,HIGH --ignore-unfixed --exit-code 1 $IMAGE'
        sh 'syft $IMAGE -o cyclonedx-json > sbom.cdx.json'
        sh 'cosign sign $IMAGE'
      }
    }
  }
}

Gates, thresholds, and waivers that won’t become permanent exceptions

You need teeth, but not fangs that bite the wrong people.

• Severity gates:
  • PRs: fail on new Critical (immediately), warn on High for first 30 days; then ratchet to fail on High.
  • Main: fail build on Critical,High; allow Medium with SLA and auto-ticket.
• Time-bound waivers:
  • Store waivers in repo as code with expiry and owner. Example conftest input:

package waivers

default allow = false

allow {
  input.finding.id == "CVE-2023-12345"
  input.repo == "acme/service-a"
  time.now_ns() < time.add(time.now_ns(), 14*24*60*60*1000000000) # 14 days
  input.owner == "team-payments"
}

• Break-glass:
  • Allow merge with label security-break-glass only if an incident number is linked, and send an audit event to Slack/SEIM.
• Pinning and image hygiene:
  • Pin base images (python:3.11.7-slim), block :latest in policy, and rebuild weekly for patched layers.

This keeps pressure where it belongs: on fixing real risk, not arguing with the scanner.

Keep it fast and sane: tuning for signal, not noise

• Incremental scanning:
  • semgrep --baseline-ref origin/main and codeql database analyze --sarif --threads 4 only on changed files.
• Caching and parallelization:
  • Cache rule downloads and language packs; parallelize secrets/SAST/IaC on PRs. Target < 90s PR budget.
• De-duplication and SARIF:
  • Emit SARIF for SAST/SCA and centralize in the code hosting security tab. Merge duplicates across tools.
• Vulnerability sources:
  • For containers, use distro feeds (e.g., trivy --ignore-unfixed to avoid churn) and prefer slim images to cut CVE surface.
• Baselines for legacy code:
  • Establish a baseline report today; only fail on “new” findings. Tackle old debt via backlog with owners and SLAs.
• False positive management:
  • Use semgrep autofix and --disable-nosem patterns sparingly; every suppression requires an issue link and expiry.
• IaC context:
  • Enrich scans with cloud context (e.g., checkov --skip-check CKV_AWS_79 only if an org-wide guardrail exists via SCP).

Metrics that prove it’s working (and when it isn’t)

Executives love big numbers; engineers love truthful ones. Track both.

• Lead time impact: median PR scan duration, target < 90s; build+scan on main < 8 min.
• Defect flow:
  • Findings per KLOC (new only), trend down week over week.
  • Remediation MTTR: Critical < 3 days, High < 14 days, Medium < 45 days.
• Gate effectiveness: percent of merges blocked for valid reasons vs reverted waivers.
• Coverage: percent of repos with SBOM + signed artifacts; percent of deploys with signature verified.
• Noise: false positive rate (< 10%) measured via suppression with evidence links.

Quick and dirty export for GitHub to a dashboard:

# Count open critical vulns from code scanning
gh api -X GET repos/:owner/:repo/code-scanning/alerts --paginate \
  | jq '[.[] | select(.rule.severity=="critical" and .state=="open")] | length'

# Average PR security job duration
gh run list --event pull_request --json duration,status | jq '[.[] | select(.status=="completed") | .duration] | add/length'

Wire these into Prometheus/Grafana, or push to Datadog as custom metrics. Set SLOs and page yourselves when MTTR breaches.

A 30/60/90-day rollout that survives contact with reality

Day 0–30: visibility, no surprises

• Turn on PR scans (secrets, SAST, IaC) in warn-only except Critical. Establish baselines. Cache and parallelize.
• Nightly SCA and container scans on main. Publish SBOMs. No gates yet except on Criticals.
• Metrics: capture durations, findings, and MTTR starting today.

Day 31–60: add teeth

• Fail on High in PRs. Enforce container scan gates on main. Introduce signature verification in staging.
• Implement waivers with expiry and owner. Start policy-as-code for :latest and privilege blocks.
• Metrics: show MTTR improvements; track gate-induced failures vs. false positives.

Day 61–90: production-grade

• Enforce signature verification in prod. Add DAST in pre-prod nightly. Ratchet to fail on Medium in sensitive services.
• Integrate SBOM diffing in CD; block if new Criticals appear between build and deploy.
• Metrics: coverage > 90% repos, PR scan p95 < 120s, Critical MTTR < 3 days.

If any metric blows up, roll back severity gates one notch and fix tooling noise before re-enforcing.

Gotchas I keep seeing (so you can dodge them)

• Monorepos: path filters are mandatory or your PR scans will be 10 minutes. Use CODEOWNERS to scope ownership and waivers.
• Private registries/air-gapped: pre-warm scanners and CVE DBs in the runner; mirror feeds; consider Grype+Syft offline modes.
• Third-party services: your CD nodes need egress to OIDC issuers for keyless signing; otherwise use KMS-backed keys.
• Alpine-based images: busybox/glibc differences can produce noisy CVEs; consider -slim Debian or Wolfi for cleaner feeds.
• Java builds: mvn -DskipTests still downloads half the internet; cache local m2 or use GitHub Actions dependency caching.
• Helm and Kustomize: template before scanning IaC or you’ll miss runtime values. helm template and feed the result to conftest.
• License scanning: legal will ask. Add SPDX/CycloneDX licenses in SBOM and auto-fail on copyleft in closed-source services.

The goal is not zero findings. The goal is zero surprise risk at deploy time, with predictable, fast feedback for developers.