Beyond the Green Checkmark: Security IN and OF Your CI Pipeline - Derek Fisher

Beyond the Green Checkmark: Security IN and OF Your CI Pipeline

Speakers: Derek Fisher, Professor at Temple University, Founder of Securely Built

Conference: Disobey

YouTube: https://www.youtube.com/watch?v=zIbj9nf670k

Overview

In his Disobey talk, "Beyond the Green Checkmark: Security IN and OF Your CI Pipeline," Derek Fisher dissects the critical, yet often misunderstood, role of security within the Continuous Integration/Continuous Delivery (CI/CD) pipeline. Fisher, a professor at Temple University and founder of Securely Built, argues that many organizations fall into the trap of "security theater" – implementing numerous scanning tools that generate a flood of low-value alerts without truly enhancing their security posture. The core message is a dual imperative: securing the code itself (security in the pipeline) and securing the pipeline infrastructure (security of the pipeline) against sophisticated attacks.

Fisher highlights the pervasive problem of alert fatigue and "normalized deviance" among developers, where a constant barrage of false positives and unactionable findings leads to warnings being ignored. This talk provides a comprehensive roadmap for moving beyond performative security measures to establish robust, actionable security controls. It emphasizes the need for fine-tuned scanning, policy-as-code, ephemeral build environments, and stringent dependency management to combat modern supply chain attacks and ensure that the "green checkmark" in the CI/CD pipeline truly signifies a secure system.

The presentation is particularly relevant for security professionals, DevOps engineers, and developers seeking to integrate effective security practices throughout the software development lifecycle. Fisher's insights are grounded in his extensive experience in engineering and cybersecurity, offering practical strategies to transform CI/CD pipelines from potential attack vectors into resilient security enforcers. By addressing both the technical and cultural aspects of security integration, the talk aims to empower teams to build confidence in their software's security from development to deployment.

Background

▶ Watch: Speaker introduction and professional background (0:00)

The evolution of software development has seen the widespread adoption of DevOps principles, accelerating delivery cycles through automation. Integral to this acceleration is the CI/CD pipeline, which automates the build, test, and deployment phases. Recognizing the need to embed security earlier in this process, the concept of DevSecOps emerged, advocating for the integration of security tools like Static Application Security Testing (SAST), Software Composition Analysis (SCA), and Infrastructure as Code (IaC) scanning directly into the pipeline workflow. These tools, often implemented as GitHub Actions or Jenkins steps, promise to identify vulnerabilities early.

However, Fisher argues that this approach, while well-intentioned, frequently devolves into "security theater." Drawing from his 30 years of engineering experience, he recalls the early days of software security where static analysis was notoriously noisy, generating thousands of vulnerabilities from a single scan. This problem persists today, leading to an overwhelming volume of low-value data and false positives. The consequence is alert fatigue, where developers and security teams become desensitized to warnings, ultimately leading to normalized deviance – a state where ignoring security alerts becomes the norm. This is exacerbated when security checks run in the pipeline but do not actually fail the build, creating a false sense of security and allowing vulnerable code to proceed.

The core issue lies in the default configurations of many security tools, which often produce less than 10% true positives. Developers, lacking context or clear guidance on how to remediate complex findings (e.g., "use parameterized queries" for SQL injection), often ignore them. Furthermore, if security gates can be bypassed or failed checks do not block code merges, the entire security enforcement mechanism is rendered ineffective. The talk seeks to address this foundational problem by advocating for a shift from passive, performative security to an active, confidence-building approach that secures both the application code and the underlying CI/CD infrastructure.

Key Findings

▶ Watch: Introduction to ideal DevSecOps pipeline and talk scope (2:50)

Derek Fisher's talk presents several key findings and principles for achieving genuine security in and of the CI/CD pipeline:

1. Differential Scanning for Pull Requests: One of the most effective methods to combat alert fatigue is implementing diff-aware scanning on pull requests (PRs). This technique analyzes only the changes between baselines, reporting on new vulnerabilities introduced by the PR rather than rescanning the entire codebase. This significantly reduces noise, provides immediate, actionable feedback to developers, and prevents "pipeline bloat" from scanning legacy technical debt.
2. Optimized Tool Configuration: Moving beyond default settings for security scanning tools (especially SAST) is crucial. Fine-tuned rule sets focused on high-confidence, high-severity findings for PR blocking, combined with broader rules for scheduled full scans, drastically improves the signal-to-noise ratio. Filtering out code quality issues from security vulnerabilities further refines results.
3. Policy as Code for Infrastructure Enforcement: Manual oversight of complex infrastructure configurations is prone to error and bottlenecks. Policy as Code, leveraging languages like Rego and tools like Open Policy Agent (OPA), Checkov, or Trivy, allows organizations to codify security standards into executable rules. These policies are version-controlled, testable, and provide immediate feedback to developers, preventing insecure infrastructure from being deployed.
4. Centralized Pipeline Logic and Two-Person Rule: To prevent pipeline poisoning and unauthorized modifications, centralizing reusable workflow logic in dedicated, managed repositories is essential. Furthermore, implementing a two-person rule for CI/CD configuration changes ensures that no single compromised credential or malicious insider can unilaterally alter the pipeline, adding a critical layer of defense against supply chain attacks.
5. Ephemeral and Isolated Build Environments: The "pet problem" of long-lived, mutable build servers is a significant security risk, as a compromised server can infect subsequent builds or exfiltrate credentials. The solution lies in using ephemeral runners or disposable build environments that are provisioned for a single job and immediately destroyed afterward, ensuring a pristine, known-good state for every build.
6. Artifact Integrity with SALSA and Pinned Dependencies: Ensuring that the deployed artifact is precisely the one that was built is paramount. The SALSA framework (Supply-chain Levels for Software Artifacts) provides a maturity model for achieving this through authenticated metadata and hardened build environments. Complementing this, pinning dependencies to immutable commit SHAs (rather than mutable version tags) and using SCA tools prevents dependency confusion, typo squatting, and malicious updates from being unknowingly incorporated.
7. Short-Lived Credentials via OpenID Connect (OIDC): Static, long-lived API keys and cloud credentials stored in repository secrets represent a persistent risk. Adopting OIDC allows for the use of short-lived, job-specific tokens, significantly reducing the exploitation window if credentials are compromised.
8. Human Judgment Remains Essential: Despite advancements in automation and AI, human code reviews and expert triage are indispensable for identifying subtle logic errors, complex business logic flaws, or intentional backdoors that automated tools frequently miss. Human oversight provides context, intent, and strategic triage, validating automated findings and preventing "security theater."

These findings collectively advocate for a proactive, intelligent, and layered approach to CI/CD security, moving beyond simplistic tool deployment to a comprehensive strategy that integrates security deeply into both the development process and the infrastructure managing it.

Technical Deep Dive

▶ Watch: Identifying the problem: security theater and alert fatigue (3:19)

Fisher's talk meticulously breaks down the technical strategies for implementing security in and of the CI pipeline.

Security IN the Pipeline: Securing the Code

The first half of the discussion focuses on ensuring the application code itself is secure as it moves through the CI/CD process.

Differential Scanning and Optimized Tooling

A cornerstone of reducing developer alert fatigue is differential scanning for pull requests (PRs). Instead of scanning the entire codebase, which often contains years of technical debt, diff-aware scanning compares the proposed changes against a baseline, identifying only new vulnerabilities introduced by the PR. This provides immediate, relevant feedback to the developer, preventing pipeline bloat and maintaining development velocity. Tools like GitHub Actions can integrate this seamlessly. Fisher emphasizes that these diff scans should be configured with fine-tuned rule sets focusing on high-confidence, high-severity findings to minimize false positives. Broader, more comprehensive scans can be reserved for nightly or weekly full codebase analyses, tracking overall security debt separately.

Optimizing scanning tools goes beyond default settings. Fisher highlights that default SAST configurations might yield less than 10% true positives. To counter this, organizations should select specific, high-confidence rules for PR blocking, potentially using specific SANS Top 25 or OWASP Top 10 related rules, rather than hundreds of generic rules. It's also critical to distinguish between code quality issues (e.g., linting, stylistic errors) and genuine security vulnerabilities, filtering the former to keep developers focused. Tools can also be configured to skip irrelevant files like minified JavaScript, machine-generated code, or large, complex files using ignore configuration files (e.g., .sundupignore, .git-leaks-ignore), preventing scan timeouts and irrelevant findings.

Result Management and Prioritization

The output from various security tools (SAST, DAST, SCA, secret scanning) often comes in formats like JSON or SARIF. These results should be pushed to a single source of truth for security posture, such as DefectDojo, GitHub Advanced Security, or custom dashboards. Crucially, these systems need to maintain bidirectional synchronization with scanning tools, ensuring that changes or remediations are reflected across all platforms, preventing vulnerabilities from being missed.

For prioritization, Fisher points out the flaws of relying solely on CVSS (Common Vulnerability Scoring System). While CVSS provides a base score, a critical CVSS vulnerability might not be critical in a specific application context. He advocates for using reachability analysis tools like EPSS (Exploit Prediction Scoring System), which helps determine if a vulnerability is actively being exploited in the wild or is actually exploitable within the application's environment. This allows teams to focus on exploitable paths rather than theoretical noise. Furthermore, automated triage alerts can be codified into scanning tool configurations, preventing recurring false positives or non-applicable issues from re-appearing, thus improving the relationship between security and development teams.

Application Security Posture Management (ASPM) and Human Oversight

Application Security Posture Management (ASPM) aims to provide a centralized framework correlating data from diverse scanning tools (SAST, DAST, SCA, IAST) into a single platform. This aggregated, multi-tool dataset allows for more accurate vulnerability scoring and runtime validation. For instance, a SAST finding flagged as theoretical can be validated with DAST (simulating attack behavior) or IAST (monitoring internal app behavior during execution) to confirm actual exploitability. This holistic view, married with bug tracking, enables strategic prioritization based on real exploitability, not just CVSS scores.

Despite automation, Fisher stresses that human oversight remains indispensable. Automated tools are effective at pattern recognition but can miss complex logic errors, business logic flaws, or even intentional backdoors (e.g., data exfiltration via email). Professional code reviews and manual audits catch what tools miss, ensuring that security measures are effective rather than merely performative. Expert human triage is essential to distinguish true positives from misidentifications flagged by tools.

Policy as Code

To address the complexity of modern infrastructure, especially with Infrastructure as Code (IaC), Fisher introduces Policy as Code. This approach codifies security standards into executable rules written in a query language like Rego, used by tools such as Open Policy Agent (OPA), Checkov, and Trivy. For example, a Rego policy could prevent the deployment of an IaC definition (e.g., Terraform, Kubernetes manifest) that opens SSH access to the world. The key advantages are that policies are version-controlled, testable, and provide consistent enforcement across all environments, offering immediate feedback to developers within the PR workflow and catching misconfigurations before deployment, drastically reducing remediation costs.

Security OF the Pipeline: Securing the Infrastructure

The second half of the talk shifts focus to the CI/CD pipeline itself as an attack surface, emphasizing that securing the code is insufficient if the pipeline building it is vulnerable.

Pipeline Poisoning Execution (PPE)

The pipeline offers a distinct attack surface, often overlooked. Attackers target pipelines because they are an effective way to compromise multiple applications, facilitating supply chain attacks. Fisher details Pipeline Poisoning Execution (PPE), where attackers exploit SCM (Source Code Management) permissions to inject malicious commands into the build process.

• Direct PPE: An attacker with write access directly modifies CI configuration files (e.g., GitHub Actions YAML) and pushes them to an unprotected branch.
• Indirect PPE: More subtle, the attacker poisons a file the build depends on (e.g., a Makefile, shell script, or test file) which is then executed as part of a legitimate pipeline step.
• Public PPE (Pawn Requests): Exploits pipelines that automatically run on pull requests from public forks. An external contributor submits a PR that modifies a setup script or injects malicious code, which then executes when the workflow runs to test the PR.

The attack flow involves: attacker gaining SCM permissions (compromised contributor, malicious insider) → modifying CI config/scripts → pushing poisoned code → CI system pulling and executing it with privileged identity → exfiltrating credentials (repository secrets, API keys, cloud credentials) or pivoting to internal resources.

Defenses include centralized pipeline logic with reusable workflows managed by DevOps or security teams in dedicated repositories. A two-person rule for CI config modifications ensures a single compromised credential cannot expose the entire supply chain.

Dependency Chain Abuse

Trusting external dependencies is a major vulnerability. Fisher outlines several attack vectors:

• Dependency Confusion: Build systems are tricked into substituting internal packages with malicious public ones of the same name (e.g., @company/authlib internal vs. public).
• Typo Squatting: Attackers register packages with misspelled or similar names (e.g., requests vs. requessts).
• Maintainer Compromise: Stolen credentials allow attackers to inject backdoors into trusted, popular libraries.
• Latest Trap: Relying on mutable tags (e.g., latest) allows bad actors to push new commits to existing tags, which are blindly pulled into builds without review.

To counter this, organizations must pin dependencies to immutable references, such as full commit SHAs or hashes, instead of mutable version tags. SCA tools are vital for identifying and vetting open-source components with known vulnerabilities. Using scoped packages or private repositories for internal dependencies also reduces exposure to public registries. The principle here is to move from implicit trust to verifying everything.

The Pet Problem and Over-Privilege

Traditional CI/CD often uses long-lived, mutable build servers (the "pet problem"). If such a server is compromised, it becomes a persistent backdoor, infecting subsequent builds, tampering with artifacts, or harvesting credentials from future jobs. The solution is ephemeral runners or disposable build environments, which are spun up for a single job and immediately destroyed, returning to a known good state after every interaction. GitHub Actions, for instance, natively uses ephemeral runners.

Furthermore, build runners are often over-privileged, having unrestricted access to the public internet (for dependencies) and internal production databases (for integration tests). A compromised runner can then exfiltrate credentials or pivot deeper into the network. A zero-trust approach is critical, involving network segmentation (isolating build infrastructure from production), traffic filtering (prohibiting unauthorized ingress/egress), and allow list policies that only permit connections to approved registries and services.

Artifact Integrity and Secrets Management

Ensuring artifact integrity means verifying that the artifact being deployed is precisely the one that was built. The SALSA framework provides a graduated maturity model (four levels, with the first three being most critical) to achieve this:

• Level 1 (Provenance): Scripted build processes generate metadata.
• Level 2 (Build Service): Authenticated metadata identifies the build system, source code, and output artifact, preventing injection of malicious binaries.
• Level 3 (Hardened Build): Builds occur in ephemeral, isolated environments, preventing malware persistence.

Tackling these levels effectively stops supply chain attacks by ensuring artifacts either lack a valid signature or have provenance data that doesn't match the expected source.

Finally, secrets management is paramount. Accidentally committing credentials to Git history is a common and dangerous mistake, as tools like truffleHog and git-leaks can mine entire Git histories for high-entropy strings and secret patterns (AWS keys, API tokens, etc.). Deleted secrets remain in history unless rotated. Best practices include pre-commit hooks to block secrets from entering the repository, layered scanning (PR checks, branch protection, periodic full history scans), and immediate rotation of any detected credentials. Replacing long-lived static keys with OpenID Connect (OIDC) for cloud access provides short-lived, job-specific tokens, significantly reducing the exploitation window. Storing secrets in secure vaults (HashiCorp Vault, AWS Secrets Manager) and favoring environment secrets over repository secrets are also crucial.

Demo / Proof of Concept

▶ Watch: Problems: ignore culture, normalized deviance, and bypass risk (6:06)

The talk "Beyond the Green Checkmark: Security IN and OF Your CI Pipeline" by Derek Fisher is a conceptual and strategic presentation, focusing on architectural patterns, best practices, and attack vectors rather than demonstrating specific tools or proof-of-concept exploits. Therefore, no live demo or detailed proof-of-concept was presented as part of the talk.

Defensive Implications

▶ Watch: First solution: differential scanning for pull requests (7:42)

The insights from Derek Fisher's talk provide a robust framework for strengthening CI/CD pipeline security, moving organizations beyond superficial "green checkmarks" to a state of genuine confidence. Defenders should prioritize the following actions:

1. Implement Differential Scanning for PRs: Adopt diff-aware security scanning for pull requests, configuring tools to report only on new, high-confidence, high-severity vulnerabilities introduced by changes. This drastically reduces alert fatigue for developers and ensures immediate, actionable feedback.
2. Optimize Scanning Tool Configurations: Move beyond default settings for SAST, SCA, and IaC scanning tools. Fine-tune rule sets to prioritize true positives and high-criticality findings. Differentiate between code quality issues and security vulnerabilities, filtering out the former to maintain developer focus.
3. Embrace Policy as Code: Codify security standards and compliance requirements into executable policies using tools like Open Policy Agent (OPA) with Rego, Checkov, or Trivy. Integrate these policies into the PR workflow to automatically enforce security controls for Infrastructure as Code (IaC) and container configurations, preventing insecure deployments.
4. Adopt Ephemeral and Isolated Build Environments: Transition from long-lived "pet" build servers to "cattle" – ephemeral, disposable build runners. Each build job should execute in a pristine, isolated environment that is destroyed immediately afterward, preventing persistence of malware and ensuring a clean slate for every build.
5. Pin Dependencies and Leverage SALSA: Mitigate supply chain risks by pinning all dependencies to immutable references (e.g., full commit SHAs) rather than mutable version tags. Utilize Software Composition Analysis (SCA) tools to vet open-source components. Implement the SALSA framework to generate signed provenance for artifacts, ensuring build integrity and verifying that deployed artifacts originate from trusted, verified sources.
6. Centralize Pipeline Logic and Enforce Two-Person Rules: Establish dedicated, centrally managed repositories for reusable CI/CD workflow logic. Implement a two-person rule for all modifications to pipeline configurations, requiring a second set of eyes or approval to prevent unauthorized changes, especially in the event of compromised credentials.
7. Implement Zero-Trust for Build Runners: Apply zero-trust principles to build environments. Implement network segmentation to isolate build infrastructure from production networks, enforce traffic filtering (deny-by-default with explicit allow lists) to restrict connections to only approved registries and services, and minimize privileges granted to build processes.
8. Strengthen Secrets Management: Proactively prevent secret leakage by implementing pre-commit hooks on developer workstations. Conduct periodic full Git history scans using tools like truffleHog or git-leaks to identify accidentally committed credentials. Adopt OpenID Connect (OIDC) for short-lived, job-specific cloud access tokens, eliminating persistent static keys. Store all secrets in secure vaults (e.g., HashiCorp Vault, AWS Secrets Manager) and implement immediate credential rotation policies upon detection of any compromise.
9. Integrate Human Oversight and EPSS: Recognize the limitations of automated tools. Incorporate professional code reviews and expert security triage to identify complex logic flaws, business logic vulnerabilities, and intentional backdoors that tools may miss. Prioritize vulnerabilities based on actual exploitability using EPSS in conjunction with CVSS scores.
10. Foster a Security Culture: Shift the mindset from security being a "blocker" to an enabler. Build trust between security and development teams by providing actionable findings, clear remediation guidance, and involving developers in security decisions, encouraging them to take ownership of security as they would with any other code defect.

Key Takeaways

• Combat Security Theater: Move beyond a performative security posture by focusing on high-confidence, actionable findings from security scanning tools. Eliminate alert fatigue by fine-tuning rules, filtering noise, and ensuring that security checks can block builds.
• Secure Both Code and Pipeline: Implement robust security measures not only in the application code (via diff-aware scanning, optimized SAST/SCA, ASPM) but also of the CI/CD pipeline infrastructure itself (e.g., against pipeline poisoning, dependency abuse, and over-privileged runners).
• Automate Enforcement with Policy as Code: Leverage tools like Open Policy Agent (OPA) with Rego, Checkov, or Trivy to codify security standards. This enables consistent, version-controlled, and automated enforcement of security policies for Infrastructure as Code, preventing insecure configurations from reaching production.
• Adopt Ephemeral and Verified Environments: Utilize ephemeral, isolated build runners for every job to eliminate the "pet problem" of persistent, vulnerable build servers. Combine this with the SALSA framework and pinned dependencies (immutable commit SHAs) to ensure artifact integrity and mitigate supply chain risks.
• Prioritize Real Exploitability and Human Context: Supplement CVSS scores with tools like EPSS (Exploit Prediction Scoring System) and reachability analysis to prioritize vulnerabilities based on their actual exploitability and risk to the organization. Incorporate human code reviews and expert triage for nuanced logic flaws and business logic vulnerabilities that automated tools often miss.
• Implement Robust Secrets Management: Prevent credential theft by using pre-commit hooks, layered Git history scanning, and OIDC for short-lived, job-specific cloud access tokens. Always store secrets in secure vaults and ensure immediate rotation of any detected credentials.

About the Speaker(s)

Derek Fisher is a seasoned expert in cybersecurity and software engineering. He is a Professor at Temple University in Philadelphia, where he has taught software security and ethical hacking for eight years. Fisher recently took on the role of Director of the cyber program, overseeing the cybersecurity curriculum for the university's master's, bachelor's, and certificate programs.

Beyond academia, Fisher is the founder of Securely Built, his consultancy dedicated to training and education in various cybersecurity domains, including software security, AI security, and building secure systems. With approximately 30 years of engineering experience, he began his career in hardware engineering for military and commercial applications before transitioning to software engineering as a C developer in the healthcare sector. His journey into cybersecurity saw him working in healthcare and financial industries before establishing his consultancy.

Fisher is also an accomplished author, having written "Application Security Program Handbook" and "Threat Modeling Best Practices," the latter released in late 2023. In his free time, he enjoys tinkering, camping, playing video games, and participating in Spartan races. He can be found on LinkedIn, Substack, and YouTube.