Building Trust Through Proactive Security - Key Parts of the Trusted Software Supply Chain

Building Trust Through Proactive Security - Key Parts of the Trusted Software Supply Chain

Speakers: Premaruski (Rogue), Principal Security Engineer, Red Hat

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=Ct_oO9q1LxY

Overview

In an era of increasing supply chain attacks and software vulnerabilities, the concept of a "trusted software supply chain" has become paramount. This talk, delivered by Premaruski, also known as Rogue, a Principal Security Engineer at Red Hat, delves into the critical role of proactive security work in establishing and maintaining this trust. The presentation outlines a strategic shift from reactive vulnerability responses to a deeply integrated, preventative security posture throughout the software development life cycle (SDLC), highlighting Red Hat's practical implementation.

Rogue articulates how proactive security, encompassing activities like static analysis, dynamic analysis, threat modeling, and malware detection, should be an intrinsic part of the build and release process rather than an afterthought. The core message emphasizes moving beyond mere "security checkboxing" to a genuine risk-based security approach. The talk culminates in demonstrating how Red Hat integrates these proactive measures into its Conflux pipeline and explores innovative methods, including AI-driven analysis, to correlate internal scanning results with public vulnerability data to identify and remediate recurring weaknesses more effectively.

The insights shared are crucial for any organization aiming to enhance its software supply chain security, reduce its attack surface, and ultimately deliver more reliable and secure products to customers. By showcasing Red Hat's efforts, the presentation provides a tangible model for how a large software vendor is addressing these complex challenges, offering valuable lessons for developers, security architects, and leadership committed to building security by design.

Background

▶ Watch: Introduction and talk agenda (0:00)

The evolution of software security practices has seen a significant transformation, moving from a predominantly reactive stance to an increasingly proactive one. Historically, security efforts were largely focused on vulnerability responses post-release, addressing issues only after they had been discovered, often by customers or external researchers. This "yesterday" approach was unsustainable in the face of rapidly evolving threats and the sheer volume of software components.

Today, there is an undeniable industry-wide push towards a "shift left" approach. This paradigm advocates for integrating security activities much earlier in the SDLC, ideally from the design and development phases. The goal is to conduct deep software testing and verification before a product is released. This includes comprehensive activities such as Static Application Security Testing (SAST), Dynamic Application Security Testing (DAST), Software Composition Analysis (SCA), and malware detection. Beyond scanning, the shift left also emphasizes collecting extensive metadata from the build process, including Software Bill of Materials (SBOMs), Vulnerability Exploitability eXchange (VEX), and CISA's Software Attestation Framework (SAF), to ensure that builds meet predefined security controls and requirements. The overarching objective is to instill a culture of risk-based security rather than merely fulfilling compliance checkboxes.

Looking towards "tomorrow," the future of software security hinges on enhanced automation and gating mechanisms. This involves leveraging findings from build-time tests to more precisely identify weaknesses and remediate them before they can escalate into exploitable vulnerabilities in customer environments. A critical aspect highlighted is that all proactive security work must be an integral part of the build and release pipeline. It should not exist as a separate, parallel process. Security, as Rogue emphasizes, must be "built-in" to the software supply chain, as addressing issues after release is often too late and significantly more costly. This foundational principle underpins Red Hat's approach to securing its extensive product portfolio.

Key Findings

▶ Watch: Evolution of security: yesterday, today, tomorrow (2:00)

The talk presents several key findings and contributions that collectively define a robust, proactive security strategy for the software supply chain:

1. Conflux Pipeline for Integrated Security: Red Hat's Conflux pipeline emerges as a central innovation. This open-source, comprehensive build, test, and release factory, built on Tekton (an open-source, cloud-native CI/CD solution), fundamentally integrates security by default. It is designed to meet SLSA (Supply Chain Level for Software Artifacts) build level 3 requirements, producing essential build provenance artifacts. This demonstrates a practical, enterprise-grade implementation of secure supply chain principles.

1. Policy-Driven Gating and Provenance: The Conflux pipeline utilizes Tekton Chains to manage supply chain security within Tekton, ensuring every task produces provenance data. This data provides an immutable record of what happened during the build. Furthermore, Conform, another open-source project, is used to validate predefined policy rules. This allows for gating releases based on security compliance, such as blocking builds that use unapproved base images or fail critical security checks, ensuring only trusted components proceed.

1. Proactive and Reactive Vulnerability Management Synergy: The presentation illustrates how proactive build-time vulnerability detection can significantly enhance reactive post-release management. Two scenarios are detailed:

• For new components, detecting a CVE at build time allows for immediate blocking and remediation, preventing the vulnerable code from ever reaching customers or public vulnerability databases.
• For updates to existing components, build-time CVE detection enables comparison with public vulnerability data (e.g., Red Hat CV pages). Any discrepancies can lead to corrective action on public data and the release of a security update, thereby improving the accuracy and timeliness of vulnerability information.

1. AI-Assisted Weakness Identification and Prioritization: A significant finding from a Red Hat Proof of Concept (PoC) is the potential of combining SAST results with historical CVE data using AI. By standardizing data between these disparate sources, AI can identify recurring weakness patterns or CWE categories within specific components. This approach moves beyond simply fixing individual CVEs to addressing the underlying, systemic issues (e.g., authorization errors, CWE-284) that repeatedly lead to vulnerabilities. This allows engineering teams to prioritize remediation efforts on the most impactful and recurring root causes, ultimately reducing the overall vulnerability footprint and improving customer experience.

These findings collectively underscore the necessity of a holistic, integrated, and intelligent approach to security across the entire software supply chain, demonstrating how Red Hat is actively pioneering solutions in this critical domain.

Technical Deep Dive

▶ Watch: Red Hat's Conflux pipeline implementation example (5:35)

The technical foundation of Red Hat's proactive security strategy, as presented by Rogue, is deeply rooted in modern cloud-native CI/CD practices and open-source tooling. At the heart of this approach is the Conflux pipeline, an open-source, comprehensive build, test, and release factory. Conflux is built upon Tekton, a powerful and flexible open-source framework for creating CI/CD systems that run on Kubernetes. This choice highlights a commitment to cloud-native principles, leveraging Kubernetes' scalability and extensibility for build and testing workflows.

Conflux's design is inherently focused on software supply chain security, aiming to meet SLSA (Supply Chain Level for Software Artifacts) build level 3 requirements. SLSA is a security framework designed to prevent tampering, improve integrity, and secure packages and infrastructure. Achieving level 3 signifies a high degree of assurance, requiring hermetic, reproducible builds with verified inputs and a robust provenance system. This level of compliance ensures that the build process itself is highly secure and produces verifiable artifacts.

A crucial component within the Conflux pipeline for achieving this security posture is Tekton Chains. This Kubernetes Custom Resource Definition (CRD) controller is instrumental in managing supply chain security within Tekton. It automatically signs and stores provenance data for every task executed within the pipeline. Provenance data provides an auditable, cryptographic record of how a software artifact was built, including inputs, build steps, and outputs. This ensures transparency and accountability throughout the software production process, making it difficult to tamper with or inject malicious code without detection.

Complementing Tekton Chains is Conform, another open-source project. Conform's role is to validate predefined policy rules against the artifacts and metadata produced by the build pipeline. Rogue provided two illustrative examples of Conform's application:

1. Blocking Policy Violation: If a build task uses a "not allowed built base image," Conform detects this violation. Even if the task technically succeeds, Conform's validation flags it, resulting in the component being blocked from release as part of an official product. This demonstrates strict policy enforcement at a critical stage.
2. Non-Blocking Alert: In another scenario, a CVE scan (likely performed by an SCA scanner) might identify an alert. If the policy specifies this type of alert as non-blocking, the build can still proceed to release. However, this alert provides crucial information for subsequent proactive vulnerability management.

This distinction between blocking and non-blocking rules allows for nuanced policy enforcement, balancing security with release velocity. The information from these non-blocking alerts feeds into the proactive and reactive vulnerability management strategy.

Rogue outlined two distinct use cases for this integrated vulnerability management:

• New Component Build: If a completely new software component (not previously released) is scanned and a CVE is detected, the build is blocked. The component owner is required to fix the vulnerability, and a new, clean build is created and released. In this case, no public CVE page updates are necessary because the vulnerable version never reached the public domain.
• Update to Existing Component: If a new version of an already released component is scanned and a CVE is found, the results are compared against existing public vulnerability data (e.g., Red Hat's public CVE records). If a discrepancy is found (i.e., the new vulnerability is not yet publicly disclosed for this component), Red Hat can proactively update its public data and release the fixed version as a security update. This ensures customers have the most accurate and up-to-date vulnerability information.

A particularly innovative aspect of the talk focuses on leveraging AI for weakness identification. Recognizing the challenge of processing vast amounts of SAST scan results, which often contain numerous false positives, Red Hat conducted a Proof of Concept (PoC). This PoC combined SAST results with historical CVE data (specifically from Red Hat's CVE records over several years) for the same component within a product. The methodology involved:

1. Data Standardization: SAST results and CVE records often use different taxonomies or levels of detail for describing weaknesses. A crucial first step was to standardize this data, mapping specific findings to broader CWE (Common Weakness Enumeration) categories or "weakness classes" (e.g., authorization errors, memory errors, data validation issues).
2. AI-Based Analysis: With standardized data, AI algorithms were employed to find recurring patterns. The goal was to identify which weakness categories consistently appeared in both SAST scans and led to reported CVEs for a given component.

The PoC revealed "pretty interesting patterns," demonstrating that if a specific weakness category, such as CWE-284 (Improper Access Control), repeatedly shows up in SAST results and has historically been associated with CVEs in that component, it indicates a systemic problem. This insight allows engineering teams to prioritize fixing the underlying root cause of the CWE category rather than just patching individual CVEs as they arise. Rogue presented a diagram illustrating sorted SAST results alongside CVE analysis for a single component, clearly showing the correlation of recurring issues like authorization errors. This approach, while still in PoC phase, promises to significantly reduce the number of reported CVEs against Red Hat's portfolio and improve customer experience by proactively addressing fundamental code quality issues.

During the Q&A, a pertinent question arose about the risk of creating a "self-fulfilling prophecy" by only prioritizing historical weakness categories. Rogue clarified that the intent is not to ignore new or less frequent weakness types but to provide a prioritization model. By addressing the most common and impactful recurring weaknesses first, teams can eliminate significant sources of vulnerabilities. Once these are tackled, attention can shift to other repeating patterns or high-severity, non-recurring issues. This strategic prioritization aims to move beyond superficial fixes to deeper, more effective remediation of root causes.

Demo / Proof of Concept

▶ Watch: Conflux: Tecton chains, provenance, and policy validation (6:50)

While the talk did not feature a live, interactive demonstration in the traditional sense, it effectively presented several concrete examples and diagrams that served as a proof of concept for Red Hat's proactive security initiatives.

The functionality of the Conflux pipeline and its integrated security controls was demonstrated through high-level diagrams and specific use cases. Rogue walked through an example pipeline, illustrating how every task, from the initial build to the final artifact creation, is executed in a controlled manner and produces provenance data using Tekton Chains. The role of Conform in validating policy rules was clearly shown with two distinct scenarios:

1. Policy Violation Blocking a Release: A diagram depicted a build with various test results, where one task failed to meet a predefined policy rule because an "not allowed built base image" was used. Even though the task technically completed successfully, Conform's validation identified the policy breach, leading to the component being blocked from release. This served as a clear demonstration of how security policies are enforced to prevent insecure components from reaching customers.

1. Non-Blocking Alert for CVE Scan: Another diagram illustrated a build where all policy tests passed, but an SCA (Software Composition Analysis) scanner triggered an alert related to a CVE. This alert, specified as non-blocking by policy, allowed the build to proceed. This example highlighted the system's ability to provide critical security information without necessarily halting the entire release process, allowing for subsequent proactive remediation steps.

Beyond the pipeline mechanics, the most significant Proof of Concept (PoC) detailed was the AI-based analysis for correlating SAST results with historical CVE data. Rogue explicitly stated that this was a PoC conducted at Red Hat earlier in the year, not yet implemented in production. The PoC involved:

• Data Collection and Standardization: Gathering SAST scan results and CVE data (specifically Red Hat's internal CVE records) for a single component within a product over several years. A critical step was the data standardization process, as SAST tools and CVE records often describe weaknesses differently. This involved mapping various findings to common CWE (Common Weakness Enumeration) categories or "weakness classes."
• Pattern Identification: Using AI-based analysis, the PoC aimed to find patterns where specific weakness categories consistently appeared in both SAST results and led to reported CVEs for that component.
• Result Visualization: A compelling diagram presented the findings: on the left, sorted SAST results for the component; on the right, an analysis of CVEs for the same component and product. The visualization clearly showed recurring patterns, such as "authorization errors" (which maps to CWE-284), indicating a systemic issue.

This PoC demonstrated the potential to move beyond individual vulnerability fixes to addressing underlying, recurring root causes. It provides a tangible example of how advanced analytics can transform raw security data into actionable insights for targeted remediation, even if it's not yet a fully deployed solution. The PoC's results sparked a lively Q&A, further validating its relevance and potential impact.

Defensive Implications

▶ Watch: Proactive and reactive vulnerability management collaboration (9:40)

The insights from Premaruski's talk offer several crucial defensive implications for organizations striving to build a more trusted software supply chain:

1. Integrate Security into the SDLC, Not as an Add-on: The fundamental takeaway is to fully embed security activities like SAST, DAST, SCA, and malware detection directly into the build and release pipelines. Security should be "built-in" from the start, as part of the normal development workflow, rather than being an external, post-build audit. This shift-left approach dramatically reduces the cost and complexity of remediation.

1. Adopt Supply Chain Security Frameworks and Tools: Organizations should evaluate and adopt frameworks like SLSA to guide their supply chain security efforts. Implementing solutions akin to Red Hat's Conflux pipeline, which leverages Tekton for cloud-native CI/CD, provides a robust foundation. Utilizing tools like Tekton Chains for generating provenance data is critical for auditability, integrity, and non-repudiation of build artifacts.

1. Implement Policy-as-Code for Automated Gating: Define clear security policies (e.g., approved base images, acceptable scan result thresholds) and enforce them automatically using tools like Conform. This enables automated gating of releases, preventing non-compliant or insecure components from proceeding further in the supply chain. Differentiate between blocking and non-blocking rules to balance security with development velocity, but ensure critical violations halt the process.

1. Leverage Build-Time Vulnerability Detection for Proactive Remediation: Establish processes to detect vulnerabilities (e.g., CVEs) during the build phase. For new components, this allows for immediate fixes before public exposure. For updates to existing components, it enables organizations to proactively update public vulnerability data and issue security patches, ensuring customers have the most accurate information.

1. Prioritize Root Cause Remediation with Data-Driven Insights: Move beyond solely patching individual CVEs. By correlating internal scan results (like SAST) with historical public CVE data, organizations can identify recurring weakness categories or CWEs specific to their components. This AI-assisted analysis (as demonstrated in Red Hat's PoC) enables engineering teams to focus on fixing the underlying, systemic issues that repeatedly lead to vulnerabilities (e.g., CWE-284 for authorization errors). This strategic prioritization leads to a more significant reduction in the overall attack surface.

1. Invest in Data Standardization and Analytics: The effectiveness of correlating diverse security data sources hinges on data standardization. Organizations should invest in mapping different vulnerability and weakness taxonomies to a common framework (like CWE) to facilitate meaningful analysis. Developing or adopting tools for AI-driven pattern recognition can further enhance the precision of weakness identification and threat modeling.

1. Maintain Accurate and Timely Public Vulnerability Data: For software vendors, ensuring that public vulnerability databases (e.g., internal security advisories, CVE records) are accurate and promptly updated based on internal findings is vital for customer trust and effective risk management across the ecosystem. This also contributes to the broader community by providing better data for everyone.

By embracing these defensive implications, organizations can shift from a reactive security posture to one that is truly proactive, integrated, and intelligent, thereby building genuine trust in their software products and supply chains.

Key Takeaways

• Security Must Be Built-In: Proactive security activities are not optional side processes; they must be fully integrated into the software development life cycle, particularly within the build and release pipelines, to ensure security by design.
• Leverage Open-Source for Supply Chain Trust: Tools like Red Hat's Conflux pipeline, built on Tekton, demonstrate how open-source, cloud-native solutions can meet stringent security requirements like SLSA build level 3 and generate critical provenance data using Tekton Chains.
• Policy-Driven Gating is Essential: Implement robust policy enforcement using tools like Conform to automatically validate build artifacts and metadata against predefined security rules, preventing non-compliant components from reaching production.
• Proactive Vulnerability Management Enhances Reactive Measures: Detecting vulnerabilities during the build process enables organizations to either prevent new vulnerable code from ever being released or to proactively update public vulnerability data and issue timely security updates for existing components.
• AI Can Identify Recurring Weakness Patterns: By standardizing and correlating SAST results with historical CVE data, AI can reveal systemic weakness categories (e.g., CWE-284) that repeatedly lead to vulnerabilities, allowing for more strategic and impactful remediation efforts focused on root causes.
• Prioritize Root Cause Fixes Over Individual Patches: Shift focus from merely patching individual CVEs to addressing the underlying, recurring weaknesses identified through data analysis. This approach leads to a more significant reduction in the long-term vulnerability footprint and improves overall software quality.

About the Speaker(s)

Premaruski, who prefers to be called Rogue, is a Principal Security Engineer at Red Hat. In this role, he is at the forefront of designing and implementing proactive security measures within Red Hat's extensive software development and delivery processes. His work focuses on integrating security deeply into the software supply chain, leveraging advanced techniques and open-source solutions to build trust and enhance the security posture of Red Hat's products. His expertise spans vulnerability management, secure SDLC practices, and the application of cutting-edge technologies like AI to security challenges.