An Industry Interview Study of Software Signing for Supply Chain Security

An Industry Interview Study of Software Signing for Supply Chain Security

Speakers: Kelechi G. Kalu

Conference: USENIX Security

YouTube: https://www.youtube.com/watch?v=OhHwC7LXuvM

Overview

Kelechi G. Kalu, a third-year PhD student at Purdue University, presented a seminal industry interview study on the practical implementation of software signing for supply chain security. This research, supported by the NSF, Cisco, and Google, marks the first in-depth interview study on software signing practices within the industry, offering critical insights into its current state, challenges, and perceived importance. The talk unveils a refined software supply chain factory model, illustrating optimal points for signature creation and verification, and provides empirical data on the prevalent organizational, technical, and human challenges hindering effective signing.

The study’s findings are particularly pertinent given the escalating threat of software supply chain attacks and the increasing regulatory pressure for enhanced software security. By analyzing practitioner views on the impact of these external factors, the research informs improvements in industry-leading technologies like Sigstore and guides organizations toward more robust software supply chain security postures. This work is a crucial contribution to both security literature and practice, highlighting the significant gap between the recognized importance of software signing and its often-flawed real-world implementation.

Background

▶ Watch: Introduction and key research contributions (0:00)

Software production is commonly conceptualized through the software factory model, which visualizes the engineering process as an intricate integration of first-party and third-party components, subsequently built, packaged, and distributed for downstream consumption. This complex interplay of engineering and distribution forms what is known as the software supply chain. The security of this chain is fundamentally defined by three key properties: validity, ensuring the integrity of components and actors; transparency, maintaining comprehensive knowledge of components and actors; and separation, isolating components and actions performed by different actors. A secure software supply chain must rigorously adhere to these properties.

Provenance, a combination of validity and transparency, relates to the traceability of actors and actions within the software supply chain. Software signing is a prime example of a provenance-enhancing security technique. It applies traceability by cryptographically ensuring that the contents of software artifacts have not been altered since they were signed, thereby guaranteeing integrity and attributing origin. In recent years, Sigstore has emerged as an industry-leading technology for implementing software signing, gaining widespread adoption across major ecosystems like PyPI, npm, and Kubernetes, and by organizations such as Verizon, Autodesk, and Yahoo.

The practical application of software signing involves two core components: the signature creation process and the signature verification process. For instance, when the Guava dependency from Maven Central is signed, it uses a private key with a specific key ID. Later, when a downstream user downloads Guava, they verify its authenticity and integrity using the corresponding public key. This process allows the user to confirm the identity of the upstream creator and ensure the package remains unaltered.

Despite its critical role, the current state of software signing in practice is concerning. Recent studies from Kalu’s lab have revealed alarmingly low quality and adoption rates of software signatures across major open-source software registries. Signature quantities, unless mandated by registries, consistently remain low. Furthermore, the quality—defined as the ability to create and successfully verify signatures—fluctuates, often falling short of ideal implementation. This indicates that software signing is frequently poorly implemented, or at least inconsistently so. The consequences of this negligence are severe: missing or weak signing infrastructures have been actively exploited in real-world attacks, such as those targeting Microsoft and Adobe, where compromised signing infrastructures were used to sign malware. Compounding these issues, existing standards and recommendations, like the NIST SP800 guidelines informing the 2021 U.S. Executive Order on Cybersecurity, often provide vague directives, merely stating "use code signing" without offering concrete guidance on where and how it should be implemented within modern software pipelines.

Given this critical context, the research aimed to answer several key questions:

1. Where and how is signing implemented by software teams?
2. What are the challenges affecting the implementation and use of software signing in practice?
3. What is the perceived importance of software signing in mitigating risks?
4. How do internal and external signing events influence the use of software signing in practice?

This article will primarily focus on the first three questions, as highlighted in the presentation.

Key Findings

▶ Watch: Visualizing software signing with Guava example (4:00)

The study utilized qualitative interviews with 18 experienced security practitioners from 13 different organizations, boasting a median of 13 years of experience. This high level of expertise ensured rich, informed responses regarding security controls and software signing. The interviews, which lasted a median of 50 minutes, were analyzed using thematic and framework analysis, leveraging the software factory model as a guiding framework. Data saturation was achieved by the 11th interview, indicating a comprehensive understanding of the practitioners' perspectives.

The research yielded several significant findings categorized by the initial research questions:

1. Where and How Signing is Implemented:

The study first refined the traditional software factory model to explicitly map points where signature creation (P) and verification (V) ought to occur. This modified model identified multiple points for creation (P_I, P_S, P_B, P_A – referring to initial component, source, build, and artifact signing, respectively) and verification (V_I, V_S, V_D, V_B – referring to initial component, source, deploy, and build verification). A critical finding was the stark discrepancy between creation and verification: while signatures are "readily created" at various points in the supply chain, they are "not readily verified." This unidirectional application of signing emerged as a significant problem, suggesting that the integrity guarantees provided by signing are often not fully realized due to a lack of corresponding verification checks.

2. Challenges Affecting Implementation and Use:

The practitioners reported a diverse array of challenges, thematically grouped into technical, organizational, and human categories. While some, like key management and lack of expertise, were anticipated from existing literature, several surprising patterns emerged:

• Lack of Verification: This was a significantly mentioned problem. Participants noted that "not much importance is given to verification." Reasons cited included a profound lack of trust in open-source actors and projects, leading organizations to not even bother verifying signatures from these sources, and tooling challenges that complicated the implementation of robust verification workflows.
• Operationalization Difficulty in Large Organizations: Counter-intuitively, only subjects from large organizations reported significant difficulty in operationalizing the signing process. This was surprising, as large organizations are typically presumed to have more resources to streamline such processes.
• Lack of Management Incentives: Only subjects from organizations with non-security-focused product areas reported a lack of management incentives as a major barrier to implementing software signing. This suggests that without direct alignment to core business objectives or explicit mandates, signing initiatives struggle to gain traction.

3. Perceived Importance of Software Signing:

Practitioners' views on the importance of software signing varied, falling into three distinct groups:

• Crucial to Provenance and Integrity: This group viewed signing as fundamental for establishing provenance and ensuring integrity, considering it an indispensable security technique.
• Secondary Security Technique: Another group saw signing as important but often secondary, emphasizing that it needed to be "paired with some sort of transparency technique like attestations to be very important in practice." This perspective suggests a recognition of signing’s limitations when used in isolation.
• Requirement-Driven Practice: The final group viewed signing primarily as a compliance or platform-driven requirement. If the platforms they were building for did not mandate signing, they often "mostly kind of left it out of the picture," indicating a reactive, rather than proactive, approach to its implementation.

These findings collectively paint a picture of software signing as a security measure that is understood in theory to be vital but struggles with consistent, effective, and fully realized implementation in practice, often due to a complex interplay of technical, organizational, and human factors.

Technical Deep Dive

▶ Watch: Concerning state: low quality and adoption of signing (4:40)

The core of this study's technical contribution lies in its refinement of the software factory model to explicitly incorporate software signing as a critical provenance mechanism. The traditional software factory model depicts a lifecycle from initial component integration to final package distribution. Kalu and co-authors augmented this model by identifying specific "points" where cryptographic operations—signature creation and verification—should ideally occur to bolster supply chain security.

For signature creation, the model designates points such as:

• P_I (Initial Component Signing): When third-party or internal components are first introduced into the build environment.
• P_S (Source Signing): After source code is committed, ensuring its integrity.
• P_B (Build Signing): After a build process completes, attesting to the integrity of the compiled artifact.
• P_A (Artifact Signing): For the final distributable package, just before it is released.

Conversely, for signature verification, corresponding points are identified:

• V_I (Initial Component Verification): When components are ingested, verifying their upstream signatures.
• V_S (Source Verification): When source code is retrieved for a build, verifying its integrity.
• V_D (Deployment Verification): Before software is deployed to production, ensuring the integrity of the release artifact.
• V_B (Build Verification): During the build process, confirming the integrity of intermediate artifacts or dependencies.

The study's empirical data revealed a critical disconnect: while organizations showed some propensity to implement signature creation at various P-points (e.g., P_I, P_S, P_B, P_A), the corresponding V-points for verification (V_I, V_S, V_D, V_B) were "not readily verified." This creates a "unidirectional" signing practice where the effort of creating a signature is undermined by the absence of a robust verification step, rendering the integrity guarantees largely moot.

The example of the Guava dependency from Maven Central illustrates the ideal process. At the signature creation phase, Guava is signed using a private key associated with a unique key ID. This private key is controlled by the package maintainer. When a downstream user downloads Guava, they are expected to verify its authenticity using the corresponding public key, which is typically made available through a trusted repository or key server. This verification step confirms two critical aspects: the identity of the upstream user (i.e., the person or entity who created and signed the package) and that the package contents have not been altered since it was signed. Without this verification, a malicious actor could replace the legitimate Guava package with a tampered version, and the downstream user would have no cryptographic means to detect the compromise.

The role of Sigstore is particularly relevant here. As an industry-leading technology, Sigstore aims to simplify the process of software signing and verification for open-source software. It provides a free, open-source service that allows developers to sign software artifacts using ephemeral keys and store the resulting signatures and certificates in a tamper-proof public log (like a transparency log). This addresses some of the historical challenges of key management and distribution that practitioners cited. However, even with tools like Sigstore, the study highlighted that tooling challenges persist, making it difficult for organizations to integrate robust verification workflows into their existing pipelines. The "lack of trust in open-source actors and projects" also contributes to this problem; if developers inherently distrust the source, they may not invest in verifying signatures, even if the tools exist.

Furthermore, the study illuminated that operationalizing signing processes, even for large organizations with ample resources, presents significant hurdles. This often involves integrating signing into complex CI/CD pipelines, managing key lifecycles, and ensuring consistent application across diverse software projects. The vague guidance from standards like NIST SP800—which simply states "use code signing" without specifying how or where—exacerbates these operational difficulties, leaving organizations to devise their own, often inconsistent, approaches. This technical gap between the theoretical benefit of signing and its practical, consistent implementation underscores the urgent need for improved tooling, clearer standards, and better integration strategies.

Demo / Proof of Concept

▶ Watch: Key research questions addressed by the study (7:00)

This presentation is an academic interview study focused on understanding the practices and challenges of software signing within the industry. As such, it does not feature a traditional technical demonstration of a tool, an exploit, or a proof of concept. The "demonstration" of this work is in its empirical data, the refined software factory model, and the qualitative insights gathered from experienced security practitioners.

Defensive Implications

▶ Watch: Study methodology and participant demographics (8:00)

The findings of this study provide crucial, actionable insights for organizations seeking to strengthen their software supply chain security through effective software signing. The primary defensive implication is a resounding call for organizations to prioritize and implement robust signature verification workflows, not just signature creation. The current "unidirectional" approach, where signatures are created but not verified, renders the entire exercise largely futile. Defenders must:

1. Mandate and Implement Verification: Organizations should establish policies that mandate the verification of all ingested software artifacts, whether internal or third-party. This includes components, libraries, and executables. Tools and processes need to be put in place to automatically check signatures at critical points in the supply chain, such as during component ingestion (V_I), before building (V_B), and prior to deployment (V_D).
2. Improve Verification Tooling and Integration: The study highlighted significant tooling challenges in implementing proper verification. Defenders should invest in or advocate for the development of more user-friendly, easily integratable, and reliable verification tools. For instance, while Sigstore simplifies signing, its verification components need to be seamlessly integrated into CI/CD pipelines and package managers to reduce friction and increase adoption. Organizations should assess their existing toolchains and identify gaps in verification capabilities.
3. Address Trust in Open-Source Components: The reported lack of trust in open-source actors and projects is a critical barrier to verification. Defenders need to evaluate the provenance and signing practices of their open-source dependencies. This might involve curating trusted lists of components, contributing to open-source security initiatives, or using services that provide verified open-source packages. Even with perceived distrust, implementing verification provides an objective integrity check.
4. Strengthen Organizational Incentives and Mandates: For organizations, especially those with non-security-focused product areas, a lack of management incentives was a significant barrier. Security leaders must proactively articulate the business risk associated with insecure software supply chains and the role of signing in mitigating these risks. This requires translating technical benefits into business value, obtaining executive buy-in, and establishing clear internal mandates for signing and verification.
5. Develop Concrete Internal Standards and Guidance: The vagueness of external standards (e.g., NIST SP800) means organizations cannot rely solely on them for implementation details. Defenders should develop detailed, internal best practices and guidelines for where (e.g., at each P and V point in the modified software factory model) and how software signing and verification should be performed within their specific pipelines. This includes guidance on key management, artifact signing policies, and incident response for signature failures.
6. Focus on Operationalization: The challenges faced by large organizations in operationalizing signing highlight the need for dedicated resources and expertise. Defenders should treat the integration of signing into CI/CD pipelines as a significant engineering effort, allocating sufficient budget, personnel, and time to ensure it is done correctly and scales effectively. This includes automating signing and verification processes as much as possible to reduce human error and overhead.

By addressing these defensive implications, organizations can move beyond a superficial understanding of software signing to a truly robust, verifiable, and resilient software supply chain security posture.

Key Takeaways

• Software signing is critically under-verified: Despite its importance for provenance and integrity, most organizations create signatures but fail to implement robust verification workflows, rendering the security benefits largely moot.
• Challenges are multifaceted: Effective software signing is hampered by a complex interplay of technical issues (e.g., key management, tooling challenges), organizational hurdles (e.g., operationalization difficulty, especially in large firms), and human factors (e.g., lack of expertise, lack of trust in open-source).
• Vague standards hinder practical implementation: Current industry standards and recommendations for code signing often lack concrete guidance on where and how to implement signing in modern software pipelines, leaving organizations to navigate complex decisions independently.
• Organizational context impacts adoption: The perceived importance and implementation of software signing vary significantly based on an organization's size and whether its product areas are security-focused, with non-security-focused teams often lacking management incentives.
• Improved tooling and strong incentives are vital: To move forward, there is an urgent need for more user-friendly and integratable verification tools, coupled with stronger internal mandates and management incentives to prioritize and adequately resource software signing initiatives.

About the Speaker(s)

Kelechi G. Kalu is a third-year PhD student at Purdue University, advised by Dr. James Davis. His research focuses on software supply chain security, particularly the practical aspects of software signing. This work on an industry interview study of software signing was supported by the NSF, Cisco, and Google, underscoring its relevance and impact within the security community. Kalu is actively engaged in further research aimed at improving the usability and effectiveness of next-generation software signing tools through formative usability evaluations. He is also investigating the use of software signing and other provenance techniques to establish trust in the open-source software supply chain, specifically by establishing consistency in published provenance data throughout the publishing pipeline. His work aims to bridge the gap between theoretical security concepts and their practical, real-world application.