Notary Project: The Key To Secure Software Supply Chain - Yi Zha & Guillaume Gill

Notary Project: The Key To Secure Software Supply Chain

Speakers: Yi Zha, Senior Product Manager, Microsoft; Guillaume Gill, Technical Lead Architect, OnLogic

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=1Fw0ajODU8

Overview

The rapid adoption of cloud-native technologies and the increasing reliance on open-source components have amplified the criticality of software supply chain security. This talk, presented by Notary Project maintainer Yi Zha from Microsoft and Guillaume Gill, Technical Lead Architect at OnLogic, delves into how the Notary Project addresses fundamental challenges in this domain. As an incubating project within the Cloud Native Computing Foundation (CNCF), Notary Project provides a standard-based framework and tooling to ensure the authenticity and integrity of cloud-native artifacts. The core problem it solves revolves around two questions: "How can I trust the container images or other artifacts I use?" and "How can I ensure these artifacts haven't been maliciously modified during publishing or distribution?"

The session highlights Notary Project's mission to provide robust, standard-based solutions and tools to secure the software supply chain end-to-end. Yi Zha outlines the project's capabilities across the entire artifact lifecycle, from acquisition and internal build processes to publishing and deployment. Guillaume Gill then provides a real-world perspective, sharing OnLogic's journey in modernizing its infrastructure with Kubernetes and cloud-native solutions, emphasizing the critical role Notary Project plays in enhancing their software supply chain security and meeting stringent compliance requirements for petabytes of data across multiple clouds.

This talk is particularly relevant in the current landscape, where initiatives like the Cyber Attack Resilience Act underscore the need for verifiable software provenance. Notary Project offers concrete, actionable tools and methodologies to achieve this, making it an indispensable component for any organization committed to strengthening its cloud-native security posture. By focusing on open standards and an extensible plugin architecture, Notary Project aims to be a flexible and future-proof solution for securing the digital assets that power modern applications.

Background

▶ Watch: Notary Project introduction: ensuring artifact authenticity and integrity (1:10)

The modern software development landscape is characterized by a complex web of dependencies, often leveraging numerous open-source components and distributed across multi-cloud environments. While this fosters innovation and accelerates development, it simultaneously introduces significant security vulnerabilities. The challenge of securing the software supply chain has grown exponentially, with incidents demonstrating how a single compromised component or an unverified artifact can lead to widespread system breaches. Organizations struggle to answer fundamental questions about the provenance and trustworthiness of the software they acquire, build, and deploy.

Traditional methods for ensuring software integrity, often rooted in desktop application signing, rely on trusted Certificate Authorities (CAs) and embedding certificates within executables. While effective for their original purpose, these mechanisms often fall short in the dynamic, distributed, and highly automated cloud-native ecosystem. Issues such as managing private keys across numerous build agents, ensuring non-repudiation, and integrating with diverse cloud provider services present significant hurdles. Furthermore, solutions like transparency logs, while offering public verifiability, may not be suitable for enterprises that cannot publicly disclose their entire build pipeline information due to proprietary concerns or regulatory compliance.

OnLogic, as detailed by Guillaume Gill, exemplifies many organizations facing these challenges. As a leading company in Digital Asset Management (DAM) systems, handling petabytes of data for major global clients across multiple regions and cloud providers, OnLogic's migration to a Kubernetes-based, cloud-native infrastructure necessitated a radical overhaul of its security practices. Their specific needs included:

• Trusting dependencies: Ensuring that all components, from base images to application libraries, originate from trusted sources.
• Asymmetric signature usage: Leveraging public/private key pairs for robust, verifiable attestations.
• Preventing unsafe key handling: Ensuring that signing keys are securely stored and accessed, preventing their compromise or unauthorized use.
• Flexible tooling: The ability to integrate with various key management systems, whether on-premise HSMs or cloud provider vaults.
• Multi-level attestations: A critical requirement to certify artifacts at different stages of their lifecycle (e.g., development, QA, UAT, production) with varying levels of trust and associated gates. This addresses the nuance that an artifact might be "safe" (unmodified) but not yet "ready for production" (unstable or unverified for specific environments).

The Notary Project emerged to fill this gap, offering a standardized approach to cryptographically sign and verify artifacts, providing the necessary assurance of authenticity and integrity throughout the software supply chain, and addressing the limitations of prior approaches in a cloud-native context.

Key Findings

▶ Watch: OnLogic's real-world software supply chain practice begins (6:20)

The Notary Project offers a comprehensive, standard-based solution for securing the cloud-native software supply chain, addressing the critical issues of authenticity and integrity. The key findings and contributions highlighted in the talk reveal a project meticulously designed for enterprise adoption, flexibility, and robust security.

Firstly, Notary Project's core mission is to provide standard-based solutions and tools to ensure the authenticity and integrity of cloud-native artifacts. This mission is realized through its primary command-line interface tool, Notation, which allows users to sign and verify various artifacts. The project supports an end-to-end security framework, enabling organizations to integrate signing and verification at every stage:

• Acquisition: Validating signatures of external images (e.g., from Bitnami on Docker Hub) before internal use.
• Build: Signing internally built application images, base images, and associated metadata like SBOMs (Software Bill of Materials) and vulnerability reports.
• Publishing: Signing artifacts before releasing them to public or private registries.
• Deployment: Verifying signatures of artifacts before they are deployed to production or other environments, often enforced by admission controllers.

A significant strength of Notary Project is its extensibility through a plugin framework. Notation allows for custom plugins to integrate with diverse key management systems, providing flexibility crucial for varied enterprise environments. This includes community plugins for major cloud providers like Azure Key Vault, AWS Signer, and Alibaba Cloud, as well as on-premise solutions like HashiCorp Vault and dedicated HSM devices. The plugin architecture is language-agnostic, supporting Golang, .NET, and other languages, by defining a simple three-command interface (describe key, generate signature, get plug-in metadata). This ensures that organizations can leverage their existing secure key infrastructure without being locked into specific vendor solutions.

Security of the Notary Project itself is paramount. The project adheres to rigorous security best practices, undergoing regular third-party security audits. Key features like timestamping and revocation checking have been audited in 2022, 2023, and are planned for 2024/2025. This commitment to self-security is critical for a tool designed to secure other software. Furthermore, Notary Project tools are built with a philosophy of minimizing dependencies to avoid inheriting supply chain vulnerabilities. The timestamping client, for instance, boasts zero dependencies, a notable achievement for a security-critical library.

The project's foundation on open standards is another key finding. Notary Project specifications are built on top of OCI (Open Container Initiative) standards, enabling seamless artifact management across registries and multi-cloud platforms. It also supports IETF standard signature formats, specifically JSON Web Signature (JWS) for general use and COSE (CBOR Object Signing and Encryption), a concise binary format particularly efficient for resource-constrained environments like edge computing or IoT devices. Future plans include transitioning to OCI 1.1 Referrals API for more elegant signature management.

Finally, a crucial finding from OnLogic's experience is the concept of multi-level attestation. This allows artifacts to be signed multiple times with different keys and trust levels as they progress through the software development lifecycle. For example, an artifact might be signed at "level 0" when it's first built and ready for internal QA, then "level 1" after passing internal QA and vulnerability scans, and finally "level 2" when approved for production. Each level implies specific gates and verification steps, enabling granular control and auditability. This approach ensures that while an artifact might be cryptographically authentic, its suitability for a specific environment (e.g., production) is explicitly attested to.

Technical Deep Dive

▶ Watch: OnLogic's journey to cloud-native security and challenges (7:30)

The Notary Project's technical architecture is meticulously designed to provide robust, flexible, and standard-compliant software supply chain security. At its core, it leverages cryptographic principles to ensure the authenticity and integrity of digital artifacts.

The project’s specifications are built upon established industry standards. It primarily interfaces with the OCI (Open Container Initiative) specification, enabling the signing and verification of container images, Helm charts, and other OCI-compatible artifacts stored in registries. Looking ahead, Notary Project is transitioning to fully support the OCI 1.1 Referrals API, which offers a more elegant and standardized way to manage and reference signatures and other metadata associated with artifacts within OCI registries. Beyond OCI, Notary Project supports IETF standard signature formats: JSON Web Signature (JWS), a widely adopted format for signing arbitrary data, and COSE (CBOR Object Signing and Encryption), a compact binary format especially suited for constrained environments like IoT devices or edge computing due to its efficiency.

The primary user interface for Notary Project is the Notation CLI. This tool orchestrates the signing and verification processes. A key architectural decision for Notation is its plugin framework, which significantly enhances its flexibility. This framework allows Notary to integrate with various key management systems (KMS) and hardware security modules (HSMs). Developers can create plugins in any language (e.g., Golang, .NET) that adhere to a simple plugin interface, responding to just three subcommands: describe key, generate signature, and get plug-in metadata. This design allows for seamless integration with solutions like Azure Key Vault, AWS Signer, HashiCorp Vault, and even custom on-premise HSMs, ensuring that signing keys are securely managed and isolated. For instance, to install a plugin, a user simply executes notation plug-in install <URL>, optionally providing a hash for verification. Registering a new key involves notation key add --plugin <plugin_name> <key_name>.

For consumers, verifying artifacts relies on a trust policy. This policy, defined by the user, specifies the acceptable signers, the root Certificate Authority (CA) certificates within a trust store, and the identity of the signer (typically the subject field of the signer's certificate). This allows organizations to enforce strict rules on which artifacts are considered trustworthy for deployment. For example, a production environment's trust policy might only accept signatures from a specific, highly secure production CA, while a development environment might accept a broader set of internal development CAs.

OnLogic's implementation highlights a critical aspect of enterprise security: multi-level attestation. This workflow involves signing an artifact at different stages of its lifecycle, each signature representing a distinct level of trust and verification.

• Level 0 (Build Chain): An artifact is initially signed immediately after being built by the CI/CD pipeline. This attests that the artifact originates from the organization's trusted build infrastructure.
• Level 1 (Internal QA/UAT): After passing internal quality assurance, vulnerability scanning, and potentially human testing, the artifact receives a second signature. This signifies that it has met internal readiness criteria and is suitable for User Acceptance Testing (UAT) or internal staging environments.
• Level 2 (Production): The final signature is applied once the artifact is fully vetted and approved for production deployment. This signature, often generated by a highly restricted and audited process (e.g., a specific human click or an automated process with stringent checks), confirms its readiness for live services.

Each of these levels can be associated with a distinct CA and trust policy. During deployment, Kubernetes admission controllers (e.g., Kyverno) are configured to check for the appropriate signature chain. An image destined for production must carry the "Level 2" signature from the production CA, or it will be rejected, preventing the deployment of unapproved or improperly vetted software. This granular control is crucial for maintaining compliance and operational stability in complex, regulated environments.

Beyond OCI artifacts, Notary Project is expanding its capabilities to sign "blob files" on the file system, such as standalone SBOMs, AI model files, or WebAssembly modules. This allows organizations to secure any digital artifact, even those not typically stored in OCI registries, by signing them in a trusted domain before distribution. The timestamping client, a zero-dependency library, is another notable contribution, addressing the challenge of providing reliable timestamping for short-lived certificates and ensuring non-repudiation over time.

Demo / Proof of Concept

▶ Watch: OnLogic's key requirements for a secure software supply chain (8:20)

The talk included a concise yet impactful demonstration of Notary Project's capabilities, specifically showcasing a new alpha feature: the ability to sign and verify arbitrary blob files on the file system, exemplified by an SBOM (Software Bill of Materials) file. This demo highlighted Notary Project's expanding scope beyond traditional OCI registry artifacts and its utility for securing critical metadata.

The scenario presented involved an SBOM file generated within a trusted domain on a local file system. Before this SBOM is published or shared, the objective is to sign it to ensure its integrity and authenticity, guaranteeing that any consumer can verify its origin and that it hasn't been tampered with.

The demonstration followed these steps:

1. Key Provisioning: The speaker first confirmed that a signing key was provisioned. For the demo, a test key was used, but in a real-world scenario, this would involve a secure key managed by a KMS or HSM via a Notation plugin.
2. Signing the SBOM: Using an alpha version of the notation CLI, the speaker executed the notation blob sign command. This command was used with specific flags:

• --signature-format cose: To specify the COSE (CBOR Object Signing and Encryption) binary format for the signature, chosen for its conciseness.
• --media-type spdx: To indicate the media type of the blob being signed, in this case, an SPDX (Software Package Data Exchange) SBOM.
• The command would look something like: notation blob sign --signature-format cose --media-type spdx my_application.spdx.json

1. Local Signature Generation: Crucially, after signing, both the original SBOM file and its generated signature file (e.g., my_application.spdx.json.cose) resided on the local file system. The artifact and its signature had not yet left the "trusted domain," emphasizing the ability to secure assets before they are exposed to external systems or networks.
2. Simulated Distribution: The demo then conceptually moved to the consumer side, assuming the SBOM file and its signature had been published and downloaded by a consumer using any file transfer method.
3. Consumer Trust Configuration: As a consumer, before verification, two crucial configurations were shown:

• Root CA Certificate: The root.crt file, representing the trust anchor (the public key of the Certificate Authority that issued the signer's certificate), was configured in the Notation client's trust store.
• Trust Policy: A trust policy was initialized and configured. This policy defines the rules for verification, including the trust store to use and the trust identity. The trust identity typically specifies the expected subject field of the signer's certificate, ensuring that only signatures from approved entities are accepted.

1. Verifying the Signature: Finally, the consumer executed the notation blob verify command:

• notation blob verify --policy <policy_name> my_application.spdx.json my_application.spdx.json.cose
• This command takes the configured policy name, the SBOM file, and the signature file as input.

1. Successful Verification: Upon successful verification, the output confirmed that the signature was valid according to the defined trust policy, allowing the consumer to confidently analyze the SBOM file, knowing its authenticity and integrity.

This demonstration effectively illustrated how Notary Project, through its notation blob commands, extends its robust signing and verification capabilities to any arbitrary file on the file system, providing a flexible and powerful mechanism for securing critical non-OCI artifacts within the software supply chain.

Defensive Implications

▶ Watch: The necessity of asymmetric and secure signatures (8:50)

The Notary Project offers a powerful suite of tools and methodologies that significantly strengthen an organization's defensive posture against software supply chain attacks. Implementing Notary Project effectively can mitigate risks related to unauthorized modifications, unverified origins, and compromised artifacts.

1. Integrate Notary Project into CI/CD Pipelines: The most fundamental defensive measure is to embed Notary Project's notation CLI throughout the entire software development lifecycle. This means:

• Acquisition Stage: Before pulling any external base images, Helm charts, or other third-party dependencies from public registries, use notation verify to validate their signatures against known and trusted publishers (e.g., Bitnami's signed images).
• Build Stage: After building internal application images, generating SBOMs, or creating vulnerability reports, immediately sign these artifacts using notation sign. This establishes an immutable proof of origin from the internal build system.
• Publishing Stage: Sign artifacts again before pushing them to internal or external registries, ensuring that the published version is the one intended.

1. Enforce Multi-Level Attestation: Adopt OnLogic's approach of staged signing. Implement distinct signing keys and CAs for different environments (development, QA, UAT, production). This allows for granular control and verification. An artifact should only be promoted to the next environment if it carries the appropriate signature from the preceding stage, ensuring it has passed all necessary gates (e.g., vulnerability scans, manual reviews, functional tests).

1. Secure Key Management: Prioritize the security of signing keys. Utilize Notary Project's plugin framework to integrate with robust key management solutions such as Hardware Security Modules (HSMs), Azure Key Vault, AWS Signer, or HashiCorp Vault. Keys should never be stored directly on developer workstations or in insecure locations. Implement strong access controls, audit trails, and rotation policies for all signing keys. Asymmetric cryptography is crucial here, as only the private key needs to be secured, while the public key can be widely distributed for verification.

1. Leverage Admission Controllers for Deployment Enforcement: Configure Kubernetes admission controllers (e.g., Kyverno, OPA Gatekeeper) to enforce Notary Project signature verification at deployment time. This is a critical last line of defense. Any container image or Helm chart attempting to deploy to a cluster should be automatically checked for valid Notary Project signatures that match the environment's defined trust policy. If an artifact is unsigned, signed by an untrusted entity, or lacks the required multi-level attestations for that specific environment, the admission controller should reject the deployment, preventing unauthorized or compromised software from running.

1. Define Strict Trust Policies: For every environment, meticulously define the trust policy within Notary Project. This includes specifying the root CA certificates in the trust store and the expected trust identity (e.g., the subject of the signer's certificate). This ensures that only artifacts signed by explicitly approved entities are permitted. Regularly review and update these policies as trust relationships evolve.

1. Utilize Appropriate Signature Formats: Choose between JWS and COSE based on the use case. JWS is suitable for general-purpose applications where human readability or JSON compatibility is preferred. COSE, with its compact binary encoding, is ideal for resource-constrained environments like edge devices or IoT, where bandwidth and processing power are limited.

1. Extend to Non-OCI Artifacts: Don't limit Notary Project to just container images. Leverage the notation blob commands to sign and verify other critical artifacts, such as SBOMs, configuration files, AI/ML models, or WebAssembly modules. This ensures end-to-end integrity for all components of an application.

1. Stay Informed on Roadmap and Community: Monitor the Notary Project's roadmap for upcoming features like enhanced support for in-toto attestations and transparent logs. Engage with the community (e.g., on the CNCF Slack channel) to stay updated on best practices, new plugins, and security advisories.

By systematically applying these defensive implications, organizations can build a robust, verifiable, and auditable software supply chain, significantly reducing their attack surface and enhancing their overall cyber resilience.

Key Takeaways

• Standard-Based Authenticity & Integrity: Notary Project provides standard-based tools and solutions, primarily through the Notation CLI, to ensure the authenticity and integrity of cloud-native artifacts (container images, Helm charts, SBOMs, etc.) across their entire lifecycle.
• Extensible Plugin Architecture: Notation features a highly extensible plugin framework that allows integration with various key management systems (HSMs, Azure Key Vault, AWS Signer, HashiCorp Vault) and custom solutions, offering flexibility without vendor lock-in.
• Multi-Level Attestation for Granular Trust: Organizations like OnLogic leverage Notary Project for multi-level attestation, signing artifacts at different stages (build, QA, production) with distinct keys and trust levels to enforce granular control and auditability throughout the supply chain.
• Secure & Lean Tooling: Notary Project itself adheres to rigorous security best practices, undergoing regular third-party audits and minimizing its own dependencies (e.g., zero dependencies for the timestamping client) to prevent introducing new supply chain vulnerabilities.
• End-to-End Supply Chain Security: Notary Project supports signing and verification for OCI artifacts and is extending to arbitrary "blob files" on the file system, enabling comprehensive security from artifact acquisition and internal builds to publishing and deployment, enforced via admission controllers.
• Foundation in Open Standards: Built upon OCI (including future OCI 1.1 Referrals API) and IETF signature formats (JWS, COSE), Notary Project ensures interoperability and long-term viability for securing cloud-native ecosystems.

About the Speaker(s)

Yi Zha is a Notary Project maintainer and a Senior Product Manager at Microsoft. His work focuses on cloud-native security and red team strategies, bringing deep expertise in securing modern application architectures to the Notary Project. He is instrumental in guiding the project's development and ensuring its alignment with industry security best practices.

Guillaume Gill serves as the Technical Lead Architect at OnLogic. He is at the forefront of his company's efforts to modernize its infrastructure by migrating to cloud-native solutions, particularly those based on Kubernetes. Guillaume's insights in the talk provided a valuable real-world perspective on the practical challenges and solutions in implementing robust software supply chain security within a large-scale, multi-cloud enterprise environment.