TUF-en up Your Software Supply Chain - Marina Moore, Edera & Kairo De Araujo, Independent

TUF-en up Your Software Supply Chain - Marina Moore, Edera & Kairo De Araujo, Independent

Speakers: Marina Moore, Research Scientist, Edera; Kairo De Araujo, Open Source Software Engineer, Independent

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=lIYXVIPsk_U

Overview

In an era of increasing software supply chain attacks, ensuring the integrity and authenticity of distributed software is paramount. This talk, "TUF-en up Your Software Supply Chain," delivered by Marina Moore and Kairo De Araujo, delves into The Update Framework (TUF), a robust security framework designed to protect software distribution and updating processes. Moore and De Araujo, both maintainers of the TUF project, highlighted how TUF addresses critical vulnerabilities often overlooked by simpler signing mechanisms, particularly in the context of distributing modern supply chain metadata like SBOMs and attestations.

The core problem TUF tackles is the secure distribution of software and its associated metadata, ensuring consumers receive not only the correct package but also the accurate, untampered information about its dependencies and vulnerabilities. The speakers demonstrated how TUF’s architectural design provides compromise resilience, a crucial feature that limits the impact of a security breach and enables secure recovery. This talk is highly relevant for any organization grappling with the complexities of software supply chain security, offering a battle-tested framework for establishing and maintaining trust in their software artifacts.

Background

▶ Watch: Introduction and the software supply chain metadata problem (0:00)

The landscape of software distribution has grown increasingly complex, moving beyond just delivering an executable to now include a rich ecosystem of associated metadata. As Marina Moore explained, alongside the primary software image (e.g., an Nginx container), developers are now expected to distribute Software Bill of Materials (SBOMs), Vulnerability Exploitability eXchange (VEX) documents, attestations (e.g., build provenance), and policy information. The challenge lies in ensuring that this entire collection of artifacts reaches the software consumer securely, without any tampering or malicious substitution.

The fundamental problem, as De Araujo and Moore articulated, is that traditional approaches to secure software distribution often fall short. A common first thought is to simply sign the software. However, this naive approach suffers from several critical flaws:

1. Malware Replacement: An attacker can replace legitimate software with malware.
2. Rollback Attacks: An attacker can replace a newer, patched version of software (e.g., Foo 2.3) with an older, vulnerable, but still validly signed version (e.g., Foo 2.2). The consumer, unaware of the newer version, would accept the old, signed package. This is a subtle yet potent attack vector.
3. Arbitrary Signature Acceptance: An attacker could sign malicious software with their own arbitrary key. Without a mechanism to verify who is signing and if they are authorized, a consumer might accept any signed package.
4. Key Distribution Problem: This quickly escalates into a key distribution problem. How does the consumer securely obtain the correct public key to verify the signature? This often leads to a recursive problem of using keys to sign other keys, lacking a clear, secure root of trust.

These limitations underscore the need for a more comprehensive framework that not only verifies signatures but also addresses issues of freshness, versioning, and the secure distribution of the keys themselves. TUF was designed precisely to overcome these challenges, offering a robust solution for a secure software supply chain.

Key Findings

▶ Watch: Why simple signatures fail for secure software distribution (2:00)

TUF introduces a sophisticated framework for secure software distribution and updating, built upon several core principles and mechanisms that ensure freshness, consistency, and integrity. The primary contribution of TUF is its compromise resilience, meaning that even if certain components of the system are compromised, the overall security of the software distribution process remains intact. As Moore emphasized, "only one of these being compromised is insufficient to actually do an attack." This significantly reduces the impact of a single point of failure and allows for secure recovery.

The key findings and contributions of TUF include:

• Layered Trust and Detached Signatures: Instead of embedding signatures directly into packages, TUF uses detached signatures and a hierarchy of metadata files. This approach separates the content from its verification data, making multi-signature schemes and metadata updates more efficient.
• Multi-Signature Root of Trust: TUF establishes a root of trust (root.json) that is protected by multiple keys, often held by different individuals or even organizations, potentially geographically distributed. This multi-signature requirement makes compromise extremely difficult and ensures the integrity of the most critical part of the system.
• Role-Based Key Management and Delegation: Keys are assigned to specific roles (e.g., targets, snapshot, timestamp), each with different responsibilities and revocation policies. This hierarchy allows more frequently used keys to be more easily revoked without compromising the entire system.
• Version Numbers for Rollback Protection: TUF metadata files include version numbers. By verifying these, consumers can ensure they are always receiving the latest intended version of a package, effectively preventing rollback attacks where an attacker might try to reintroduce an older, vulnerable version.
• Timestamps for Freshness and Replay Attack Prevention: A timestamp role (timestamp.json) provides a "heartbeat" for the repository. This metadata file, updated frequently, includes a hash of the latest snapshot and a current timestamp. Consumers check this timestamp against a configured window, ensuring the metadata they are receiving is fresh and preventing replay attacks with outdated but still valid metadata.
• Internal Repository Control: Organizations can establish their own internal TUF repositories. This allows them to ingest, verify, and re-sign packages from upstream sources, ensuring that only artifacts meeting their internal security policies (e.g., SBOMs checked, CVE scans passed) are deployed into production environments. This grants significant control over the ingested images and reduces reliance on the security practices of external upstream repositories.
• Secure Recovery: In the event of a CVE or other security incident, TUF's rapid update and verification mechanism allows organizations to quickly remove compromised images from their internal repository. Consumers will then cease to trust and use these images within a defined timeframe, enabling a swift and secure recovery process.

These mechanisms collectively provide a robust framework that goes far beyond simple code signing, offering a comprehensive solution for maintaining trust and integrity across the entire software supply chain.

Technical Deep Dive

▶ Watch: Introducing the demo application: Personalized bill invoice service (4:00)

TUF's power lies in its structured approach to metadata management and cryptographic verification, organized into distinct roles and files. Let's break down the technical components:

Content Integrity with Detached Signatures

At its most basic, TUF aims to secure a software package (referred to as a "target"). Instead of embedding a signature directly within the package, which would alter its hash and complicate multi-signature scenarios, TUF employs detached signatures.

1. Target File: The actual software package (e.g., nginx.tar.gz).
2. Target Metadata File: A separate JSON file (e.g., foo.json) contains the cryptographic hash of the target file. This metadata file is then signed. This ensures the integrity of the target file itself.
3. Key Identification: To verify the signature on foo.json, the consumer needs the correct public key. This key is identified and associated with the foo role in another metadata file, higher up in the hierarchy.

The Root of Trust (root.json)

The highest level of trust in TUF is the root metadata file (root.json). This file defines the public keys for all other top-level roles (targets, snapshot, timestamp) and, crucially, for itself.

• Multi-Signature Trust: The root.json is protected by multi-signature trust. This means multiple, highly secured keys (e.g., hardware tokens, geographically dispersed) are required to sign any update to the root file. This makes it incredibly difficult for a single attacker to compromise the entire system.
• Infrequent Updates: Because of its critical importance and the high bar for signing, the root.json is updated very infrequently, typically only when keys need to be revoked or added. This allows the associated private keys to be kept offline or in highly secure environments.

Version Control and Rollback Protection (targets.json and snapshot.json)

To address rollback attacks, TUF introduces version numbers for its metadata:

1. Targets Metadata (targets.json): This file (or a delegated role-specific file like foo.json in the example) lists the hashes and version numbers of specific target files. When a new version of foo is released (e.g., foo 2.0), foo.json is updated with the new hash and its own version number increments (e.g., foo.json v2). This file is signed by its designated key (e.g., Key A).
2. Snapshot Metadata (snapshot.json): To prevent an attacker from replaying an old foo.json (even if it's signed and valid for foo 1.0), TUF introduces the snapshot.json. This file contains the hashes and version numbers of all other target metadata files (e.g., foo.json v2, bar.json v1). The snapshot.json itself has a version number and is signed by its own key. Every time any target metadata file changes, the snapshot.json is updated and resigned, ensuring that consumers always know the latest versions of all target metadata. The snapshot key is stored in the root.json.

Freshness and Replay Attack Prevention (timestamp.json)

Even with version numbers, an attacker could potentially replay an entire snapshot.json if it's still cryptographically valid but outdated. To counter this, TUF employs a timestamp role:

• Timestamp Metadata (timestamp.json): This file acts as a "heartbeat" for the repository. It contains the hash of the latest snapshot.json and a current timestamp. This file is signed by a dedicated timestamp key, whose public key is also defined in root.json.
• Efficiency: The timestamp.json is very small and efficient to resign and redistribute, allowing for frequent updates.
• Configurable Window: Consumers are configured with a time window (e.g., 24 hours) within which the timestamp must be valid. If the timestamp is outside this window, the metadata is considered stale, forcing the consumer to request fresh metadata. This mechanism ensures that consumers are always operating with the most current information and prevents replay attacks of even the latest snapshot.json if it's no longer fresh.

Internal TUF Repositories and Policy Enforcement

A critical application of TUF is in establishing internal, organization-controlled repositories. As Marina Moore outlined, this allows an organization to:

1. Define Requirements: Before an image (e.g., Nginx) enters production, an organization can enforce checks like SBOM generation, CVE scanning, and external signature verification.
2. Ingest and Re-sign: Once these requirements are met, the image is ingested into the organization's own TUF repository. The organization then signs the image and its associated metadata with its own TUF keys.
3. Controlled Distribution: Only images verified and signed by the internal TUF repository are allowed to be installed in the production environment. This ensures that the organization has full control over the trusted state of its software supply chain, independent of upstream providers.

The TUF Project

The TUF project, housed within the CNCF, consists of:

• Specification: A detailed document outlining the rules and secure practices for implementing TUF.
• Implementations: Libraries and tools in various programming languages, including Python, Go, Rust, PHP, and Java, enabling developers to integrate TUF into their systems.
• Deployments: TUF is used in critical services across various industries, including the "call service" example presented by Kairo. A notable implementation discussed is r-tuf, a repository service designed to simplify TUF deployment using containers and Helm charts.

Demo / Proof of Concept

▶ Watch: What is TUF? Core principles and compromise resilience (4:50)

Kairo De Araujo presented a practical demonstration using a "call service" – a critical Kubernetes-deployed service that generates personalized "bin voice" invoices. This service, being critical, required robust security, which TUF provided. The demonstration utilized r-tuf, a TUF repository service that simplifies the deployment and management of TUF metadata.

The demo focused on two primary use cases: securing supply chain provenance and managing container images.

Securing Provenance and Attestations

The "call service" generates and relies on various artifacts:

• Provenance: SBOMs, attestations (e.g., build attestations), and build artifacts.
• Content: User-generated "bin messages."

These artifacts were stored in GitHub releases. The goal was to ensure that a user downloading these attestations would receive only trusted versions.

Attack Scenario 1: Malicious Attestation Replacement

• The Attack: An attacker gains access to the GitHub releases repository (e.g., via a vulnerability or misconfiguration) and replaces a legitimate build attestation with a malicious one.
• Without TUF: If a user simply uses wget to download the attestation, they would receive the malicious file, unaware of the compromise.
• With TUF: When the user attempts to download the attestation using a TUF client, the client detects that the attestation is not trusted according to the configured TUF metadata. The download is blocked, protecting the user from the malicious artifact. This highlights TUF's ability to prevent tampering with critical supply chain metadata.

Securing Container Images (Nginx Example)

The demonstration then shifted to securing a critical third-party dependency: the Nginx container image (version 1.25.5). The process showcased how TUF manages updates and ensures policy compliance before deployment.

Scenario: Updating Nginx from 1.25.5 to a Newer Version

1. Initial State: The organization uses Nginx 1.25.5, which is currently trusted within the TUF repository.
2. Attempted Deployment of New Version (Pre-TUF Integration): If someone tries to deploy a newer version of Nginx (e.g., 1.26.0) without it being properly added to the TUF metadata, the deployment is not allowed. The image cannot be pulled because it's not recognized as trusted by the TUF system.
3. Adding the New Artifact to TUF Metadata: The new Nginx version is added to the organization's TUF repository metadata. However, merely adding it isn't enough; it still requires a trusted signature.
4. Signing with Trusted Keys (e.g., Sixstore): After internal verifications (license checks, CVE scans, etc.) are complete and approved, a trusted entity (represented by Marina signing with a Sixstore key) signs the TUF metadata for the new Nginx version.
5. Authorized Deployment: Once signed, the deployment of the new, authorized Nginx version (1.26.0) is successful. The TUF client verifies the signature and the metadata, allowing the image to be pulled and deployed.
6. Revoking a Malicious Version: Imagine Nginx 1.25.5 is later discovered to be malicious or has a critical vulnerability that requires immediate deprecation.

• Attempted Deletion from Metadata (Unsigned): Simply deleting the entry for 1.25.5 from the TUF metadata is not enough. Like adding, deleting also requires a signature.
• Signed Revocation: A trusted entity (Marina, again, signing) signs the updated TUF metadata, explicitly revoking trust in Nginx 1.25.5.
• Deployment Blocked: After this signed revocation, any attempt to deploy Nginx 1.25.5 is immediately blocked. The TUF client no longer trusts that version, demonstrating TUF's ability to enforce rapid, secure recovery.

The demo vividly illustrated how TUF, combined with tools like r-tuf and Sixstore, provides granular control over software artifacts, preventing both malicious injection and the use of outdated or compromised versions.

Defensive Implications

▶ Watch: How TUF ensures content integrity with detached signatures (6:00)

TUF offers several profound defensive implications for organizations aiming to fortify their software supply chains:

1. Assured Verification of Checks: By integrating TUF into build and release pipelines, organizations can ensure that critical security checks (e.g., SBOM generation, vulnerability scanning, static code analysis) have been performed and attested to. The act of signing TUF metadata becomes an attestation that these requirements have been met, providing cryptographic assurance.
2. Granular Control Over Ingested Images: TUF empowers organizations to establish their own internal trust boundaries. Instead of blindly trusting upstream open-source repositories, they can ingest, verify, and re-sign packages within their own TUF repository. This allows them to enforce custom security policies and ensure that only images meeting their specific standards (e.g., no known CVEs, approved licenses) are deployed to production. This is crucial for managing third-party dependencies.
3. Secure and Rapid Recovery from CVEs: One of TUF's most significant advantages is its ability to facilitate swift and secure recovery. If a CVE is discovered in a deployed package, organizations can update their internal TUF metadata to revoke trust in the vulnerable version. Due to TUF's freshness mechanisms (timestamps), consumers will quickly learn of the update and cease using the compromised image within a configurable timeframe, mitigating the attack surface.
4. Single, Strong Root of Trust: TUF provides a single, cryptographically strong root of trust for all software artifacts an organization consumes. This highly secured root.json, protected by multi-signatures, anchors the entire trust hierarchy, simplifying trust management and reducing the attack surface compared to managing numerous disparate trust anchors.
5. Integration with Existing Workflows and Tools:

• GitOps (Argo CD, Flux): While direct TUF client integration into GitOps tools like Argo CD or Flux is an ongoing area of development, organizations can implement TUF verification before artifacts reach the final deployment pipeline. This "shift-left" approach ensures that only TUF-verified images are pushed to Git repositories that GitOps tools consume. The TUF project actively collaborates with these projects to build more native support.
• Build Pipelines (GitHub Actions): TUF signing and verification steps can be integrated directly into CI/CD pipelines (e.g., GitHub Actions). This ensures that developers cannot inadvertently ship unsigned or partially signed artifacts. Verification can be performed at multiple stages, providing immediate feedback if signing requirements are not met.
• Richer Attestations (in-toto): For more complex verification needs, such as ensuring proper static code analysis reports (e.g., SonarQube) or detailed build provenance, TUF's sister project, in-toto, can be leveraged. In-toto provides a generic format for rich attestations, detailing inputs, outputs, and outcomes of supply chain steps. These in-toto attestations can then be securely distributed and verified using TUF, allowing for policy enforcement (e.g., "don't deploy if code coverage is below X%").
• SLSA Compatibility: TUF complements frameworks like SLSA (Supply-chain Levels for Software Artifacts). SLSA focuses on build provenance and attestation formats. TUF then provides the secure distribution mechanism for SLSA attestations and the associated software, ensuring that consumers receive the correct, fresh SLSA attestations alongside the software itself, preventing rollback or tampering of the provenance data.
• Flexible Storage: TUF is agnostic to where artifacts are stored. While r-tuf can manage metadata using PVC volumes or S3 buckets, artifacts themselves can reside in OCI registries, S3, or other storage solutions. This flexibility allows integration into diverse existing infrastructure.

1. Kubernetes Admission Controllers: A future defensive implication, as discussed in the Q&A, is the potential for Kubernetes admission controllers to enforce TUF verification at the cluster level. Such a controller could deny admission for any container image that is not TUF-verified and trusted by the organization's policies, providing a powerful gatekeeper for production environments.

By adopting TUF, organizations can move from a reactive security posture to a proactive, cryptographically enforced trust model for their software supply chains, significantly raising the bar for attackers.

Key Takeaways

• TUF addresses critical software supply chain vulnerabilities: It goes beyond simple code signing to prevent sophisticated attacks like rollback, replay, and arbitrary signature acceptance.
• Compromise Resilience is Core: TUF's multi-layered architecture ensures that a single point of compromise (e.g., a key, repository, or developer account) is insufficient to enable an attack, allowing for secure recovery.
• Freshness and Versioning are Key Protections: Through version numbers on metadata and frequently updated timestamps, TUF guarantees that consumers always receive the latest, intended, and untampered versions of software and associated metadata.
• Empowers Internal Supply Chain Control: Organizations can establish their own TUF repositories to enforce internal security policies (SBOM checks, CVE scans) and re-sign artifacts, gaining full control over what enters their production environments.
• Complements Other Security Frameworks: TUF integrates well with and enhances other supply chain security initiatives like in-toto (for richer attestations) and SLSA (for secure distribution of provenance data).
• Practical Implementation with r-tuf: Tools like r-tuf simplify the deployment and management of TUF, making it accessible for securing critical services and container images in Kubernetes environments.

About the Speaker(s)

Marina Moore is a Research Scientist at Edera and a prominent figure in the CNCF security community. She serves as one of the co-chairs of CNCF's TAG Security and is a dedicated maintainer of The Update Framework (TUF) project. Her expertise lies in enhancing the security and integrity of software distribution systems.

Kairo De Araujo is an Open Source Software Engineer and an independent contributor to critical security projects. He is a maintainer of both The Update Framework (TUF) and in-toto, demonstrating his commitment to advancing software supply chain security standards and implementations.