The Immediate and Lasting Benefits of TAG Security Assessments - Brandt Keller & Ben Hirschberg

The Immediate and Lasting Benefits of TAG Security Assessments

Speakers: Brandt Keller, Open Source Maintainer (Air-gapped Kubernetes), Tech Lead for TAG Security; Ben Hirschberg, Maintainer of Cubescape, CTO at Armo

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=Mbk6FY_9FKM

Overview

This talk, presented by Brandt Keller and Ben Hirschberg at KubeCon EU, delves into the critical role and tangible benefits of security assessments conducted by the Technical Advisory Group for Security (TAG Security) within the Cloud Native Computing Foundation (CNCF). The speakers highlight how these assessments serve as a vital mechanism for improving the security posture of cloud-native projects, fostering greater end-user confidence, and standardizing security practices across a rapidly expanding ecosystem. Keller, representing TAG Security, outlines the group's mission and the different assessment types, while Hirschberg provides a compelling case study from Cubescape, an incubating CNCF project, illustrating the immediate and lasting value derived from undergoing a self-assessment.

The core message emphasizes that security assessments are not merely bureaucratic hurdles but essential processes that drive genuine security enhancements, even for projects developed by seasoned security professionals. By establishing clear baselines, providing structured review processes, and encouraging proactive security measures, TAG Security aims to scale security efforts across the diverse landscape of CNCF projects. This ultimately contributes to a more secure and trustworthy cloud-native ecosystem, benefiting both project maintainers struggling with limited security resources and end-users seeking reliable, secure solutions.

Background

▶ Watch: Welcome, speakers, and TAG Security's role (0:00)

The CNCF landscape is vast and constantly evolving, with hundreds of projects at various stages of maturity and with differing levels of security expertise among their maintainers. Some projects are inherently security-focused, boasting dedicated security personnel, while others may lack specialized security resources. This disparity creates a significant challenge for ensuring a consistent and robust security baseline across the entire cloud-native ecosystem. Recognizing this, the Technical Advisory Group for Security (TAG Security) was established under the Technical Oversight Committee (TOC) within the CNCF.

TAG Security serves several crucial functions designed to address these challenges. Primarily, it acts as an advisory body, guiding projects on security best practices and standard processes. This includes defining security expectations for projects navigating the CNCF graduation process—moving from sandbox to incubating, and then to graduated status. For projects undergoing graduation, TAG Security helps define what a "secure" project entails, providing a much-needed baseline and clear documentation.

A key component of TAG Security's work involves security assessments. These are categorized into two main types:

1. Self-Assessment: This is a self-service layer, where projects utilize a template provided by TAG Security to evaluate their own security posture. The goal is to establish a minimum baseline, prompting projects to proactively address security considerations and document their efforts. This documentation is also valuable for consumers of these projects, who might look for such attestations of security.
2. Joint Assessment: This is a more comprehensive and collaborative experience, typically for projects moving from incubating to graduated status, especially those that are security-related. Volunteers and contributors from TAG Security work directly with project teams to understand their architecture, identify potential vulnerabilities based on past compromises documented in their repository, and ensure they meet minimum security criteria.

The rationale behind these assessments is multifaceted. Firstly, they aim to lower barriers to accessibility and increase diversity in contribution by providing structured avenues for security personnel to get involved. Secondly, they are crucial for scaling security across the landscape. Instead of each project independently struggling with security, TAG Security provides a centralized, collaborative approach to lift the overall security baseline. This offers a better return on investment for maintainers, who are often time-constrained, by providing clear, concise expectations and a documented attestation of their security posture. Such documentation is vital for outlining design decisions, privilege requirements, and known constraints upfront, which can significantly enhance adoption by end-users.

Key Findings

▶ Watch: Understanding the self-assessment process and baseline (3:00)

The central finding presented in the talk is the profound, often unexpected, value of TAG Security assessments, particularly the self-assessment process, even for projects developed by experienced security professionals. Ben Hirschberg's account of Cubescape's journey through the self-assessment serves as a powerful testament to this.

Initially, the Cubescape team, comprising maintainers with extensive backgrounds in the security industry (Hirschberg himself with over 20 years), viewed the self-assessment as a potentially bureaucratic and tedious requirement. They anticipated it would be an "easy" task, something to be left until the last minute, given their inherent expertise in security. However, the experience proved to be quite the opposite. The structured template and the need to document and re-evaluate seemingly "obvious" aspects of their project's security forced them into a rigorous process of self-reflection.

The most significant key finding from Cubescape's self-assessment was the discovery of a critical security vulnerability: unencrypted and unauthenticated communication between two internal components of the Cubescape operator—its aggregated API server extension and its storage component. This oversight meant that sensitive data, including vulnerability scans, configuration details, and Software Bill of Materials (SBOMs), was being transmitted without the protection of TLS (Transport Layer Security). This was a direct consequence of a development-time decision to temporarily skip TLS for ease of implementation, which was subsequently forgotten and never re-enabled.

Beyond this specific technical flaw, the broader findings highlight:

• The necessity of external validation and structured review: Even deeply knowledgeable teams can overlook critical issues without a formal, external-facing review process.
• Improved documentation and architectural clarity: The assessment compelled Cubescape to create detailed diagrams and documentation, leading to a clearer understanding of their own architecture and security considerations.
• Identification of "known unknowns" and constraints: The process helped articulate why certain architectural decisions were made, what security constraints existed, and what mitigation strategies were in place or needed.
• Enhanced project quality and maintainability: The effort, though initially perceived as a burden, ultimately led to a more robust, secure, and well-documented project, benefiting both maintainers and consumers.

In essence, the talk underscores that these assessments are not just about compliance, but about instigating a valuable internal audit that can uncover fundamental security gaps, reinforce best practices, and significantly mature a project's security posture.

Technical Deep Dive

▶ Watch: Joint assessment for graduating projects explained (4:10)

Cubescape, the project highlighted in Ben Hirschberg's case study, is an open-source Kubernetes security platform. It began as a command-line interface (CLI) tool designed for assessing the security and compliance of Kubernetes clusters, scanning configurations, YAML files, and Helm charts, with capabilities for integration into CI/CD pipelines. Over time, it evolved into a comprehensive platform, installable as an operator within a Kubernetes cluster, offering a "one-stop shop" for various security features.

Key features of Cubescape include:

• Vulnerability Scanning: Identifies known vulnerabilities in images and deployed components.
• Configuration Scanning: Checks Kubernetes configurations against best practices and compliance standards.
• Policy Enforcement: Applies security policies across the cluster.
• Runtime Incident Detection: Detects active attacks and suspicious behavior within the cluster, moving beyond just static posture issues.

The architecture of the Cubescape operator, as described by Hirschberg, involves multiple microservices:

• Operator Component: Manages all other Cubescape components within the Kubernetes environment.
• kris (Configuration Scanner): A microservice responsible for performing configuration scans by interacting with the Kubernetes API server.
• kubulnar (Vulnerability Scanner): A microservice dedicated to vulnerability scanning, designed to be scaled independently.
• Node Agents: These agents run on cluster nodes, primarily for eBPF (extended Berkeley Packet Filter) data collection. This enriches vulnerability and configuration scanning data and is crucial for detecting runtime incidents.
• Synchronizer: An interesting component used for synchronizing objects created in the cluster to external systems.

A significant design principle for Cubescape was to tightly integrate with the Kubernetes ecosystem, meaning all object creation and API communication were intended to go through the Kubernetes API server. This design choice aimed to ensure that any existing system communicating with Cubescape could tap into its data and configurations via standard Kubernetes mechanisms.

However, this design encountered a technical challenge, specifically related to Software Bill of Materials (SBOMs) and image scan data. SBOMs can be very large, often ranging from 50 to 100 megabytes. The standard Kubernetes API server is not designed to handle objects of this size efficiently, potentially leading to performance issues or failures.

To circumvent this limitation while maintaining Kubernetes-native interaction, Cubescape leveraged a lesser-known Kubernetes feature: Aggregated API Server Extensions. This feature allows developers to create a separate, custom API server within the Kubernetes cluster, specifically serving particular types of objects. The main Kubernetes API server then reroutes requests for these custom objects to the aggregated extension. From an external user's perspective, these custom objects appear as standard Kubernetes API objects, but their implementation and storage reside in a separate component. In Cubescape's architecture, this separate component became its storage component, responsible for handling the large SBOM and scan data.

The communication between the main Kubernetes API server (acting as a proxy) and Cubescape's aggregated storage component, like any other API extension, is intended to be secured. This typically involves mutual TLS (mTLS), requiring certificate requests and robust security measures to ensure authenticated and encrypted communication.

The critical security flaw uncovered during Cubescape's self-assessment stemmed from the implementation of this aggregated API server extension. During the initial development phase, the team encountered challenges with implementing mTLS for the communication between the aggregated API server and their storage component. To expedite development, they temporarily enabled a flag to skip TLS, allowing unencrypted communication. Regrettably, this temporary measure was never reverted. As a result, up until the self-assessment, the communication channel between Cubescape's API server extension and its storage component was unauthenticated and unencrypted.

This oversight was particularly egregious because the storage component houses highly sensitive data: all vulnerability data, configuration scans, SBOMs, and other critical security configurations collected and managed by Cubescape. The lack of TLS meant this data was vulnerable to interception and tampering, representing a significant security risk for a project specifically designed to enhance Kubernetes security. The self-assessment process forced the team to re-examine these foundational security assumptions, leading to the identification and prompt remediation of this critical vulnerability by enabling TLS in a subsequent Helm chart release.

Demo / Proof of Concept

▶ Watch: Cubescape's self-assessment journey begins (8:00)

The talk did not feature a live technical demonstration or a proof of concept in the traditional sense. Instead, Ben Hirschberg presented a compelling real-world case study of how Cubescape, an incubating CNCF project, benefited directly from the TAG Security self-assessment process.

Hirschberg’s narrative served as an anecdotal proof of concept for the value of these assessments. He detailed the specific security flaw discovered within Cubescape: the oversight of unencrypted communication (lack of TLS) between its aggregated API server extension and its storage component. This issue was not theoretical; it was a live vulnerability present in the deployed software. The self-assessment acted as the mechanism that brought this critical flaw to light.

The "demonstration" of the assessment's effectiveness was in the discovery and subsequent remediation of this vulnerability. Hirschberg explained how the team, despite their security expertise, had overlooked the disabled TLS flag. The structured review process compelled them to revisit this architectural detail, leading to the realization of the security gap. He explicitly stated, "we just found out that we never turned this on, never enabled this," and that they "obviously... handled it and we released the version... of the Helm chart enabling this and solved the problem as of today."

Therefore, while there was no live code execution or exploit demonstration, the entire presentation, particularly the Cubescape segment, functioned as a powerful real-world example of how TAG Security assessments directly lead to tangible security improvements and bug fixes, even for mature and security-focused projects.

Defensive Implications

▶ Watch: Overview of Cubescape's security assessment tool (8:50)

The insights from this talk offer several crucial defensive implications for various stakeholders within the cloud-native ecosystem:

1. For Project Maintainers and Developers:

• Embrace Self-Assessments Proactively: Do not view TAG Security self-assessments as bureaucratic overhead. Instead, treat them as a valuable, structured internal audit opportunity. Engaging proactively, rather than waiting until the last minute, can prevent critical issues from lingering.
• Continuous Security Review: The ideal state is for security assessments to become a regular, periodic review (e.g., annually) rather than a one-off event. Security posture is not static, and continuous re-evaluation is essential.
• Document Everything (Especially Constraints): Clearly document architectural decisions, security considerations, and known constraints or trade-offs. If a part of the architecture is "known insecure" but unavoidable, document the rationale and implemented mitigations. This transparency is vital for end-users and future maintainers.
• Prioritize Fundamentals: The Cubescape example highlights that even experienced security developers can overlook basic security hygiene, like enabling TLS. Fundamental security controls must be rigorously checked and re-checked.
• Define Security Roles and Processes: Establish clear roles (e.g., security manager within GitHub organizations) and processes for handling security reports, advisories, and vulnerabilities. Projects need to know "who to go to" and "what to do" when a security incident occurs.

1. For End-Users and Cloud-Native Adopters:

• Leverage Assessment Information: When evaluating cloud-native projects, look for those that have undergone TAG Security assessments. This indicates a higher level of maturity, transparency, and a commitment to security.
• Understand Project Constraints: Pay attention to documented security constraints and requirements. This information helps in making informed decisions about project adoption and potential risks in specific deployment environments.
• Demand Transparency: Encourage projects to openly share their security posture and assessment outcomes, fostering a more secure supply chain.

1. For the Cloud-Native Community and TAG Security Contributors:

• Contribute to Scaling Security: The growth of the CNCF landscape necessitates scaling security efforts. This involves contributing to TAG Security by helping automate assessment processes, reviewing assessments, and refining the self-assessment templates.
• Broaden the Baseline: Efforts to make the assessment templates more generic and integrate them with broader initiatives like OpenSSF are critical for extending their impact beyond the CNCF.
• Foster Contributor Paths: Actively recruit and mentor security professionals who wish to contribute to open-source projects, even if they lack specific language or architecture skills. Security assessments provide a structured entry point for such contributions.
• Collaborate and Share Knowledge: Encourage cross-project collaboration and knowledge sharing regarding security challenges and solutions. Many projects face similar issues, and shared experiences can accelerate improvements across the ecosystem.

In essence, the talk underscores that effective defense in the cloud-native space requires a multi-pronged approach: proactive self-assessment, transparent documentation, continuous review, robust foundational security, and a collaborative community effort to raise the overall security baseline.

Key Takeaways

• Self-assessments are invaluable, even for security-focused projects: The Cubescape case study clearly demonstrates that structured security self-assessments can uncover critical vulnerabilities and architectural flaws, even when conducted by highly experienced security professionals.
• TAG Security standardizes and scales cloud-native security: The Technical Advisory Group for Security provides essential guidance, templates, and review processes that help lift the overall security baseline across the diverse and rapidly growing CNCF landscape, benefiting projects with varying levels of security expertise.
• Proactive security posture enhances adoption and trust: Engaging with security assessments early and documenting security decisions, constraints, and mitigations upfront builds greater end-user confidence and streamlines project adoption.
• Fundamental security hygiene is paramount: The discovery of an unencrypted communication channel in Cubescape highlights that basic security controls like TLS must be rigorously implemented and verified, as oversights can lead to significant vulnerabilities.
• Continuous security review is essential: Security is not a one-time achievement. Projects should aim for regular, periodic security reviews to adapt to evolving threats and maintain a strong security posture.
• Community involvement is crucial for ecosystem-wide security: Contributing to TAG Security, participating in assessments, and advocating for security roles in open-source projects are vital steps towards enhancing the security of the entire cloud-native ecosystem.

About the Speaker(s)

Brandt Keller is an open-source maintainer with a specific focus on air-gapped Kubernetes environments. He also serves as a Tech Lead for TAG Security, playing a pivotal role in guiding and implementing the security assessment processes within the CNCF. His work involves defining baselines, documenting expectations, and advising projects on security best practices.

Ben Hirschberg is a maintainer of the Cubescape project, an incubating CNCF project that provides an open-source Kubernetes security platform. He is also the CTO at Armo, a cloud security startup. With over 20 years of experience in the security industry, Ben brings a deep understanding of security challenges and solutions, as demonstrated by his candid sharing of Cubescape's journey through the TAG Security self-assessment.