Public Technical Oversight Committee (TOC) Meeting - Moderated by Chris Aniszczyk

Public Technical Oversight Committee (TOC) Meeting - Moderated by Chris Aniszczyk

Speakers: Moderated by Chris Aniszczyk, Karina Angel (Chair, Red Hat), Alex Kerop (Akami), Kevin Juan, Jeremy Rickard (Co-chair SIG release Kubernetes, Microsoft), Dims (Kubernetes community, AWS), Katie Gamji (Apple, TAG/TAB), Facil (ISTIO community, Ericsson), Ricardo (CERN, TOC, End User TAB), and a newly elected TOC shadow.

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=V7HbBO_umOU

Overview

This KubeCon EU session provided a unique opportunity for attendees to engage directly with members of the Cloud Native Computing Foundation (CNCF) Technical Oversight Committee (TOC) in an "Ask Me Anything" (AMA) format. As one of the three top-level governing bodies of the CNCF, the TOC plays a crucial role in stewarding projects and defining the technical vision for the entire cloud-native ecosystem. The discussion, moderated by Chris Aniszczyk, delved into the committee's responsibilities, the major challenges facing the CNCF landscape, and the strategic initiatives being undertaken to foster project health, sustainability, and innovation in the coming years.

The core of the talk centered on the TOC's efforts to address critical technical gaps, scale community expertise, and streamline the project maturity process. Speakers highlighted the restructuring of Technical Advisory Groups (TAGs) and the introduction of time-scoped "initiatives" designed to tackle specific challenges like multicluster management, cost optimization, and robust security practices. This session was not a traditional presentation of a specific security vulnerability or tool, but rather a transparent look into the high-level governance and strategic planning that underpins the security and stability of the vast collection of CNCF projects, providing invaluable insights for maintainers, end-users, and contributors alike.

The discussion underscored the CNCF's evolution, now approaching its tenth year, balancing the imperative for innovation with the equally vital need to ensure the long-term health and sustainability of its growing portfolio of projects. This includes addressing contributor burnout, improving project readiness for graduation, and enhancing communication channels between the TOC and the broader cloud-native community. The insights shared are critical for anyone invested in the future direction and technical integrity of the cloud-native landscape.

Background

▶ Watch: Welcome and introduction of the TOC members (0:00)

The Cloud Native Computing Foundation (CNCF) serves as the vendor-neutral home for many of the fastest-growing open-source projects, including Kubernetes, Prometheus, and Envoy. Its mission is to make cloud-native computing ubiquitous, supporting the development and adoption of a diverse set of technologies. At the apex of its governance structure are three primary bodies: the Governing Board, the End User Community, and the Technical Oversight Committee (TOC). The TOC is specifically tasked with providing technical oversight for all projects within the CNCF ecosystem. This encompasses a broad range of responsibilities, including licensing, trademark issues, and, most importantly, ensuring the technical viability and sustainability of projects for end-users.

Projects within the CNCF typically progress through a maturity lifecycle: Sandbox, Incubating, and Graduated. Each stage involves a rigorous due diligence process where the TOC evaluates a project's technical specifics, community health, governance, release transparency, and security posture. This structured progression is designed to ensure that projects meet high standards before receiving the full endorsement and resources of the CNCF. However, as the ecosystem has grown rapidly, challenges have emerged. The TOC has observed issues such as project unreadiness for advancement, maintainer burnout, and a need to scale the technical expertise available to guide projects. This background sets the stage for the TOC's current strategic focus on restructuring its advisory groups and fostering new initiatives to address these systemic challenges and prepare the CNCF for its next decade of growth.

Prior to this meeting, the TOC engaged in internal exercises, including a Maintainer Summit workshop, to define its vision and direction for the upcoming year. These efforts aimed to identify critical gaps within the ecosystem and brainstorm concrete solutions. The workshop, as described by Katie Gamji and Dims, involved breaking down participants into tables focused on areas like developer experience, operational resiliency, testing, and security, to generate ideas for new initiatives. This collaborative approach highlights the TOC's commitment to community-driven solutions and its recognition that the diverse expertise within the CNCF community is essential for tackling complex, cross-cutting technical challenges. The outcomes of these discussions form the bedrock of the "Key Findings" and "Technical Deep Dive" sections of this article.

Key Findings

▶ Watch: Identifying key technical challenges and ecosystem gaps (3:00)

The TOC articulated several key findings and strategic directions during the session, reflecting a comprehensive assessment of the CNCF ecosystem's current state and future needs. These findings can be broadly categorized into identified technical gaps, internal restructuring efforts, and challenges within the project maturity process.

Firstly, three major technical gaps were highlighted as critical areas requiring focused attention and cross-TAG collaboration:

1. Multicluster Management and Observability: This domain presents significant challenges, particularly in cross-provider environments. The TOC aims to bring together existing TAGs like TAG App Delivery and TAG Observability to address these complexities.
2. Cost Management and Sustainability: Beyond mere cost spending, there's a growing emphasis on managing the carbon footprint of cloud-native deployments. This area necessitates collaboration between groups focused on financial and environmental sustainability.
3. Infrastructure Provisioning and Secret Management: A long-standing gap in the ecosystem, the need for fully open-source tooling in these areas remains acute. The TOC seeks to foster initiatives that will consolidate efforts to provide robust, open solutions for managing infrastructure and sensitive data.

Secondly, the TOC recognized the need to scale its technical expertise and address issues like attrition and burnout within the community. This led to a significant internal restructuring:

• TAG Restructuring: The existing Technical Advisory Groups (TAGs) are being reorganized to provide better guidance to projects as they navigate the maturity levels (Sandbox, Incubation, Graduation). This reboot aims to enhance the depth and breadth of technical mentorship available.
• New Initiatives: A novel concept of time-scoped initiatives with concrete deliverables and specific domain focuses has been introduced. These initiatives are designed to be more agile and targeted than broader TAG mandates, allowing for focused collaboration on specific problems. Examples discussed include standardizing cold recovery/disaster recovery best practices, addressing challenges in exposing industrial devices like PLCs for automotive sectors, and improving security automation for self and joint assessments.

Finally, the TOC identified recurring challenges in the project maturity process:

• Project Readiness: A significant backlog of projects seeking incubation or graduation often stems from their lack of readiness, particularly in areas where technical guidance could have prepared them better. This underscores the importance of the TAG restructuring.
• Security First Approach: Projects frequently do not prioritize security from the outset, leading to deficiencies in their security hygiene and planning. The TOC emphasizes the need for projects to integrate security considerations early in their development lifecycle.
• Maintainer Responsiveness: A critical bottleneck in the due diligence process is sometimes the lack of responsiveness from project maintainers after submitting an application. The TOC stressed that the process is a collaboration requiring active engagement.
• Community vs. Company Balance: Projects, especially those originating from commercial entities, sometimes struggle to clearly delineate community governance from company product interests, leading to governance gaps that the TOC helps to identify and rectify.
• Time Zone and Language Issues: The global nature of the community presents challenges for participation in meetings and effective communication, indicating a need for more inclusive engagement strategies.

These findings collectively highlight the TOC's proactive approach to evolving the CNCF ecosystem, ensuring its continued health, security, and relevance in the rapidly changing cloud-native landscape.

Technical Deep Dive

▶ Watch: Balancing innovation with existing project sustainability and health (4:20)

The technical discussions, while not centered on a single vulnerability, provided a deep dive into the architectural and governance mechanisms that underpin the health and security of the CNCF ecosystem. The TOC's initiatives are fundamentally technical, aiming to standardize, integrate, and improve the operational resilience of cloud-native projects.

The TAG Restructuring is a critical technical governance shift. By reorganizing TAGs and actively seeking technical leads and chairs, the TOC aims to amplify the community's collective expertise. This isn't merely an administrative change; it's about channeling specialized knowledge to projects. For instance, a project aiming for graduation might lack deep expertise in supply chain security or observability metrics. The revitalized TAGs are intended to provide this targeted guidance, helping projects implement best practices for telemetry, tracing, logging, and secure software development lifecycle (SSDLC) processes. This direct engagement helps ensure that projects are not just functional but also architecturally sound and maintainable in the long term.

The introduction of time-scoped initiatives represents a more agile approach to tackling specific technical problems. These initiatives are designed to produce concrete deliverables within a defined timeframe, contrasting with the broader, ongoing mandates of TAGs. Several examples from the Maintainer Summit workshop illustrate their technical scope:

• Cold Recovery/Disaster Recovery Standardization: This initiative aims to define and standardize best practices for recovering cloud-native applications and infrastructure after catastrophic failures. Technically, this involves establishing common patterns for backup and restore operations, data replication strategies, recovery time objective (RTO) and recovery point objective (RPO) definitions, and the orchestration of complex application stacks across different availability zones or regions. This could lead to shared frameworks, playbooks, or even open-source tools that simplify disaster recovery for CNCF projects.
• Exposing Industrial Devices (PLCs): For sectors like automotive manufacturing, integrating legacy industrial control systems (such as Programmable Logic Controllers - PLCs) with cloud-native environments presents unique challenges. This initiative would explore technical solutions for secure and reliable communication between edge devices and the cloud, potentially leveraging projects like KubeEdge or OpenYurt. It involves defining API specifications, data models, and communication protocols suitable for low-latency, high-reliability industrial use cases, while ensuring operational technology (OT) security.
• Security Automation for Self and Joint Assessments: This initiative focuses on improving the efficiency and effectiveness of security evaluations for CNCF projects. Technically, this could involve developing automated tools or frameworks that integrate with CI/CD pipelines to perform checks against security best practices, dependency scanning, and vulnerability assessments. The goal is to make it easier for projects to conduct and maintain their security posture, perhaps by leveraging tools like OpenSSF Scorecard or integrating with existing CNCF security projects.
• Dependency Management and Identity Integration: A critical security initiative discussed was the integration of SPIFFE (Secure Production Identity Framework for Everyone) and SPIRE (SPIFFE Runtime Environment) into the ecosystem in a more fundamental way. SPIFFE provides a universal identity for workloads, while SPIRE implements the SPIFFE specification, enabling cryptographic identity for services across heterogeneous environments. Deep integration of SPIFFE/SPIRE would provide a standardized, robust mechanism for workload identity and authentication, enabling zero-trust architectures across the CNCF landscape. This involves defining how projects can leverage SPIFFE IDs, integrate with service mesh solutions, and establish secure communication channels without relying on traditional network-based security.
• Conformance Programs for Downstream Vendors: This initiative aims to provide practical guidelines and tools for projects to develop their own conformance test suites. This is crucial for ensuring that downstream vendor implementations of CNCF projects adhere to defined standards, preventing fragmentation and ensuring interoperability. Technically, this involves defining API conformance tests, behavioral tests, and performance benchmarks that can be run against different distributions or integrations of a project. The idea of standardizing a TestGrid-like system, similar to Kubernetes' own testing infrastructure, for broader CNCF project use, was also floated as a way to improve testing and validation across the ecosystem.

The emphasis on a "security first" approach for projects moving through the maturity levels is a non-negotiable technical requirement. This means projects must demonstrate robust security hygiene, including vulnerability management processes, secure coding practices, and regular security audits. The ongoing review of security assessments and audits, in collaboration with TAG Security, aims to streamline these requirements, ensuring they are comprehensive yet practical. The TOC's insistence on seeing these security measures "in action" rather than just documented in a governance.md file highlights a commitment to tangible technical security outcomes.

Overall, the technical deep dive reveals a sophisticated understanding by the TOC of the systemic challenges within the cloud-native space. Their proposed solutions, from structural changes in governance to targeted, technical initiatives, aim to build a more resilient, secure, and sustainable ecosystem for the next generation of cloud-native technologies.

Demo / Proof of Concept

▶ Watch: Announcing new time-scoped initiatives with concrete deliverables (7:00)

The session, being a live Public Technical Oversight Committee (TOC) meeting structured as an "Ask Me Anything" (AMA) with an expert panel, did not include any live demonstrations or proof-of-concept presentations. The focus was entirely on discussion, strategic planning, and community engagement regarding the governance and future direction of CNCF projects.

Defensive Implications

▶ Watch: Call for community to propose new initiatives and get involved (7:40)

While not a traditional security talk detailing specific exploits, the TOC meeting carried profound defensive implications for anyone operating within or contributing to the cloud-native ecosystem. The strategic direction outlined by the TOC directly impacts the security posture and resilience of CNCF projects and, by extension, the applications built upon them.

For project maintainers and contributors, the defensive message is clear: security must be a "first-class citizen" from inception. The TOC's observation that projects often fail to consider security early on highlights a critical vulnerability. Maintainers are implicitly urged to:

• Adopt a Security-First Mindset: Integrate security considerations into design, development, and operational phases, rather than treating it as an afterthought. This includes secure coding practices, threat modeling, and understanding potential attack vectors.
• Improve Security Hygiene: Implement robust practices for vulnerability management, dependency scanning, and regular security assessments. The mention of automating self and joint assessments as an initiative is a direct call for better tooling and processes in this area.
• Ensure Robust Governance and Release Processes: Transparent and well-defined release processes, coupled with strong community governance, are defensive mechanisms against supply chain attacks and unauthorized changes. Projects should clearly document their security policies and incident response plans.
• Be Responsive and Collaborative: Active engagement with the TOC during the due diligence process, especially regarding security questions, is crucial. This collaborative approach helps identify and remediate potential weaknesses before a project reaches wider adoption.
• Embrace Identity Management: The initiative to integrate SPIFFE/SPIRE more fundamentally into the ecosystem is a significant defensive step. Maintainers should explore how to leverage cryptographic workload identities to enforce zero-trust principles, reducing reliance on network-based controls and enhancing authentication and authorization for services.
• Participate in Conformance Programs: For projects with downstream vendors, establishing and participating in conformance programs ensures that different implementations adhere to security standards, preventing fragmentation that could introduce vulnerabilities.

For end-users and organizations deploying cloud-native technologies, the TOC's work provides a critical layer of assurance and guidance for defensive strategies:

• Leverage Matured Projects: Prioritizing the adoption of CNCF projects that have reached the Incubating or Graduated stages, as they have undergone rigorous TOC due diligence, including security reviews. This provides a baseline level of confidence in their security posture and sustainability.
• Monitor TOC Initiatives: Keep an eye on the progress of initiatives like disaster recovery standardization and secret management tooling. These efforts will directly contribute to more resilient and secure cloud-native deployments, offering blueprints or tools for improving operational defenses.
• Demand Robust Security from Vendors: Understand that the TOC's push for "security first" and conformance programs means end-users can and should expect a higher standard of security and adherence to best practices from their cloud-native vendors and service providers.
• Contribute to Security Efforts: The call for community involvement in TAGs and initiatives, particularly TAG Security, offers an avenue for end-users to contribute their operational security insights and help shape the defensive landscape of the CNCF.

In essence, the TOC's strategic vision is a long-term defensive play for the entire cloud-native ecosystem. By fostering project health, standardizing technical practices, and embedding security at every level of governance and development, the committee is working to build a more secure, resilient, and trustworthy foundation for the future of cloud computing.

Key Takeaways

• CNCF TOC's Evolving Role: The TOC is actively adapting its stewardship role to address the growing complexity and maturity of the cloud-native ecosystem, focusing on technical vision, project health, and sustainability as the CNCF approaches its 10th year.
• Three Critical Technical Gaps: The TOC identified multicluster management and observability, cost management and sustainability (including carbon footprint), and infrastructure provisioning and secret management tooling as key areas for focused community effort and cross-TAG collaboration.
• TAG Restructuring & New Initiatives: A major initiative is the restructuring of Technical Advisory Groups (TAGs) to scale technical expertise and provide better guidance to projects. This is complemented by new time-scoped initiatives for concrete deliverables, such as standardizing disaster recovery, integrating industrial devices, and enhancing security automation.
• "Security First" for Project Maturity: Projects seeking to advance through CNCF maturity levels are strongly urged to adopt a "security first" approach, demonstrating robust security hygiene, transparent release processes, and active engagement with security assessments and audits.
• Community-Driven Solutions: The success of the Maintainer Summit workshop and the call for community members to become TAG chairs or leads underscores the TOC's commitment to leveraging collective expertise and fostering collaboration for addressing ecosystem challenges.
• Focus on Operational Resilience: Beyond individual project health, the TOC's efforts—including initiatives around disaster recovery, identity management (SPIFFE/SPIRE), and conformance programs—are geared towards building a more resilient, interoperable, and secure cloud-native landscape for all users.

About the Speaker(s)

The session was moderated by Chris Aniszczyk, who opened the floor to the distinguished members of the CNCF Technical Oversight Committee. The panel included:

• Karina Angel: The current chair of the TOC, representing Red Hat.
• Alex Kerop: A TOC member working at Akamai.
• Kevin Juan: A TOC member involved in multiple projects.
• Jeremy Rickard: A co-chair of SIG release in Kubernetes, working at Microsoft.
• Dims: A prominent member of the Kubernetes community, representing AWS.
• Katie Gamji: Representing Apple and also part of the Technical Advisory Group (TAG).
• Facil: Part of the Istio community and working for Ericsson.
• Ricardo: A TOC member and also involved with the End User Technical Advisory Board, representing CERN.

The panel also acknowledged the presence of a newly elected TOC shadow, a new role introduced to help the TOC and provide development opportunities for future committee members. Collectively, these individuals represent a broad spectrum of expertise from leading technology companies, open-source projects, and research institutions, reflecting the diverse and collaborative nature of the CNCF TOC.