Kubernetes SIG Architecture Intro and Updates - John Belamaric, Google & Davanum Srinivas, AWS

Kubernetes SIG Architecture Intro and Updates

Speakers: John Belamaric, Google; Davanum Srinivas, AWS

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=oLZ2EjjKibw

Overview

This talk provides a comprehensive introduction and update on the pivotal role of the Kubernetes Special Interest Group (SIG) Architecture. Presented by co-chairs John Belamaric of Google and Davanum Srinivas (Dims) of AWS, the session elucidates the core responsibilities, processes, and ongoing initiatives of SIG Architecture within the broader Kubernetes project. It delves into how this SIG acts as the technical guardian of Kubernetes, ensuring its architectural integrity, long-term sustainability, and adherence to foundational principles like portability, extensibility, and generality.

The speakers highlight that SIG Architecture is not merely a technical review board but a crucial facilitator for the project's evolution, balancing innovation with stability and consistency. They emphasize its role in shaping the project's structure, defining conventions, and resolving inter-SIG disagreements. The discussion underscores the importance of a well-defined architectural body in an open-source project of Kubernetes' scale and impact, particularly as it continues to expand its applicability to new domains such as AI/ML workloads and diverse hardware environments.

Ultimately, the talk serves as both an educational resource for understanding Kubernetes governance and a call to action for community involvement. It makes a compelling case for why SIG Architecture's efforts are indispensable for maintaining the trust and reliability that millions of users and critical applications place in Kubernetes, ensuring that the platform remains robust, secure, and adaptable for years to come.

Background

▶ Watch: Welcome and SIG Architecture's core mission (0:00)

The Kubernetes project, a cornerstone of the cloud-native ecosystem, operates under a sophisticated governance model designed to manage its rapid growth and vast contributor base. At the highest level, the project is part of the Cloud Native Computing Foundation (CNCF), which itself is a foundation within the Linux Foundation (LF). Within Kubernetes, the Steering Committee oversees the overall health, sustainability, and advocacy of the project, focusing on non-technical aspects like community curation, resource allocation (e.g., cloud credits), and representation on the CNCF Governing Board.

Distinct from the Steering Committee, SIG Architecture is explicitly tasked with the technical architectural decisions and long-term technical vision of Kubernetes. Its mandate is to uphold the project's core technical goals: portability, general purpose applicability (now extending to AI/ML workloads), flexibility, and extensibility. This involves translating community values into the software's structure, recognizing the inherent link between organizational architecture and software architecture. As Dims Srinivas noted, "there's a close relationship between the architecture of your organization and the architecture of the software you produce."

The project is further organized into numerous Special Interest Groups (SIGs), each focusing on a specific area like Compute, Networking, Storage, UI, or Scalability. SIG Architecture plays a critical role in defining the scope and charters of these SIGs, ensuring minimal overlap and fostering collaboration. A notable example of SIG Architecture's influence is its decision to absorb the etcd project under the Kubernetes umbrella when etcd faced health and contributor challenges. This strategic move provided etcd with the governance, security, and process support it needed, allowing its core contributors to focus on code while benefiting from Kubernetes' established practices like the Enhancement Proposal (KEP) process.

In addition to SIGs, Kubernetes utilizes Working Groups, which are intentionally temporary and have defined exit criteria. These groups typically involve multiple SIGs collaborating on a specific, time-bound problem, with their outputs eventually integrated back into relevant SIGs. This structure prevents groups from becoming self-perpetuating and ensures that efforts remain focused and efficient. SIG Architecture's own charter outlines its scope, encompassing critical areas such as the conformance program, API conventions, deprecation policies, production readiness reviews, and the enhancement proposal process, all of which are vital for maintaining the project's consistency, quality, and evolutionary path.

Key Findings

▶ Watch: SIG Architecture's role in cross-SIG coordination (2:00)

The talk highlights several critical contributions and ongoing achievements of SIG Architecture that collectively ensure the robustness and future-proofing of Kubernetes:

• Robust Conformance Program: SIG Architecture has been instrumental in establishing and meticulously maintaining the Kubernetes conformance program. This program guarantees that any Kubernetes distribution, whether on-premises or in the cloud, provides a consistent set of APIs and behaviors. The speakers proudly announced that, after years of dedicated effort, the project closed significant gaps in API test coverage by Kubernetes versions 1.31 or 1.32. Now, a proactive approach is in place, requiring all new features and APIs to include conformance tests as they graduate through their alpha, beta, and General Availability (GA) stages, preventing future backlogs.
• Standardized API Conventions: To prevent a fragmented and inconsistent API landscape, SIG Architecture curates and enforces a set of API conventions. These conventions, refined over a decade of Kubernetes development, serve as institutional knowledge for designing new APIs, custom resources, and controllers. Adherence to these guidelines ensures ecosystem-wide compatibility, predictability, and better integration with existing tooling, helping developers avoid common pitfalls and past mistakes.
• Structured Enhancement Proposal (KEP) Process: The Kubernetes Enhancement Proposal (KEP) process, championed by SIG Architecture, transforms how new features are introduced. Instead of merely pushing code, contributors must articulate their proposals in detail, covering design, implementation milestones, potential impact on existing features and workloads, and rollback strategies. This "living document" approach ensures thoughtful design, community consensus, and adaptability as features evolve from alpha to beta.
• Clear Deprecation Policy and Management: SIG Architecture oversees the project's deprecation policy, providing a structured and communicative approach to phasing out old features or components. This ensures users have ample warning and clear migration paths, preventing sudden disruptions. The transition from Docker as an in-tree container runtime to external runtimes like containerd and CRI-O was cited as a prime example, demonstrating how the SIG managed a significant shift by clearly communicating the rationale and timeline.
• Aggressive Code Organization and Dependency Management: A significant, often unseen, achievement highlighted by Dims Srinivas is the ongoing effort in code organization. This sub-project actively works to reduce the number of vendored files and lines of code from external GoLang dependencies. The talk revealed a dramatic reduction from nearly 2 million lines of vendored code – a massive potential attack vector – to a significantly lower, more maintainable number. This effort directly enhances security, reduces binary size, and minimizes memory/CPU footprint, improving the overall health of the project.
• Rigorous Production Readiness Reviews (PRR): John Belamaric detailed the importance of Production Readiness Reviews (PRR). As features progress from experimental (alpha) to stable (beta) and finally GA, they undergo stringent reviews to ensure they meet high standards for reliability, operability, and safety. This includes verifying that features can be safely toggled, upgraded, and downgraded, and that they provide adequate monitoring capabilities, ultimately protecting cluster stability.
• Strategic Architectural Guidance and Dispute Resolution: SIG Architecture serves as the ultimate technical arbiter for disagreements between different SIGs or working groups. This ensures that architectural decisions are made holistically, considering the broader project impact, and that conflicting approaches are resolved through a structured, often asynchronous, process leading to consensus.

Technical Deep Dive

▶ Watch: Understanding Kubernetes governance structure: Steering Committee, SIGs (3:30)

SIG Architecture's influence permeates the technical fabric of Kubernetes through several specialized sub-projects and rigorous processes. These mechanisms are designed to maintain consistency, ensure quality, and guide the project's evolution.

The conformance program is a cornerstone of Kubernetes' "write once, run anywhere" promise. It mandates a comprehensive suite of tests that any Kubernetes distribution must pass to be considered conformant. This isn't merely about basic functionality; it ensures specific API behaviors, resource interactions, and operational guarantees are consistent across diverse environments. Historically, there were gaps in coverage, but through dedicated effort, SIG Architecture achieved near 100% API coverage in Kubernetes versions 1.31/1.32. Crucially, the process now requires new APIs to include conformance tests before graduating to GA, embedding consistency from inception.

API conventions are another critical technical artifact. These are not just recommendations but a living document of best practices and institutional knowledge accumulated over a decade of building Kubernetes APIs. They dictate everything from naming conventions and field types to versioning strategies and error handling. For instance, the process for adding a field is straightforward, but dropping or changing the meaning of a field is complex due to backward compatibility requirements. API review sessions, often involving core contributors like Jordan Liggitt, scrutinize every API change to ensure adherence, considering upgrade and downgrade paths for users. Developers building custom resources (CRs) and custom resource definitions (CRDs) are strongly encouraged to follow these conventions to ensure their extensions integrate seamlessly with the Kubernetes ecosystem and tooling.

The Enhancement Proposal (KEP) process (often referred to as 'caps' in the transcript) is the formal gateway for introducing significant new features or changes. A KEP is a detailed markdown document outlining the what, why, and how of a proposed change. It requires contributors to think through design principles, impact on existing features, potential interference with workloads, test plans, and even mechanisms for switching off or rolling back a feature if issues arise. KEPs are living documents, evolving through alpha, beta, and GA stages, reflecting lessons learned and adapting to calendar-based release schedules. This structured approach prevents ad-hoc development and ensures community-wide visibility and review.

The deprecation policy provides a clear framework for phasing out features, a necessity for a rapidly evolving project. The most prominent example discussed was the transition away from supporting Docker as an in-tree container runtime to relying on external runtimes like containerd and CRI-O. This decision was driven by maintainability concerns, as Docker required special in-tree support not extended to other runtimes. SIG Architecture guided this process, ensuring clear communication, a gradual deprecation timeline, and guidance for users to migrate, demonstrating how the project manages significant changes without breaking its vast user base.

The code organization sub-project, championed by Dims Srinivas, tackles the often-invisible but critical task of managing Kubernetes' vast dependency graph, particularly in the Golang ecosystem. This involves scrutinizing vendored code for security vulnerabilities, duplication, and excessive resource consumption. The speakers presented a "convoluted graph" illustrating the complexity of these dependencies. A significant technical achievement was the reduction of vendored lines of code from a peak of nearly 2 million to a much lower, more manageable number. This directly mitigates security risks (reduced attack surface), decreases binary size, and improves runtime performance by minimizing memory and CPU overhead from third-party libraries. This ongoing maintenance is a continuous battle against dependency bloat.

Production Readiness Reviews (PRR), led by John Belamaric, are a crucial gatekeeping mechanism for feature graduation. Before a feature can move from alpha to beta, and especially to GA, it must demonstrate production readiness. This involves meeting specific criteria related to reliability, observability, scalability, and operational safety. Key questions addressed include: Can the feature be easily turned off? Can clusters be safely upgraded and downgraded with the feature enabled? Are there sufficient metrics and logging for monitoring? These reviews are fundamental to Kubernetes' reputation for stability and operational excellence.

Finally, the concept of a compatibility version was briefly mentioned as an ongoing effort involving SIG Architecture. The goal is to decouple binary upgrades from API upgrades, which would allow for more flexible and safer multi-stage cluster upgrades, a significant operational challenge for large-scale deployments. This initiative represents a forward-looking architectural evolution to enhance the upgrade experience.

Demo / Proof of Concept

▶ Watch: Real-world example: HCD becoming a SIG (5:30)

This talk was primarily an organizational overview and update on the responsibilities and ongoing work of the Kubernetes SIG Architecture. As such, it did not feature a live technical demonstration or proof of concept of a specific new feature or vulnerability. Instead, the speakers focused on explaining the processes, governance structures, and the impact of SIG Architecture's work through examples and statistics, such as the reduction in vendored code lines and the progress in conformance test coverage.

Defensive Implications

▶ Watch: Pruning old SIGs and temporary working groups (7:50)

The work of Kubernetes SIG Architecture has profound implications for anyone operating, developing on, or contributing to Kubernetes. Understanding their mandates and processes is key to building more secure, stable, and compliant cloud-native environments.

For Kubernetes Users and Operators:

• API Consistency and Tooling: Adhering to Kubernetes API conventions when developing custom controllers, operators, or custom resource definitions (CRDs) is critical. This ensures that custom extensions behave predictably, integrate seamlessly with existing Kubernetes tooling, and benefit from the ecosystem's institutional knowledge, reducing the likelihood of unexpected behavior or difficult-to-debug issues.
• Anticipating Deprecations: Staying informed about the deprecation policy and upcoming feature removals (e.g., the Docker runtime transition) allows operators to plan migrations proactively, avoiding last-minute scrambling or unexpected breaks in their infrastructure. Monitoring release notes and SIG Architecture communications is essential.
• Trusting Production Readiness: The rigorous Production Readiness Reviews (PRR) mean that features graduating to beta and GA stages have been vetted for operational stability, observability, and safe upgrade/downgrade paths. Operators can have higher confidence in the reliability of these features in production environments.
• Leveraging Conformance: For enterprises or cloud providers offering Kubernetes, ensuring their distributions pass conformance tests guarantees a consistent and standard Kubernetes experience. This builds trust and reduces vendor lock-in by confirming API compatibility.
• Security through Code Hygiene: The significant efforts in code organization and dependency reduction directly translate to a smaller attack surface for Kubernetes itself. Fewer vendored libraries, and better managed ones, mean fewer potential vulnerabilities introduced through third-party code, enhancing the overall security posture of the platform.

For Kubernetes Developers and Contributors:

• Structured Innovation: The Enhancement Proposal (KEP) process forces thorough design and planning for new features. This disciplined approach ensures that contributions are well-thought-out, consider broader architectural implications, and are reviewed by the community, leading to higher quality and more stable code.
• Architectural Soundness: Engaging with SIG Architecture for architectural guidance or dispute resolution ensures that new components or changes align with the project's long-term vision and principles. This prevents fragmentation and maintains the coherence of the Kubernetes ecosystem.
• Contributing to Core Health: Developers can contribute directly to the platform's defensive posture by participating in sub-projects like code organization, helping to prune dependencies, identify security issues in vendored code, and reduce technical debt.

In essence, SIG Architecture acts as a central defensive layer, not against external threats, but against internal architectural decay, inconsistency, and operational fragility. By enforcing standards, streamlining processes, and meticulously reviewing changes, they empower both users and contributors to build and operate more robust, secure, and maintainable Kubernetes systems.

Key Takeaways

• Architectural Guardianship: Kubernetes SIG Architecture serves as the primary technical authority, ensuring the architectural integrity, consistency, and long-term health of the Kubernetes project.
• Comprehensive Oversight: Its responsibilities span critical areas including the conformance program, API conventions, enhancement proposals (KEPs), deprecation policies, and production readiness reviews.
• Prioritizing Consistency: The conformance program is paramount, ensuring that all Kubernetes distributions offer a consistent API experience, with new features now requiring conformance tests before GA.
• Structured Development: The KEP process enforces rigorous design, planning, and review for all significant new features, fostering thoughtful development and community consensus.
• Security through Simplicity: Aggressive code organization efforts have significantly reduced the project's vendored code dependencies (from nearly 2 million lines), directly lowering the attack surface and improving performance.
• Operational Reliability: Production Readiness Reviews (PRR) ensure features meet high standards for stability, observability, and safe upgrade/downgrade paths before being released to users.
• Community-Driven Evolution: SIG Architecture plays a crucial role in fostering collaboration between SIGs and resolving technical disagreements, demonstrating the project's commitment to continuous improvement and adaptability for new workloads like AI/ML.

About the Speaker(s)

John Belamaric is a co-chair of Kubernetes SIG Architecture and works at Google. He has been involved with the Kubernetes project for approximately six to seven years, playing a significant role in guiding its architectural evolution and focusing on areas like production readiness reviews.

Davanum Srinivas, affectionately known as "Dims," is also a co-chair of Kubernetes SIG Architecture and is associated with AWS. A long-time and active member of the Kubernetes community, Dims has been a "partner in crime" with John in various project initiatives, particularly championing efforts in code organization and dependency management.