Helm 4 You - Matt Farina, SUSE & Andrew Block, Red Hat

Helm 4 You

Speakers: Matt Farina, Distinguished Engineer, SUSE; Andrew Block, Distinguished Architect, Red Hat

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=rdTPbm9f_fc

Overview

In this KubeCon EU session, Helm maintainers Matt Farina and Andrew Block unveiled the ambitious plans and progress for Helm 4, the next major iteration of the Kubernetes package manager. After more than five years since the release of Helm 3, the project is embarking on a significant update that includes intentional breaking changes – a departure from Helm's previous commitment to backward compatibility. This talk articulates the necessity of these changes, driven by the evolving Kubernetes ecosystem, advancements in the Go programming language, and the need to address long-standing architectural limitations and user experience challenges.

The speakers highlighted Helm's enduring popularity and its critical role as a trusted tool for managing Kubernetes applications, even amidst numerous alternative frameworks that have emerged over the years. However, its longevity also meant carrying technical debt, particularly due to a steadfast avoidance of breaking changes in Helm 3. Helm 4 represents a pivotal moment for the project, allowing maintainers to modernize the codebase, enhance extensibility, and integrate contemporary cloud-native patterns, ultimately aiming to deliver a more robust and flexible experience for both CLI users and those leveraging Helm as an SDK.

This deep dive into Helm 4's roadmap covers completed work, features currently in development, and experimental initiatives like Charts v3. The overarching goal is to prepare Helm for the next decade of Kubernetes development, ensuring it remains a cutting-edge and adaptable tool for deploying and managing applications in an increasingly complex containerized world. For users and developers alike, understanding these changes is crucial for migrating to Helm 4 and leveraging its new capabilities.

Background

▶ Watch: Introduction and Helm's long history (0:00)

Helm has been a cornerstone of the Kubernetes ecosystem for a remarkable duration, predating even the CNCF's 10-year anniversary. Its journey began in 2016, with Helm v1 introducing the concept of charts for defining, installing, and upgrading even the most complex Kubernetes applications. Over time, it evolved, with Helm v3 being released over five years ago. A defining characteristic of Helm 3's development was an unwavering commitment to backward compatibility. This philosophy, while ensuring stability for its rapidly growing user base, inadvertently created a significant amount of technical debt and hindered the adoption of newer technologies and best practices.

The speakers provided several concrete examples of this self-imposed limitation. For instance, when the helm repo list command encountered an error updating a repository in Helm 3, its exit code remained 0 (success) by default. To rectify this without breaking existing scripts, a specific flag was introduced to allow users to opt-in to a non-zero error code. Similarly, Helm 3 continued to support deprecated Kubernetes API versions that Kubernetes itself had stopped serving years prior, including beta APIs from Kubernetes 1.15. This was a direct consequence of the "do not break anything" mantra, leading to a codebase burdened with legacy support. Another critical example involved the oras library, used for handling OCI artifacts. Helm was the last major project still tied to oras v1, preventing it from benefiting from the significant improvements and feature tracking available in oras v2 due to potential SDK compatibility issues. Even unused public functions, long abandoned by Helm itself, had to be maintained due to their potential use by external SDK consumers.

These constraints meant that despite the rapid evolution of Kubernetes, Go language features (like structured logging), and cloud-native patterns over the past five years, Helm's core architecture and capabilities remained largely static. The project reached a point where fundamental improvements, modernization, and increased extensibility could only be achieved by accepting the necessity of breaking changes, signaling a new era for Helm. This understanding paved the way for Helm 4, a major version upgrade designed to shed this legacy and embrace the future.

Key Findings

▶ Watch: Why Helm 4 embraces breaking changes (3:15)

The central "key finding" or strategic decision driving Helm 4 is the embrace of breaking changes. This marks a significant philosophical shift from Helm 3, which prioritized backward compatibility above almost all else. After five years of accumulating technical debt and observing the rapid evolution of the cloud-native landscape, the maintainers recognized that a major version bump was essential to modernize the project effectively. This decision allows Helm to shed support for outdated Kubernetes APIs, remove legacy code, and adopt contemporary Go language features and architectural patterns.

A core contribution of Helm 4 is its commitment to improving the developer and SDK user experience. By refactoring internal components, like making the command parser a standalone Go package, Helm 4 enables other applications to embed Helm's functionality more seamlessly, akin to how K3S integrates kubectl commands. The migration to Go's new slog package for structured logging is another testament to this, offering richer, more queryable output for both CLI users and applications consuming Helm as a library.

Furthermore, Helm 4 aims to significantly enhance extensibility and future-proofing. The introduction of an experimental Charts v3 API demonstrates a new approach to evolving chart capabilities without immediately breaking the vast ecosystem of existing v1 and v2 charts. This allows for iterative development and community feedback on ambitious features like pluggable rendering engines, advanced CRD handling, and custom resource ordering. Coupled with improvements to the plugin system, including discussions around WebAssembly (Wasm) for multi-language plugin support, Helm 4 is designed to be more adaptable to diverse user needs and future technological shifts. The planned integration of modern security tools like cosign and sigstore for chart signing also addresses a critical long-standing pain point, making Helm charts more trustworthy and easier to secure. These collective efforts position Helm 4 not just as an update, but as a foundational re-architecture for the next generation of Kubernetes application management.

Technical Deep Dive

▶ Watch: Embedding Helm commands in other applications (4:20)

Helm 4 represents a significant technical overhaul, addressing long-standing issues and introducing new capabilities across its core components. The changes span from internal architectural refactorings to user-facing features, all aimed at modernization and enhanced extensibility.

One of the most foundational changes is the introduction of intentional breaking changes. This strategic decision, a departure from Helm 3's strict backward compatibility, is evident in several areas:

• Command Exit Codes: The helm repo list command, which previously returned a 0 exit code even on update errors, now correctly returns a non-zero exit code. This rectifies a long-standing issue that required a special flag in Helm 3 to achieve correct error reporting.
• Kubernetes API Support: Helm 4 removes support for deprecated Kubernetes APIs, specifically mentioning beta APIs from Kubernetes 1.15 and older versions. This eliminates the burden of carrying legacy compatibility, allowing Helm to focus on modern Kubernetes environments and reduce its codebase complexity.
• Removed Unused Functions: Numerous public API functions that Helm itself no longer used, but were maintained for SDK compatibility, have been removed. This streamlines the codebase, though a migration guide will be provided for SDK users transitioning from Helm 3.

Internally, Helm 4 has undergone significant architectural improvements:

• Command Package Refactoring: The Helm command parser has been refactored into a standalone Go package. This enables other Go applications, such as specialized Kubernetes tools like K3S, to embed and utilize Helm commands directly, fostering deeper integration within the cloud-native ecosystem.
• OCI Library Upgrade: Helm is migrating from oras v1 to oras v2. This is a crucial update, as Helm was the last major project using the older version, which had limited functionality and complicated feature tracking. The upgrade to oras v2 promises improved handling of OCI artifacts and better integration with the broader OCI ecosystem, addressing past challenges users faced with OCI registries.

A major focus for Helm 4 is logging. Recognizing the fragmented state of Go logging five years ago and the diverse logging solutions used by applications embedding Helm's SDK, the project is migrating to Go's new slog package. This standard library package provides structured logging, allowing for richer, more queryable log output. While slog offers compatibility layers for existing Kubernetes logging (like Klog) and shims for popular third-party loggers (e.g., Logrus, Zap), the maintainers are still evaluating which specific logger implementation will provide the best default output experience for the Helm CLI. The structured output will greatly aid debugging and integration with centralized logging systems.

One of the most forward-looking initiatives is the introduction of Charts v3, an experimental API version for Helm charts. Unlike previous chart versions, Charts v3 is being developed in an experimental phase, allowing the maintainers to iterate and gather feedback on potentially breaking changes without constraining the Helm 4 release timeline. Key features being explored for Charts v3 include:

• Pluggable Rendering Engines: This feature aims to allow users to replace Helm's default Go template rendering engine with alternatives, such as YAML script or other programmatic templating tools, directly within the chart definition. This addresses a common request for more flexible and powerful templating capabilities.
• Alternative Dependency Handling: Exploring new mechanisms for managing chart dependencies beyond the current charts subdirectory approach, potentially offering more robust and flexible solutions.
• Improved CRD Handling: Addressing the complex and often problematic management of Custom Resource Definitions (CRDs). Helm's current approach is highly conservative due to the global nature of CRDs and the potential for collisions or breaking changes when multiple chart releases (even in different namespaces) interact with them. Charts v3 aims to provide more intelligent and safer ways to handle CRD lifecycle management, including scenarios like rollbacks and versioning.
• Custom Resource Ordering: While Kubernetes aims for eventual consistency, real-world deployments often require specific resource creation or update orders. Charts v3 may introduce mechanisms for users to define or influence this ordering, especially for CRDs, to ensure application stability.

Status checking is another area of significant improvement. Helm 3's status logic was often described as opinionated and generic, lacking the intelligence needed for complex deployments. Helm 4 integrates Kstatus, a library originating from the Flux project, designed for more intelligent and accurate waiting and status checking of Kubernetes resources. This will provide users with better visibility into the readiness of their deployed applications.

For OCI registries, Helm 4 is adopting the registries.conf concept, familiar from container image management tools. This provides a standardized way to configure OCI artifact proxies, mirrors, aliases, and credential helpers, simplifying the management of chart repositories in air-gapped or complex network environments. This aligns Helm with broader container ecosystem practices and involves collaboration with other container engine communities.

The plugin system is also slated for enhancements. Helm 4 aims to provide greater extensibility, allowing users to hook into more points of Helm's workflow. While initial improvements for Helm 4 might be minimal, the maintainers are exploring ambitious ideas, including leveraging WebAssembly (Wasm) to enable plugins to be written in various programming languages and executed across different architectures, significantly broadening the plugin ecosystem.

Finally, security is a major focus. Helm 4 plans to move beyond the cumbersome PGP/GPG signing mechanism for charts. The integration of cosign and sigstore will provide a more modern, user-friendly, and robust solution for signing and verifying Helm charts, leveraging the broader supply chain security initiatives in the cloud-native space. While a cosign plugin exists for Helm 3, Helm 4 aims for more native integration. Furthermore, the maintainers have a plan to address and replace the legacy Go crypto library currently used for PGP, which has known limitations like a lack of EDDSA support, making PGP signing more secure even for those who continue to use it.

Demo / Proof of Concept

▶ Watch: Migrating to ORAS v2 for OCI artifact improvements (6:20)

This session focused primarily on detailing the roadmap, completed work, and future plans for Helm 4 through presentation and discussion. There was no live demonstration or proof-of-concept shown during the talk. The speakers outlined features and architectural changes, some of which are already implemented, while others are still in active development or an experimental phase.

Defensive Implications

▶ Watch: Cleaning up legacy and unused public APIs (7:30)

For organizations and individual users, the advent of Helm 4 brings several crucial defensive implications and recommended actions:

1. Prepare for Migration: Helm 4 introduces intentional breaking changes. While a direct migration path with documentation will be provided, users, especially those leveraging Helm as an SDK in their tooling (e.g., CI/CD pipelines, operators), must anticipate and plan for necessary code updates. CLI users should also review their scripts for compatibility, particularly regarding command exit codes and deprecated Kubernetes API usage. The planned support window for Helm 3 is expected to be around 6-8 months post-Helm 4 release, providing a transition period.
2. Modernize OCI Registry Management: Leverage the new registries.conf capabilities to standardize and secure how Helm interacts with OCI registries. This includes configuring proxies, mirrors, and credential helpers, which is particularly vital for air-gapped environments or ensuring consistent access to trusted chart sources. This aligns Helm's OCI behavior with other container tools.
3. Adopt Enhanced Security Features: Prioritize the adoption of cosign and sigstore for signing Helm charts. This move away from legacy PGP/GPG streamlines the signing process and integrates with modern supply chain security practices, significantly enhancing the trustworthiness and integrity of deployed applications. Defenders should integrate chart signing and verification into their CI/CD pipelines.
4. Leverage Improved Logging: The migration to slog and structured logging provides richer, more queryable output. Defenders should update their logging aggregation and analysis tools to parse and benefit from Helm 4's structured logs, improving observability and making debugging and incident response more efficient.
5. Monitor Charts v3 Development: While experimental, Charts v3 promises significant improvements, especially in CRD handling and pluggable rendering engines. Defenders should follow its development closely, as these features could offer more robust and secure ways to manage complex applications and custom resources, potentially mitigating risks associated with CRD collisions or misconfigurations.
6. Engage with the Community: The Helm maintainers explicitly invited community feedback and contributions. Defenders encountering specific pain points, especially concerning security, CRD management, or extensibility, should participate in the Helm Dev Slack channel, weekly meetings, or file issues. This ensures that their operational needs are considered in future Helm development.

By proactively addressing these areas, organizations can smoothly transition to Helm 4, enhance their security posture, and leverage the new features to build more resilient and manageable Kubernetes environments.

Key Takeaways

• Intentional Breaking Changes: Helm 4 marks a significant departure from Helm 3's strict backward compatibility, introducing necessary breaking changes to modernize the codebase and shed technical debt accumulated over five years.
• Modernization and Extensibility: The release focuses on internal architectural improvements (e.g., command package as a Go package), updated dependencies (e.g., oras v2), and enhanced extensibility through an improved plugin system and the experimental Charts v3 API.
• Enhanced Developer Experience: Key updates like the migration to Go's slog package for structured logging and the integration of Kstatus for intelligent status checking aim to provide a richer, more debuggable, and integrated experience for both CLI and SDK users.
• Future-Proofing with Charts v3: The experimental Charts v3 introduces a flexible framework for future chart capabilities, including pluggable rendering engines, improved CRD handling to mitigate collision risks, and custom resource ordering, developed iteratively with community feedback.
• Improved Security Posture: Helm 4 will natively support cosign and sigstore for chart signing, moving beyond PGP/GPG to align with modern software supply chain security practices, and plans to update its legacy Go crypto library.
• Community-Driven Evolution: The maintainers actively solicit community involvement in shaping Helm 4 and beyond, emphasizing contributions to code, documentation, and feature ideas to address diverse real-world use cases.

About the Speaker(s)

Matt Farina is a Distinguished Engineer at SUSE and a prominent Helm maintainer. His extensive experience in the cloud-native space and deep involvement with Helm's development are evident in his detailed understanding of the project's history, architectural challenges, and future direction. Matt has been a driving force behind many of Helm's past versions and continues to champion its evolution, particularly in areas like API compatibility and internal code health.

Andrew Block is a Distinguished Architect at Red Hat and also a dedicated Helm maintainer. Andrew brings a wealth of experience from working with various container engines and the broader cloud-native ecosystem. His perspective is particularly valuable in aligning Helm's developments with industry standards and community practices, as demonstrated by his focus on OCI registry enhancements and security features like cosign integration. Together, Matt and Andrew represent the core leadership guiding Helm into its next major iteration.