Keynote: Driving Innovation at Michelin: How We Scaled Cloud & On-Prem In... G. Quennesson & A. Pons

Keynote: Driving Innovation at Michelin: How We Scaled Cloud & On-Prem In... G. Quennesson & A. Pons

Speakers: G. Quennesson, A. Pons

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=lFaSEevdZvU

Overview

This KubeCon EU keynote, delivered by G. Quennesson and A. Pons from Michelin, provides an insightful look into the tire manufacturing giant's ambitious journey to scale its Kubernetes footprint across both cloud and on-premises environments while simultaneously achieving significant cost reductions. The talk details Michelin's strategic pivot from a vendor-based Kubernetes solution to a completely open-source platform, driven by evolving vendor strategies, a maturing internal open-source culture, and a desire to foster internal technical expertise.

The core of Michelin's challenge was managing a rapidly expanding Kubernetes ecosystem, which by 2023 encompassed 62 clusters deployed across 42 different locations, supporting 450 business applications and orchestrating 36,000 pods. This massive scale, coupled with the limitations of their existing vendor solution, necessitated a fundamental re-evaluation of their platform strategy. The speakers articulate how this transition not only addressed immediate operational and cost concerns but also fundamentally transformed their engineering culture and accelerated platform innovation.

The article will explore the driving forces behind this architectural overhaul, delve into the technical specifics of their new open-source platform built around Cluster API and Argo CD, and highlight the remarkable efficiency gains and cost savings achieved. It will also examine the broader implications for organizations facing similar challenges, emphasizing the strategic advantages of embracing open-source solutions for critical infrastructure.

Background

▶ Watch: Introduction and Michelin's large Kubernetes footprint (0:00)

Michelin's journey with Kubernetes commenced in 2018, marking the initial steps into container orchestration for their diverse application landscape. Over the subsequent years, their Kubernetes footprint grew substantially, necessitating a robust, scalable platform. Until 2023, Michelin relied on a vendor-based solution to manage its Kubernetes infrastructure. This approach, while initially serving its purpose, eventually encountered significant friction points that compelled a strategic shift.

By 2023, two critical streams of events converged, forcing Michelin to reconsider its platform strategy. Firstly, the incumbent vendor altered its product strategy, announcing the deprecation of the specific solution Michelin was utilizing. Crucially, the vendor did not provide a clear migration path from the existing product to its new offering, effectively forcing Michelin into a complex and resource-intensive cluster migration. This scenario highlighted the inherent risks of vendor lock-in, where an organization's operational continuity and innovation trajectory become tethered to an external entity's strategic decisions.

Concurrently, Michelin's internal open-source strategy had matured considerably. The company had established an Open Source Project Office, signifying a dedicated commitment to leveraging and contributing to the open-source ecosystem. This internal cultural shift aligned with Michelin's long-standing ambition to empower its internal specialists and allow them to thrive. The "make instead of buy" philosophy gained significant traction, driven by the recognition that building internal expertise around open-source technologies could lead to greater control, flexibility, and long-term cost efficiencies, fostering innovation from within.

Faced with an impending mandatory migration and armed with a revitalized open-source mandate, Michelin made the decisive choice at the end of 2023 to rebuild its entire Kubernetes platform from the ground up. This monumental undertaking involved exclusively adopting open-source software, moving away from proprietary solutions to engineer a highly customized, scalable, and cost-effective infrastructure tailored precisely to their unique operational needs and future growth ambitions. This decision was not merely a technical one but a strategic organizational pivot towards greater self-reliance and internal capability building.

Key Findings

▶ Watch: Decision to rebuild platform entirely with open-source software (1:57)

The comprehensive rebuild of Michelin's Kubernetes platform using exclusively open-source software yielded several profoundly positive impacts, validating their strategic pivot. The results demonstrate not only significant financial benefits but also substantial improvements in operational efficiency and engineering engagement.

Firstly, and perhaps most strikingly, Michelin achieved a remarkable 44% year-to-year reduction in engineering costs for their platform. This figure is particularly impressive as it includes the cost of additional IT engineers brought in to support the transition, indicating the inherent efficiency and reduced overhead associated with managing an open-source stack. This cost saving was a direct result of eliminating vendor licensing fees, reducing reliance on proprietary tooling, and streamlining operational processes with community-driven solutions.

Secondly, the upgrade lead time for integrating new components or updating existing ones into their ecosystem was drastically reduced. What previously took "up to months" with the vendor solution is now "a matter of weeks." This improvement is critical for maintaining platform security, adopting new features rapidly, and responding to evolving business requirements. Furthermore, the streamlined nature of the new open-source processes means that even junior engineers can contribute to these upgrade tasks, significantly lowering the barrier to entry and distributing expertise across the team.

Thirdly, despite the massive undertaking of rebuilding the entire platform, Michelin's Kubernetes footprint continued its aggressive expansion. The speakers noted that their infrastructure grew about 100% during the period they were actively working on the project. This statistic underscores the inherent scalability and robustness of their new open-source design, demonstrating its ability to accommodate rapid growth even amidst a major architectural transition. It highlights that the rebuild did not impede, but rather facilitated, the company's continuous adoption of containerized applications.

Finally, and equally important for long-term sustainability, platform engagement and engineer satisfaction saw a significant boost. Working with open-source software proved to be "very much more interesting" for engineers compared to proprietary vendor solutions. This increased engagement fosters a culture of innovation, encourages knowledge sharing, and helps attract and retain top talent, creating a virtuous cycle of continuous improvement and internal expertise development. The ability to inspect, modify, and contribute to the underlying tools provides engineers with a sense of ownership and deeper understanding previously unavailable.

Technical Deep Dive

▶ Watch: Detailed architecture of the new open-source platform (2:06)

Michelin's new open-source Kubernetes platform is a sophisticated, GitOps-driven architecture designed for high scalability, multi-cloud and on-premises deployment, and operational efficiency. The core of this architecture revolves around a Cluster API (CAPI) management cluster, which serves as the central control plane for provisioning and managing all workload clusters across various infrastructures.

Within this management cluster, several key open-source components are deployed and integrated:

1. Argo CD: This is the cornerstone of Michelin's GitOps at scale strategy. Argo CD is used to automate the deployment and lifecycle management of applications and configurations on the workload clusters. By treating Git as the single source of truth, Argo CD ensures that the desired state of all clusters and their applications is continuously reconciled with the configurations stored in Git repositories.

1. Cluster API Providers: These providers are essential for managing the lifecycle of workload clusters across diverse infrastructure types. CAPI itself provides a declarative API for Kubernetes cluster creation, configuration, and management. By leveraging specific infrastructure providers (e.g., for AWS, Azure, vSphere, or bare-metal), Michelin can abstract away the underlying infrastructure details, allowing their CAPI management cluster to provision and manage Kubernetes clusters consistently, regardless of whether they are in the cloud or on-premises. This standardization is critical for managing 42 different locations.

1. Crossplane: This component plays a crucial role in handling the prerequisites for infrastructure and managing add-ons. Crossplane extends Kubernetes to manage and provision infrastructure resources and services from various cloud providers and on-premises systems as if they were native Kubernetes resources. In Michelin's setup, Crossplane likely provisions the underlying virtual machines, networks, and storage that the Cluster API providers then use to build the Kubernetes clusters. Furthermore, Crossplane is utilized for managing add-ons, meaning it can provision and configure services or components that run alongside the Kubernetes clusters themselves, potentially external databases, object storage buckets, or other foundational services required by the applications.

1. Custom Component: A critical integration piece developed in-house is a custom component designed to link Cluster API cluster objects with Argo CD cluster secret objects. This custom component bridges the gap between the cluster lifecycle management provided by CAPI and the application deployment capabilities of Argo CD. When CAPI provisions a new workload cluster, this custom component ensures that the necessary credentials and configuration details for that new cluster are securely transformed into an Argo CD cluster secret. This secret then allows Argo CD to discover and manage applications on the newly provisioned workload cluster, seamlessly integrating the infrastructure and application layers. This automation is vital for handling the dynamic creation of clusters and applications.

The entire architecture is underpinned by a robust Git repository structure:

• Cluster Inventories, Cluster Definitions, and Add-ons Repository: This repository stores the declarative configurations for all workload clusters, including their desired specifications (e.g., Kubernetes version, node count, network settings) and the add-ons that should be deployed onto them.
• Argo CD Configuration Repository: This repository holds the configurations for Argo CD itself, primarily defining ApplicationSets. ApplicationSets are powerful Argo CD objects that enable dynamic application creation based on cluster labels. This means that as new workload clusters are provisioned by CAPI and registered with Argo CD (via the custom component), applications can be automatically deployed to them based on their characteristics, ensuring consistency and rapid deployment across the fleet.
• Helm Chart Repositories: Michelin maintains several Helm chart repositories for managing common add-ons and applications. Examples mentioned include:
• Gatekeeper: Used for enforcing policies across the Kubernetes clusters. Gatekeeper is an admission controller that allows defining and enforcing policies (e.g., disallowing certain image registries, requiring specific labels, enforcing resource limits) using Open Policy Agent (OPA). This is crucial for maintaining security, compliance, and operational best practices.
• Cilium: Employed as the Container Network Interface (CNI). Cilium provides advanced networking, security, and observability for container workloads. It enables fine-grained network policies based on application identity rather than IP addresses, offering enhanced security controls and performance.
• Custom Charts: For applications and services specific to Michelin's internal needs.

This comprehensive GitOps-driven approach, combining CAPI for infrastructure, Crossplane for prerequisites and external services, and Argo CD for application deployment, allows Michelin to manage its vast and distributed Kubernetes environment with unprecedented automation, consistency, and efficiency. The custom integration component is a testament to their engineering prowess in stitching together complex open-source tools to meet specific enterprise needs.

Demo / Proof of Concept

▶ Watch: Drastically reduced upgrade lead time from months to weeks (4:18)

The talk primarily focused on the architectural design, strategic motivations, and quantitative outcomes of Michelin's platform transformation rather than a live demonstration. The speakers presented a high-level architectural diagram illustrating the interplay of the various open-source components and their Git repositories. While no specific live demo or detailed proof-of-concept walkthrough was provided, the detailed explanation of the system's components, their interconnections, and the resulting performance metrics serve as a compelling evidence of the platform's functionality and success. The emphasis was on the implemented solution and its measurable impact, rather than a step-by-step technical execution, which is typical for a keynote presentation focused on strategic outcomes.

Defensive Implications

▶ Watch: Increased engineer engagement and continued Kubernetes footprint growth (4:45)

While the talk primarily addresses platform scaling and cost optimization, Michelin's shift to a fully open-source Kubernetes platform has significant positive defensive implications, enhancing the overall security posture and resilience of their infrastructure.

Firstly, the adoption of GitOps with Argo CD inherently improves security by enforcing a declarative, immutable infrastructure. All changes to cluster configurations and application deployments must be committed to a Git repository, providing an auditable trail and enabling easy rollback to previous states. This eliminates snowflake configurations and reduces the attack surface by preventing manual, ad-hoc changes that could introduce vulnerabilities. The principle of least privilege is also reinforced, as direct access to production clusters for configuration changes can be minimized, with Git becoming the primary interface.

Secondly, the integration of Gatekeeper for policy enforcement is a critical defensive measure. By leveraging Gatekeeper as an admission controller, Michelin can define and automatically enforce security policies across all their Kubernetes clusters. This includes preventing the deployment of unapproved container images, enforcing mandatory security labels, ensuring resource limits are set to prevent denial-of-service attacks, and blocking configurations known to be insecure. This proactive policy enforcement shifts security left, catching potential issues before they are deployed to production.

Thirdly, the choice of Cilium as the CNI provides advanced network security capabilities. Cilium allows for identity-aware network policies, meaning traffic can be controlled based on the workload's identity (e.g., a specific application or service) rather than just its IP address. This enables a robust micro-segmentation strategy, isolating applications and services from each other and limiting lateral movement in case of a breach. Cilium's eBPF-based architecture also offers deep visibility into network traffic, aiding in threat detection and incident response.

Fourthly, the significant reduction in upgrade lead time (from months to weeks) is a direct defensive advantage. Faster upgrade cycles mean that security patches, vulnerability fixes, and updated software versions can be deployed across the entire fleet much more rapidly. This minimizes the window of exposure to newly discovered CVEs and ensures that the platform remains current with the latest security enhancements from the open-source community. The ability for junior engineers to perform these updates further democratizes security operations and reduces bottlenecks.

Finally, the move to open-source fosters a deeper understanding of the underlying technology among Michelin's internal engineering team. Engineers gain the ability to inspect the source code, understand its inner workings, and even contribute fixes or enhancements. This increased transparency and internal expertise reduce reliance on vendor-specific black boxes, allowing for more thorough security reviews and quicker identification of potential vulnerabilities, ultimately strengthening the collective defensive capabilities of the organization.

Key Takeaways

• Strategic Pivot to Open Source: Michelin successfully transitioned from a vendor-based Kubernetes solution to a fully open-source platform, driven by vendor strategy changes and a maturing internal open-source culture.
• Significant Cost Reduction: The shift resulted in a remarkable 44% year-to-year reduction in engineering costs for the platform, even with additional hires, demonstrating the financial benefits of open-source adoption.
• Enhanced Operational Efficiency: Upgrade lead times were drastically cut from months to weeks, improving agility and enabling faster deployment of security patches and new features, even allowing junior engineers to manage these processes.
• Scalability and Growth: Despite a major platform rebuild, Michelin's Kubernetes footprint grew by 100%, showcasing the new architecture's robust scalability across 62 clusters and 42 locations supporting 36,000 pods.
• GitOps-Driven Architecture: The platform leverages a sophisticated GitOps model with Cluster API for infrastructure management, Crossplane for prerequisites and add-ons, and Argo CD for application deployment, ensuring consistency and automation.
• Improved Security Posture: Integration of tools like Gatekeeper for policy enforcement and Cilium for advanced network security, combined with the auditable nature of GitOps, significantly enhances the defensive capabilities and resilience of the Kubernetes environment.
• Increased Engineer Engagement: Working with open-source software led to higher engineer satisfaction and engagement, fostering internal expertise and driving innovation from within the company.

About the Speaker(s)

G. Quennesson and A. Pons are key members of Michelin's "Container as a Service" platform team. In their roles, they are directly responsible for delivering and managing the Kubernetes platform for Michelin's internal project teams. Their expertise lies in scaling complex containerized infrastructures, navigating the challenges of hybrid cloud and on-premises deployments, and strategically leveraging open-source technologies to drive efficiency and innovation within a large enterprise environment. Their presentation reflects deep practical experience in architecting and implementing large-scale Kubernetes solutions under demanding business constraints.