From Metal To Apps: LinkedIn’s Kubernetes-based Compute Platform - Ahmet Alp Balkan & Ronak Nathani

From Metal To Apps: LinkedIn’s Kubernetes-based Compute Platform

Speakers: Ahmet Alp Balkan, Ronak Nathani (Engineers, Central Compute Team, LinkedIn)

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=dDkXFuy45EA

Overview

This talk, "From Metal To Apps: LinkedIn’s Kubernetes-based Compute Platform," presented by Ahmet Alp Balkan and Ronak Nathani from LinkedIn's central compute team, delves into the intricate architecture and operational strategies behind running one of the world's largest Kubernetes deployments on bare metal. It highlights how LinkedIn has heavily extended and customized Kubernetes to meet the demands of its massive scale, serving over a billion members with thousands of services and millions of containers across half a million physical servers. The presentation offers a unique perspective on enterprise-grade Kubernetes adoption, particularly for organizations operating their own data centers without reliance on public cloud providers.

The talk provides a comprehensive journey from the foundational Infrastructure as a Service (IaaS) layer, through the Kubernetes Cluster Management layer, and finally to the Workload Platform layer where applications reside. It meticulously details the custom-built components, design philosophies, and operational challenges overcome in orchestrating a diverse array of stateless, stateful, and batch workloads. The speakers emphasize LinkedIn's pragmatic approach to leveraging open-source Kubernetes while developing bespoke solutions for specific pain points related to scale, performance, and developer experience.

Why this talk matters is multifaceted. For enterprises contemplating or executing large-scale Kubernetes migrations, especially those with significant on-premises infrastructure, LinkedIn's experience offers invaluable lessons. It demonstrates that Kubernetes, while powerful, requires substantial adaptation and custom tooling to thrive in environments with unique constraints like bare-metal operations, extreme scale, and existing internal service infrastructure. The insights shared regarding control plane scalability, multi-cluster management, automated maintenance, and developer guardrails provide a blueprint for building resilient and efficient compute platforms in complex, high-demand environments.

Background

▶ Watch: Introduction and LinkedIn's massive scale (0:00)

LinkedIn operates at an extraordinary scale, boasting over a billion members served by approximately 3,000 services, encompassing databases, stream processing, and microservices. These services are deployed across more than 500,000 physical servers and collectively manage 1.5 million containers. A critical distinguishing factor for LinkedIn is its complete reliance on bare metal infrastructure within its own data centers, eschewing virtualization or public cloud providers primarily for performance reasons. This foundational choice significantly impacts their compute platform design, necessitating custom solutions for infrastructure management that cloud users often take for granted.

Before its migration to Kubernetes, LinkedIn had developed and operated its own proprietary container runtime and scheduler for over a decade, predating the widespread adoption of Docker and Kubernetes. This legacy system had successfully served its diverse workloads, but the company embarked on a strategic migration to Kubernetes to modernize its compute platform. The goal was a seamless transition without downtime, driven by central automation and minimal user involvement, while simultaneously challenging legacy requirements to reduce technical debt. At the time of the talk, over half of their stateless fleet and several stateful systems were already running in production on Kubernetes.

The compute platform at LinkedIn is structured into three distinct layers:

1. Infrastructure as a Service (IaaS) Layer: This foundational layer manages the physical hardware, including inventory, allocation, and maintenance. Notably, this layer does not incorporate Kubernetes concepts but serves as the bedrock upon which Kubernetes clusters are built.
2. Kubernetes Cluster Management Layer: This layer focuses on the lifecycle and management of Kubernetes clusters themselves, including node pools, cluster scaling, and control plane operations. It leverages and extends open-source Kubernetes components.
3. Workload Platform Layer: This topmost layer provides the developer-facing interface for deploying and managing applications on the Kubernetes clusters. It abstracts away much of the underlying Kubernetes complexity, offering custom resources and workflows tailored for LinkedIn's engineering teams.

A significant aspect of LinkedIn's background is its established internal service infrastructure. Unlike many Kubernetes adopters who rely on native Kubernetes features like Services, DNS, Configs, Secret Management, and Network Policies, LinkedIn largely bypasses these for application workloads. The reasons are twofold: these features often struggle with the overhead of spanning hundreds of clusters in a region, and LinkedIn already possesses robust, scaled, and hardened equivalents developed independently over the years. Consequently, Kubernetes primarily serves as the pod orchestrator within this existing ecosystem, integrated with LinkedIn's mature regional services for discovery, configuration, and security.

Key Findings

▶ Watch: Overview of LinkedIn's 3-layer compute platform architecture (2:00)

LinkedIn's journey into large-scale, bare-metal Kubernetes reveals several critical findings and architectural contributions that diverge significantly from typical deployments:

1. Extreme Customization is Essential for Bare-Metal Scale: Running Kubernetes on 500,000+ bare-metal servers for 1.5 million containers necessitates a deeply customized stack. LinkedIn built an entire Infrastructure as a Service (IaaS) layer, including an Inventory Manager and Compute Broker, to manage physical machines, pools, and maintenance operations. This bespoke foundation ensures optimal performance and control, which standard cloud-provider integrations or even typical cluster API solutions cannot provide.

1. Kubernetes as a Pod Orchestrator, Not a Full Platform: LinkedIn intentionally limits the scope of Kubernetes within its ecosystem. Critical functionalities like service discovery, DNS, configuration, and secrets management are handled by existing, battle-tested internal regional services, rather than native Kubernetes constructs. This approach leverages Kubernetes' strengths in container orchestration while avoiding the complexities of scaling certain features across hundreds of clusters.

1. Massive Multi-Tenant Clusters are Achievable but Challenging: LinkedIn aims to push cluster sizes to 5,000 nodes and beyond, running a mix of stateless, stateful, and batch workloads within the same cluster. This strategy reduces hardware fragmentation and allows applications to grow in place. However, it introduces significant challenges for control plane scalability, particularly with etcd and custom controllers, requiring aggressive tuning, increased etcd storage (from 8GB to 16GB), and ongoing efforts to address controller scaling limitations.

1. Developer Experience is Paramount, Abstracting Kubernetes Complexity: To cater to thousands of engineers, LinkedIn developed custom, simplified Kubernetes resources like LI deployments and LI stateful sets. Users interact solely with these high-level abstractions, preventing them from needing to be Kubernetes experts or from crafting complex native Deployment or StatefulSet specs. Extensive validation and self-serve troubleshooting tools significantly reduce support load.

1. Automated, Coordinated Maintenance is Crucial for Bare Metal: With a fleet of half a million servers, automated maintenance is non-negotiable. LinkedIn implemented Maintenance Zones (software update domains, typically 20 per data center) and a Coordinated Maintenance Operation system. This system automates planned and unplanned disruptions, drains nodes, performs upgrades (e.g., OS, kernel, kubelet), and reintroduces machines, ensuring no more than 5% of capacity is impacted at any given time.

1. Stateful Workloads Require Generic, Integration-Focused Operators: Recognizing the difficulty for data system teams (e.g., Kafka, Espresso) to become Kubernetes experts, LinkedIn built a generic stateful controller. This controller integrates with existing shard managers via a custom protocol, handling pod updates, evictions, and maintenance, thereby centralizing complex stateful operations.

1. GitOps and Guardrails are Foundational for Reliability: A robust GitOps workflow, initially using Argo CD per cluster (though now migrating to an in-house engine due to scale issues), combined with extensive API guardrails, is critical. These guardrails include delete protection for all custom resources and namespaces, limits on scale-down operations, and caps on max surge and canary percentages, preventing human error from causing incidents.

Technical Deep Dive

▶ Watch: Deep dive into the Infrastructure as a Service layer (2:50)

LinkedIn's compute platform is a testament to engineering ingenuity, meticulously designed to operate at extreme scale on bare metal. The architecture is stratified into three layers, each with custom components and integrations.

Infrastructure as a Service (IaaS) Layer

At the very bottom, the IaaS layer manages the physical hardware. This layer predates and exists independently of Kubernetes.

• Inventory Manager: This serves as the authoritative metadata store for all machines in LinkedIn's data centers. It tracks machine names, hardware characteristics (CPU, memory, GPU), and other attributes from the moment a technician racks a machine.
• Compute Broker: Described as the "heart and brain" of the machine layer, the Compute Broker is a centralized gRPC service (not a Kubernetes operator). It functions similarly to a cloud provider's virtual machine scale set or autoscaling instance group API, managing pools of machines and adding/removing capacity.
• Node Profiles: Each machine pool is defined by a node profile, which specifies minimum hardware requirements as a lowest common denominator, allowing for heterogeneous hardware within pools.
• Source of Truth: The Compute Broker acts as the source of truth for machine allocation and maintenance operations.
• Host Health Remediation: Given the sheer number of servers, machine failures are a constant. LinkedIn has automated systems to detect failing machines, take them out of rotation, service them, and perform upgrades.
• Maintenance Orchestrator: This system manages fleet-wide upgrades (e.g., kernel, security updates). It incorporates concepts inspired by Kubernetes.
• Maintenance Zones: These are logical software update domains, not physical availability zones. Data centers are striped into 20 maintenance zones (e.g., MZ1 to MZ20). Software updates are rolled out one zone at a time, ensuring that a bad update impacts no more than 5% of the fleet's capacity. Compute pools are evenly striped across these zones to maintain resilience.
• Coordinated Maintenance Operations: All maintenance is automated, removing human intervention.
• Disruptions: A core concept where control of a machine is temporarily transferred from Kubernetes (or other users) to a maintenance actor.
• Planned Maintenance: Regular OS/kernel upgrades, switch upgrades, hardware decommissioning, kubelet upgrades.
• Unplanned Operations: Hostile remediation for unexpected failures.
• Workflow: A machine disruptor initiates a disruption, which is sent to the Compute Broker. The Kubernetes cluster management layer observes this, approves the disruption (e.g., by draining the node), the maintenance actor performs its task (reboot, reimage), removes the disruption, and the machine returns to rotation. This entire cycle is automated.

Kubernetes Cluster Management Layer

LinkedIn's Kubernetes clusters are built from open-source bits, without relying on specific distributions, kubeadm, or Cluster API.

• Simplistic Stack: A single provisioner is used, aligning with their bare-metal environment.
• Tuning: Extensive tuning of API server and etcd flags is performed for performance and scale.
• Cluster Size: Clusters are pushed to 5,000 nodes and beyond to reduce hardware fragmentation and accommodate multi-tenant growth. Clusters run a mix of stateless, stateful, and batch workloads.
• Kubelet Upgrades: These are integrated into the OS maintenance cycle, managed by the Maintenance Orchestrator.
• Centralized Hub Cluster: A single hub cluster manages all other clusters and dictates application placement.
• Kubernetes Resource Model (KRM) APIs: LinkedIn uses KRM APIs and in-house controllers to manage pools and clusters. For example, Kubernetes pools are modeled as a Custom Resource (CR) in the management cluster. Changes to this CR are converted into a Compute Broker CR, which then synchronizes with the Compute Broker's gRPC API. This allows capacity adjustments by simply changing an integer in a YAML file, resulting in bare-metal machines appearing in the cluster.

Scaling Kubernetes Itself

Scaling the control plane for 5K+ node clusters is a significant challenge.

• Control Plane as Shared Resource: Control plane stability is critical for all tenants.
• RBAC Review: Customer-brought operators undergo heavy RBAC review to control their permissions.
• API Priority and Fairness (APF): Used to ensure fair resource access and limit load on the API server.
• etcd Bottlenecks: etcd is identified as the primary bottleneck for large clusters due to increased event submission and pod churn.
• Storage Increase: etcd storage limit was increased from 8GB to 16GB, with plans for further increases. They run etcd on SSDs.
• Backup/Restore: An internal backup and restore system is in place for catastrophic failures.
• Separate Events Clusters (experiment): While they experimented with separate etcd clusters for events, they plan to merge them back due to performance issues with separate disks, preferring to scale the main database size instead.
• Controller Scalability: Horizontal scaling of controllers remains an active area of research and development, acknowledged as a common industry problem.

Workload Platform Layer

This layer is designed for application developers, abstracting Kubernetes complexity.

• Custom Resources for Applications: Developers interact with two custom resources: LI deployments (for stateless) and LI stateful sets (for stateful). Users have no permissions to native Kubernetes resources.
• Batch Workloads: For platforms like Spark and Machine Learning, Volcano is used as a batch scheduler, providing features like fair and gang scheduling. LinkedIn built its own regional job queue and quota systems because Volcano's existing systems are cluster-scoped, whereas LinkedIn requires regional effectiveness across hundreds of clusters.
• Internal Service Infrastructure:
• No Native K8s Services/DNS/Configs/Secrets/Network Policies: These are not used for applications due to the overhead of spanning hundreds of clusters and the existence of scaled, hardened internal equivalents at LinkedIn.
• Kubernetes as Pod Orchestrator: Kubernetes' primary role is to orchestrate pods, integrating with LinkedIn's established regional services for all other infrastructure needs.

Stateful on Kubernetes

Running stateful systems on Kubernetes, especially those using local disk for performance (e.g., Kafka, Espresso, Venice, Pino), is notoriously challenging.

• Generic Stateful Controller: Instead of requiring each data system team to write a Kubernetes controller, LinkedIn built a generic stateful controller.
• Shard Manager Integration: This controller integrates with existing shard managers (which data teams are experts at managing) via a custom protocol, handling pod updates, version updates, evictions, and maintenance coordination. This approach centralizes Kubernetes-specific logic while leveraging existing domain expertise.

Stateless on Kubernetes

• Simplified Spec: Users define their stateless applications with a simple 10-line spec for the LI deployment CR.
• Orchestration Capabilities: The CR offers various pod orchestration features like canary deployments (e.g., ramp to 10%, validate, then proceed).
• CloneSet Under the Hood: Instead of native Deployments, LinkedIn uses CloneSet (an open-source operator) because it supports volume claim templates (PVCs), which is not possible with native Deployments.
• Heavy Init Container Usage: The 10-line user spec is translated into a 500+ line pod spec, heavily leveraging init containers to integrate with the rest of LinkedIn's infrastructure. This includes creating default volume mounts and environment variables, ensuring applications get everything needed without modifications when migrating from legacy systems.

User Workflows and Experience

• Manifests in Git: Developers write manifests and check them into their repositories. Helm charts are published as part of the build process.
• Pre-Merge Validation:
• GitHub Actions: Mandatory fields (e.g., memory) are validated by GitHub Actions, blocking PRs if missing.
• Webhooks: Similar schema and policy checks are run via webhooks, providing immediate feedback in the PR before requests hit the API server, significantly reducing user support load.
• Namespace per Application: Each application gets its own namespace, forming the basis for RBAC permissions. If a namespace doesn't exist, it's created and routed to a cluster.
• Deployment Orchestration Service: Users interact with this service to deploy applications.
• Cluster Selection: The service determines the app name, tenant type (stateless/stateful), and desired node profile. A hub controller then finds a matching cluster pool or creates one, propagating the namespace and role bindings.
• Argo CD (Current/Phasing Out): The application version is passed to Argo CD (one per cluster), which syncs manifests from the Helm repo and applies them. However, Argo CD has shown scalability issues with high object counts and replica numbers, leading to slow sync times and health status updates.
• Custom GitOps Engine (Future): LinkedIn is developing its own GitOps engine to replace Argo CD for most app deployments, retaining Argo CD for some infrastructure components.
• Controllers and Regional Services: LinkedIn's controllers translate custom resources to pod specs, interacting with regional services for networking, configuration, and other needs.
• Observability: Logs and events from controllers and applications are streamed via Kafka to Azure Data Explorer. Default dashboards are created for every application, providing insights into CPU/memory usage, container restarts, and replica status.

Failures and Categorizations

• Granular Failure Feedback: To minimize support requests, LinkedIn provides specific failure categorizations.
• Validation Errors: Manifest errors are failed immediately with clear reasons.
• Terminal Errors: Failed on the first reconciliation, indicating a fundamental problem.
• Transient/Health Issues: Custom progress deadline seconds are implemented on CRs. If no progress is made within a set time, the deployment fails, categorized as an "app issue" or "infra issue."
• User-Readable Errors: These categorizations are propagated to object conditions and displayed in the UI and via the custom kubectl plugin, giving users exact reasons for deployment failures.

User Tools

• kubectl plugin: Application owners use a custom kubectl li plugin. It mimics kubectl semantics (e.g., kubectl li get pods, kubectl li status my-alli-deployment) but intelligently figures out where the app is running (across potentially hundreds of clusters) and scatters/gathers data for a unified view. The status command provides a summary including desired specs, health status, and reasons for unhealthiness.
• Custom UI: A regional system watches all custom resources, pods, and nodes, collecting data in a central database. A custom UI sits on top, providing a bird's-eye view for debugging without hitting API servers, thus aiding API server scalability.

API Guardrails

Extensive guardrails are in place to prevent accidental outages.

• Delete Protection: All user-facing custom resources, every namespace (even for admins), and internal custom resources have delete protection.
• Scale Down Limits: Operations that scale down applications beyond a certain percentage (e.g., 10%) in a single shot are prevented. The percentage is dynamic based on application size.
• Max Surge/Canary Limits: Upper bounds are enforced on max surge and canary percentages to prevent applications from exceeding their quotas.

Future Directions

LinkedIn's platform continues to evolve with several key initiatives:

• Federated Workloads Across Multiple Clusters: Move beyond running each application in a single cluster. This aims to:
• Align clusters with maintenance zones to prevent single cluster failures from taking down an entire application.
• Ensure all data plane or control plane rollouts impact only up to 5% of an application.
• Allow applications to grow beyond a single cluster's capacity.
• Better utilize fragmented machine types (e.g., 5 GPUs split across two clusters can be used by one federated workload).
• CPU Pinning: While applications can burst CPU by default, performance-critical applications need predictability. They are rolling out CPU pinning, specifically to physical cores (not logical cores) for better isolation.
• IPv6 VLAN-based Pod IPs (Pod CNI): Every pod will receive a globally routable IPv6 address within LinkedIn's production regions. This creates a flat network, eliminating concerns about cluster-to-cluster routing.
• Kubeception: Running Kubernetes control planes as pods within a management cluster to simplify cluster management and increase efficiency by stacking components.

Migration Lessons

The talk concluded with valuable lessons from their ongoing migration:

• Start Early, Make Incremental Progress: Migrating from a decade-old legacy system is complex; expect unexpected discoveries.
• First 50% is Easiest, Remaining 50% is a Long Tail: The initial migration phases are often straightforward, but the long tail of edge cases and specific requirements takes significantly longer.
• Be Pragmatic About Tech Debt: Don't try to solve all tech debt simultaneously; prioritize and choose battles wisely to ensure migration completion.
• Automated Container Image Generation: Instead of teaching every app owner to write Dockerfiles, an automated system generates Dockerfiles and container images on the fly.
• Intentional Kubernetes Feature Usage: Carefully select which Kubernetes features to use and expose, as raw Kubernetes may not scale or fit internal needs.
• Invest in Guardrails: Users will inevitably use APIs in unexpected ways; robust guardrails are crucial.
• Develop Good User Guides and Self-Serve Troubleshooting: These reduce support load significantly.

Demo / Proof of Concept

▶ Watch: Automated host health remediation and maintenance zones (4:00)

The talk focused on presenting the architectural design, implementation strategies, and operational workflows of LinkedIn's compute platform. While numerous custom tools, controllers, and user interfaces were described, the presentation did not include a live demonstration or a specific proof of concept. The speakers illustrated concepts using diagrams and descriptions of their systems in action, such as the automated maintenance flow or the user experience with their custom kubectl li plugin and UI.

Defensive Implications

▶ Watch: Coordinated, automated maintenance operations and "disruptions" (6:00)

LinkedIn's bare-metal Kubernetes strategy incorporates several robust defensive mechanisms and architectural choices that enhance security, reliability, and operational resilience.

1. Isolation through Bare Metal and Flat Networking: By running on bare metal with a flat network where every machine can talk to each other, LinkedIn maintains direct control over its entire network stack. This allows for the implementation of MTLS and authorization at the application layer, ensuring that even with a flat network, services can only communicate if they have explicit permissions and valid certificates/tokens. This removes reliance on potentially more complex and less performant overlay networks or network policies at the Kubernetes level for inter-service communication.

1. Automated and Zoned Maintenance for High Availability: The Maintenance Zones strategy is a critical defensive measure against widespread outages. By limiting the impact of any software update or maintenance operation to a maximum of 5% of the fleet, LinkedIn dramatically reduces the blast radius of faulty deployments (e.g., a bad kernel update). The automated Coordinated Maintenance Operations ensure consistent and controlled draining, upgrading, and re-rotation of machines, minimizing human error and ensuring rapid remediation of failed hosts.

1. Strict API Guardrails and Pre-Deployment Validation: The extensive use of GitHub Actions and webhooks for pre-merge validation, coupled with API guardrails like delete protection and scale-down limits, acts as a strong defense against misconfigurations and human errors. Preventing problematic manifests from even reaching the API server, or blocking destructive operations, significantly reduces the risk of self-inflicted outages, which are common in large-scale environments.

1. Granular Failure Categorization and Observability: The detailed failure categorization (validation, terminal, transient, app vs. infra issues) and out-of-the-box observability dashboards are crucial for rapid incident response. By clearly communicating the root cause of deployment failures to developers through custom UIs and kubectl plugins, the time to detect and resolve issues is drastically reduced, preventing prolonged service degradation.

1. Controlled Kubernetes Feature Exposure: By limiting user access to native Kubernetes resources and exposing only custom, high-level LI deployments and LI stateful sets, LinkedIn reduces the attack surface and potential for misconfiguration. This approach ensures that developers operate within well-defined and validated parameters, minimizing the chances of introducing security vulnerabilities or operational instability through direct manipulation of complex Kubernetes primitives.

1. Centralized Control Plane Management and RBAC: The use of a centralized hub cluster for managing other clusters and reviewing RBAC for custom operators provides a single point of control for security policy enforcement. API Priority and Fairness (APF) mechanisms further defend the control plane from abuse or overload by rogue or misbehaving applications.

1. Future Resilience with Federated Workloads and CPU Pinning: The planned move to federated workloads across multiple clusters, aligned with maintenance zones, will further enhance resilience by distributing application blast radii across fault domains. CPU pinning to physical cores will provide stronger performance isolation, mitigating noisy neighbor problems that can impact performance and potentially lead to resource exhaustion vulnerabilities for critical applications. The global IPv6 VLAN-based Pod IPs also simplify network security by providing a consistent, routable identity for every pod, making network policy management more straightforward in a flat network.

Key Takeaways

• Bare-metal Kubernetes at scale demands extensive customization: LinkedIn’s experience highlights that running Kubernetes on hundreds of thousands of bare-metal servers requires a bespoke Infrastructure as a Service (IaaS) layer and custom management tooling, such as the Compute Broker and Maintenance Orchestrator, to achieve performance and operational control.
• Kubernetes can be a focused pod orchestrator, not an all-encompassing platform: LinkedIn intentionally bypasses many native Kubernetes features (Services, DNS, Configs, Secrets, Network Policies) in favor of its existing, hardened, and scaled internal regional services, demonstrating a pragmatic approach to leveraging Kubernetes' strengths while integrating with established enterprise infrastructure.
• Developer experience is paramount, necessitating abstraction and guardrails: To serve thousands of engineers, LinkedIn built high-level Custom Resources (CRs) like LI deployments and LI stateful sets, abstracting complex Kubernetes details. Extensive pre-deployment validation, API guardrails, and clear failure categorization are critical for preventing errors and reducing support burden.
• Control plane scalability, especially etcd and controllers, is a continuous challenge for large clusters: Pushing clusters to 5,000+ nodes strains core Kubernetes components. LinkedIn's efforts include increasing etcd storage, using APF, and actively working on horizontal controller scaling solutions, indicating that this remains an active area of development.
• Automated, zoned maintenance is fundamental for bare-metal fleet reliability: The implementation of Maintenance Zones and Coordinated Maintenance Operations ensures that fleet-wide upgrades and remediation are automated and controlled, limiting impact to a small percentage (5%) of capacity and significantly improving overall system resilience.
• Strategic migration requires incremental progress and pragmatic tech debt management: LinkedIn's migration lessons emphasize starting early, making incremental progress, and being selective about which legacy tech debt to address, recognizing that the "long tail" of migration can be the most challenging.

About the Speaker(s)

Ahmet Alp Balkan is an engineer on the Central Compute Team at LinkedIn, based out of Seattle. A seasoned attendee and speaker, this was his seventh or eighth KubeCon and his third time giving a talk. Outside of his work on Kubernetes, Ahmet enjoys gardening and house plants. He is also known for developing kubectl plugins, some of which may be in use by the Kubernetes community.

Ronak Nathani is also an engineer on the Central Compute Team at LinkedIn, located in Toronto. This was Ronak's third KubeCon and his first time speaking at the conference, having attended his first KubeCon in Detroit in 2022. Beyond his contributions to Kubernetes, Ronak is an enthusiast of racket sports, particularly badminton, and hosts a podcast called "Soft Misadventures."