Building & Operating a Large-scale HPC AI Cluster on Kubernetes - Kalyan Saladi & Chandan Avdhut

Building & Operating a Large-scale HPC AI Cluster on Kubernetes - Kalyan Saladi & Chandan Avdhut

Speakers: Kalyan Saladi, Software Engineer, Research Infrastructure Team, Meta; Chandan Avdhut, Production Engineer, Meta

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=7KCBigZi_Rk

Overview

This KubeCon EU talk, presented by Chandan Avdhut and Kalyan Saladi from Meta, delves into the intricate process of constructing and operating a high-performance computing (HPC) AI cluster at a massive scale within a public cloud environment, leveraging Kubernetes. The speakers share Meta's journey from custom on-premise infrastructure to a cloud-native, multi-provider solution, highlighting the unique challenges posed by large-scale machine learning (ML) training workloads and how Kubernetes was adapted to meet these demands. The core focus is on maintaining near bare-metal performance, ensuring reliability for long-running, fault-intolerant jobs, and crucially, preserving an uncompromised researcher experience.

The presentation outlines the critical requirements for AI/ML training infrastructure, which fundamentally differ from traditional web services. These include extended job lifetimes (hours to months), "all-or-nothing" job semantics where a single node failure can restart an entire multi-GPU job, and the need for consistent configurations across hundreds or thousands of GPUs. Meta’s solution demonstrates a sophisticated hybrid architecture that integrates existing HPC tools like Slurm with Kubernetes' orchestration capabilities, providing a scalable, resilient, and developer-friendly platform for cutting-edge AI research.

The talk is particularly relevant for infrastructure engineers, cloud architects, and ML platform developers grappling with the complexities of deploying and managing large-scale AI training environments. It provides practical insights into tackling issues such as node consistency, dynamic storage provisioning, observability, and abstracting underlying infrastructure from researchers, ultimately accelerating innovation in the rapidly evolving field of artificial intelligence.

Background

▶ Watch: Introduction to AI HPC on Kubernetes (0:00)

Machine learning training, particularly for large models like Llama, Mistral, and GPT, involves iteratively computing and refining model weights on vast datasets, often utilizing hundreds or thousands of GPUs. These workloads exhibit distinct characteristics that challenge traditional infrastructure paradigms. Unlike stateless web services with millisecond response times, ML training jobs can run for hours, weeks, or even months. Furthermore, they often adhere to "all-or-nothing" semantics; if even one GPU in a 16-GPU job becomes unhealthy, the entire job is typically impacted and may need to restart, making reliability paramount. This is a stark contrast to web services where individual node failures can often be retried on healthy nodes without broader impact.

The ML research lifecycle itself is highly iterative. Researchers move from understanding state-of-the-art models, to experimentation (ablations, hyperparameter sweeps, new architectures), and finally to large-scale training runs on massive GPU clusters. Throughout this process, researcher experience is paramount to unlocking innovation. Key aspects include maximizing iteration speed, clearly attributing failures (distinguishing between research code bugs and infrastructure issues), providing first-class monitoring, and ensuring training reliability by minimizing interruptions for long-running jobs at scale.

Distributed training reliability presents significant hurdles. All worker nodes must maintain consistent configurations to ensure predictable outcomes. The entire compute, storage, and network stack must be highly reliable, as any weakness can lead to reduced productive training time. A non-obvious aspect is the "lockstep" nature of many training approaches, which can result in massive load spikes on central components, such as storage servers during checkpointing, followed by periods of near-zero activity. Designing infrastructure to handle these dramatic fluctuations is critical. Moreover, attributing failures in a multi-week, multi-thousand GPU job is expensive and requires robust observability to pinpoint the root cause.

Meta's journey began with custom-built on-premise clusters launched in 2021-2022. These clusters comprised thousands of DGX servers running CentOS, orchestrated by Slurm, the widely adopted HPC scheduler. They featured directly managed backend network fabric, multiple petabytes of flash storage, and purpose-built storage services optimized for training acceleration. While offering direct control and performance, the challenge was to replicate or even improve upon this experience in the public cloud, meeting requirements for near bare-metal performance, supporting long-running jobs with heavy data exchange and demanding data loading, and isolating faulty nodes rapidly to shield workloads from infrastructure failures.

Key Findings

▶ Watch: Key Differences: ML Training vs. Traditional Workloads (2:20)

Meta's transition to building and operating large-scale HPC AI clusters on Kubernetes in the public cloud yielded several key findings and architectural shifts designed to address the unique demands of ML training:

1. Kubernetes as the Orchestration Layer: The initial approach in the public cloud, utilizing Meta Instance (a containerized layer on cloud provider images) with slumd running directly on the host, proved unscalable across multiple clusters and cloud providers. Kubernetes emerged as the obvious choice due to its industry standardization and multi-cloud compatibility, streamlining infrastructure management.

1. Hybrid Cluster Architecture: The evolved Kubernetes architecture splits the cluster into two logical parts. A "right-hand side" hosts common, reusable infrastructure components like ingress and ingress gateways on Kubernetes, which can be deployed uniformly across different cloud providers. The "left-hand side" comprises the GPU cluster, login environments, and specific telemetry components, also running on Kubernetes but allowing for customization based on cloud provider specifics or hardware requirements. This split provides both reusability and necessary flexibility, connected by L3 network connectivity.

1. Preserving Researcher Experience via Abstraction: To avoid forcing researchers to learn Kubernetes-specific commands (Helm, kubectl), Meta developed a custom CLI. This CLI, upon a researcher's login request, dynamically creates a Helm release Custom Resource Definition (CRD). A Flux Helm controller within the cluster then uses this CRD to provision a dedicated login pod. Researchers SSH into this pod and interact with the cluster using familiar Slurm commands, completely abstracted from the underlying Kubernetes complexity.

1. Enhanced Security and Isolation: The custom CLI and dynamic login pod creation, combined with a Kyon admission controller (a validating webhook), significantly improved the security posture. Researchers now get dedicated pods with allocated resources, eliminating the "noisy neighbor" problem common in shared VM environments. The admission controller enforces policies, preventing unauthorized privilege escalation or impersonation.

1. Automated Node Consistency and Reliability: Maintaining consistent configurations and image versions across hundreds or thousands of nodes is critical for distributed training. Meta achieved this through Helm deployment automation. Any change to a Helm chart or image triggers a rolling update process: nodes are drained by Slurm (allowing existing jobs to finish), the slumd pod is replaced with the new version, and then the node is undrained and returned to service. This ensures consistency and minimizes disruption.

1. Dynamic Storage Management: For data-intensive AI workloads, flexible and consistent access to various storage options (POSIX, object storage, AI-accelerated) is essential. While object storage clients required no changes, NFS mounts on Kubernetes typically require pod restarts for new volume mounts. Meta addressed this with an automount DaemonSet. This DaemonSet runs on every node, dynamically mounting NFS volumes into categorized parent directories on the host. These parent directories are then bind-mounted into the slumd and login pods, making new file systems available without disrupting running workloads.

1. Comprehensive Observability: A robust observability story is crucial for distinguishing between application (research) failures and infrastructure failures. Meta implemented a combination of open source, vendor-provided, and custom-built telemetry components. These include job metrics, a core dumper for unexpected process terminations, and custom Kubernetes controllers for user/service identity and integrating Cert Manager with Meta's private CA for X.509 certificates. This allows for rapid triage and root cause analysis.

Technical Deep Dive

▶ Watch: Overview of the ML Research Life Cycle (4:00)

Meta's journey to a Kubernetes-native HPC AI cluster involved several architectural iterations and custom components to marry the demands of large-scale ML with cloud infrastructure.

Initial Cloud Deployment and the Shift to Kubernetes:

Initially, Meta deployed its AI HPC infrastructure in the public cloud using Meta Instance, a custom machine image built atop cloud provider host images. This image provided a containerized layer for managing Meta's infrastructure components. While control plane elements were easily containerized, the Slurm daemon (slumd) running on GPU nodes presented a challenge. Due to its need for direct host access for resource allocation, cgroup manipulation, and bare-metal control, slumd was initially run directly on the host OS rather than within a container. As Meta's public cloud footprint grew across multiple clusters and providers, this approach proved unscalable. The need for a more robust, standardized, and multi-cloud compatible container orchestration mechanism led to the adoption of Kubernetes.

Evolved Hybrid Kubernetes Architecture:

The mature architecture is divided into two distinct logical segments, connected via L3 network connectivity:

1. Common Infrastructure Cluster (Right-Hand Side): This segment hosts reusable infrastructure components like ingress gateways and other common services. Running on Kubernetes, these components are largely cloud-provider agnostic and can be deployed consistently across various environments without significant customization.
2. GPU Compute Cluster (Left-Hand Side): This segment is where the actual GPU compute power resides, along with login environments, control plane components specific to the AI workloads, and all necessary telemetry infrastructure. While also running on Kubernetes, this side allows for specific customizations tailored to the underlying cloud provider's hardware (e.g., specific GPU types, network configurations) and the unique needs of the AI clusters.

This split enables efficient reuse of common infrastructure while maintaining the flexibility required for specialized HPC environments.

Researcher Experience and Abstraction:

A core design principle was to avoid disrupting the researcher's established workflow, which heavily relied on Slurm commands. To achieve this, Meta implemented a custom CLI. When a researcher wishes to log into the cluster:

1. The custom CLI creates a small Helm release Custom Resource Definition (CRD).
2. A Flux Helm controller, running within the Kubernetes cluster, detects this CRD.
3. Dynamically, a dedicated login pod is created for the researcher.
4. An SSH session is then launched directly into this login pod.

From the researcher's perspective, they simply use their familiar Slurm commands, unaware of the underlying Kubernetes orchestration.

Security and policy enforcement for these login pods are managed by a Kyon admission controller. This validating webhook ensures that the Helm release objects conform to predefined policies, preventing actions like requesting elevated privileges or impersonating other users. This approach provides dedicated resources, improved security posture, and eliminates the "noisy neighbor" problem often found in shared login environments.

Infrastructure Component Management on Kubernetes:

Beyond the core Slurm functionality, various infrastructure components are managed as Kubernetes deployments or DaemonSets:

• cisco DaemonSet: This DaemonSet runs on every node, configuring essential kernel parameters required for optimal performance of Meta's AI workloads.
• automount DaemonSet: A critical component for storage, this DaemonSet ensures consistent mounting of NFS volumes across all hosts. It categorizes file systems (e.g., checkpoints, datasets, infra) and mounts them into corresponding parent directories on the host. These parent directories are then bind-mounted into the slumd and login pods. This innovative approach allows new file systems to be added or removed dynamically without requiring pod restarts, crucial for uninterrupted long-running AI jobs.
• Telemetry Components: A comprehensive observability stack integrates open source tools, vendor-provided solutions, and custom-built components. Examples include custom job metrics and a core dumper to capture state during unexpected process terminations.
• Custom Kubernetes Controllers: Meta developed custom controllers for specific internal needs, such as user identity and service identity management. They also integrated Cert Manager (an open-source tool for managing certificates on Kubernetes) with their internal X.509-based private CA to enforce custom certificate policies. These controllers allow seamless integration with Meta's existing internal tooling and infrastructure.

Node Consistency and Reliability:

Maintaining a consistent environment across hundreds of GPU nodes is paramount. Meta's Helm deployment automation plays a central role:

1. Any update to a Helm chart or container image triggers a rolling update process.
2. The Kubernetes controller initiates a drain on slumd nodes. During this period, Slurm prevents new jobs from being allocated and allows existing jobs to complete.
3. Once a node is drained, the slumd pod is replaced with the new version.
4. The node is then undrained, making it available again for new workloads.

This ensures that all nodes run the latest, consistent software versions without abrupt disruptions.

For detecting and isolating faulty nodes, Meta employs a combination of telemetry health checks running in the Kubernetes control plane and within Slurm itself. These checks continuously monitor node health. Detected faulty nodes are tracked, allowing support teams to triage and, if necessary, work with cloud providers for remediation or replacement. The team also tracks failures on specific nodes across various jobs to identify persistently "bad actors," even considering advanced tracking methods using GPU IDs and serial IDs to account for cloud providers potentially returning the same physical hardware with a different host identity.

Demo / Proof of Concept

▶ Watch: Researcher Experience as Paramount Design Principle (6:00)

The talk focuses on the architectural design, implementation strategies, and operational aspects of building and managing a large-scale HPC AI cluster on Kubernetes. While highly technical and detailed, it does not feature a live demonstration or proof of concept of a specific tool or exploit. Instead, the speakers present the overall system, its components, and their functional interactions through diagrams and explanations of how Meta solved various challenges in a production environment.

Defensive Implications

▶ Watch: Challenges in Distributed Training Reliability (7:50)

The strategies and architecture presented by Meta offer several critical defensive implications for organizations operating or planning to deploy large-scale AI/HPC clusters, particularly in cloud environments:

1. Embrace Kubernetes for Scalability and Management: For any organization aiming for large-scale, multi-cloud AI infrastructure, Kubernetes provides the necessary orchestration, standardization, and flexibility. It centralizes infrastructure management, allowing teams to scale efficiently and iterate faster on new features.

1. Prioritize Researcher Experience through Abstraction: Developers and researchers should not be burdened by the complexities of the underlying orchestration platform. Implementing custom CLIs and dedicated login environments (like Meta's dynamic login pods) that leverage existing familiar tools (e.g., Slurm) significantly improves productivity and reduces the friction of adopting new infrastructure.

1. Implement Robust Node Health Management: Given the "all-or-nothing" nature of many ML jobs, proactive identification and isolation of faulty nodes are paramount. Comprehensive telemetry, health checks (both at the Kubernetes and application/Slurm level), and automated rolling updates for node consistency are essential. Tracking "bad actors" by hardware identifiers (e.g., GPU IDs) helps prevent recurring failures from problematic hardware.

1. Design for Load Spikes in Distributed Systems: ML training, especially with checkpointing, can create sudden, massive load spikes on central components like storage. Infrastructure designs must account for these bursty demands to maintain performance and avoid bottlenecks. This includes optimizing storage services and network fabric.

1. Build a Comprehensive Observability Stack: A strong observability story is critical for distinguishing between application-level failures (researcher code bugs) and infrastructure failures. Integrating open-source tools, cloud provider metrics, and custom components (e.g., job metrics, core dumpers) enables rapid root cause analysis, minimizing costly debugging time for long-running jobs.

1. Enforce Security Policies with Admission Controllers: Leveraging Kubernetes admission controllers (like Kyon) is vital for enforcing security policies, preventing unauthorized actions (e.g., privilege escalation, impersonation) within the cluster, and ensuring a secure multi-tenant environment for researchers.

1. Adopt Dynamic Storage Solutions: Data-intensive AI workloads require flexible and performant storage. Solutions like the automount DaemonSet for dynamic NFS volume management allow new data sources to be introduced without disrupting ongoing training jobs, a significant operational advantage.

1. Leverage DaemonSets for Host-Level Configuration: For tasks requiring host-level configuration (e.g., kernel parameter tuning), DaemonSets provide a Kubernetes-native way to ensure consistent application across all nodes, integrating these specific requirements into the overall Kubernetes management plane.

Key Takeaways

• Kubernetes Enables Scalable Multi-Cloud HPC AI: Meta successfully transitioned from custom on-prem clusters to a multi-cloud Kubernetes-based architecture, demonstrating that Kubernetes can effectively orchestrate large-scale, high-performance AI training workloads without sacrificing performance or reliability.
• Researcher Experience is Paramount: Abstracting Kubernetes complexity via custom CLIs and maintaining familiar workflows (e.g., Slurm commands) is crucial for research velocity, allowing scientists to focus on innovation rather than infrastructure.
• Reliability in Distributed ML Demands Consistency and Robustness: Long-running, fault-intolerant AI jobs require consistent node configurations, highly reliable compute, storage, and network components, and sophisticated mechanisms for rapidly identifying and isolating faulty nodes.
• Hybrid Architectures Bridge Gaps: Combining Kubernetes for overall infrastructure management with specific components (like slumd initially, or later, host-level DaemonSets for kernel tuning) that interact directly with the host OS can effectively address unique HPC requirements.
• Dynamic Storage Management is Essential: For data-intensive AI workloads, flexible storage solutions like the automount DaemonSet, which allow on-the-fly addition/removal of file systems without pod restarts, are critical for operational efficiency.
• Comprehensive Observability is Non-Negotiable: A robust observability stack, integrating open-source, vendor, and custom telemetry, is vital for quickly attributing failures (infra vs. application) and maintaining high cluster uptime and productivity.

About the Speaker(s)

Chandan Avdhut is a Production Engineer at Meta. He co-presented this talk and is actively involved in the operational aspects of building and maintaining large-scale AI HPC clusters for research purposes in the public cloud.

Kalyan Saladi is a Software Engineer in the Research Infrastructure Team at Meta. While unable to attend KubeCon EU in person, his pre-recorded segment provided foundational insights into ML training, researcher experience, and the initial architectural considerations for Meta's cloud-based HPC AI clusters. Both Kalyan and Chandan collaborate across different teams at Meta to design, build, and operate these critical infrastructure platforms.