Beyond Kubernetes: Adapting To Spe... Rags Srinivas, Dawn Chen, Sachi Desai, Vara Bonthu & Erin Boyd

Beyond Kubernetes: Adapting To Specialized Application Workloads

Speakers: Rags Srinivas, Software Engineer; Dawn Chen, Software Engineer and Tech Lead at Google; Sachi Desai, Product Manager on Azure Kubernetes Service; Vara Bonthu, Principal Open Source Specialist SA at AWS; Erin Boyd, Senior Director and Distinguished Engineer at NVIDIA

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=jtGSzIvw9jI

Overview

This insightful panel discussion from KubeCon EU delves into the critical evolution of Kubernetes to effectively support specialized, high-performance application workloads, particularly in the domains of Artificial Intelligence (AI), Machine Learning (ML), and High-Performance Computing (HPC). Featuring leading experts from Google, Azure, AWS, and NVIDIA, the talk explores the current state, inherent challenges, and future directions for adapting Kubernetes from its origins as a platform for stateless microservices to a robust orchestrator for highly demanding, stateful, and resource-intensive computations.

The speakers collectively highlight the increasing demand from customers across various industries—from automated financials to retail and supply chain—to run their complex AI/ML workloads on Kubernetes. The discussion underscores the significant strides being made within the Kubernetes community and by major cloud providers to address the unique requirements of these applications, such as specialized hardware management, advanced scheduling paradigms, and robust data handling.

This article aims to provide a comprehensive technical overview of the panel's discussion, outlining the foundational shifts necessary within Kubernetes, the innovative projects underway, and the strategic vision of industry leaders to meet the escalating demands of the AI/ML and HPC landscape. It is a crucial read for anyone involved in deploying, managing, or developing applications on Kubernetes that leverage specialized compute resources and complex batch processing.

Background

▶ Watch: Introduction and panelist introductions (0:00)

Kubernetes, originally designed to orchestrate stateless web services and microservices, has undergone a significant transformation since its inception. Early iterations excelled at managing ephemeral containers, scaling horizontally, and providing self-healing capabilities for applications with minimal state. However, the burgeoning fields of AI, ML, and HPC present an entirely different set of requirements that push the boundaries of Kubernetes' traditional design.

Historically, these specialized workloads were often deployed on dedicated infrastructure such as HPC clusters or Hadoop environments for large-scale data processing. These environments offered specialized schedulers, direct hardware access, and optimized communication fabrics tailored for distributed, tightly coupled computations. The shift towards containerization and the desire for a unified orchestration platform have led many organizations to explore Kubernetes for these workloads. This transition is driven by Kubernetes' promise of consistent deployment, operational efficiency, and simplified resource management across heterogeneous environments.

The panel emphasizes that while Kubernetes has successfully evolved to support stateful workloads through features like Persistent Volumes (PVs) and Persistent Volume Claims (PVCs), integrating AI/ML and HPC applications introduces new layers of complexity. These include the need for specialized hardware like GPUs and TPUs, which require fine-grained resource allocation and topology awareness. Furthermore, AI/ML training jobs often involve complex batch workloads with inter-job dependencies, "all-or-nothing" scheduling requirements (where all components of a distributed job must start simultaneously), and robust mechanisms for detecting and recovering from hardware failures without losing significant progress. The challenge lies in adapting Kubernetes' core primitives and ecosystem to natively understand, schedule, and manage these unique demands, moving beyond simple resource requests to a more intelligent, workload-aware orchestration.

Key Findings

▶ Watch: NVIDIA's perspective on diverse AI/ML and HPC workloads (2:32)

The panel discussion unveiled several key findings regarding the adaptation of Kubernetes for specialized AI/ML and HPC workloads:

1. Kubernetes is Actively Evolving for AI/ML/HPC: The community, led by efforts in SIG Node (as highlighted by Dawn Chen from Google), is making significant progress in evolving Kubernetes to support these demanding workloads. This marks a departure from its initial focus on stateless microservices, recognizing the growing importance of AI/ML and HPC in enterprise environments.

1. Standardization of Hardware Abstraction is Critical: A major initiative is the Device Resource Allocation (DRA) project. DRA aims to standardize how specialized hardware (like GPUs and TPUs) is abstracted and exposed to Kubernetes. This allows the scheduler to intelligently manage and allocate these precious resources, moving towards a more uniform way for Kubernetes to interact with diverse hardware accelerators. While in the production stage, challenges remain around migration and monitoring.

1. Batch Workloads Require New Primitives: Kubernetes' existing primitives are not fully optimized for complex batch workloads, especially those with intricate dependencies and "all-or-nothing" (gang) scheduling requirements. The panel identified a need for new core objects or standardized ways to describe job workflows and part groups, enabling more efficient scheduling and management of these distributed computations.

1. Ecosystem Tools are Bridging Gaps: A vibrant ecosystem of tools and projects has emerged to address Kubernetes' current limitations for AI/ML/HPC. These include custom schedulers like Volcano and Apache Unicorn, job queuing systems, and frameworks like KubeFlow. NVIDIA's contributions, such as Kaduner and the newly open-sourced Kai scheduler (supporting fractional GPU), exemplify industry efforts to provide specialized solutions. The Kubernetes AI Tool Chain Operator (Kaido), a CNCF sandbox project, aims to create a composable architecture for integrating these tools.

1. Hyperscalers See Diverse AI Adoption and Unique Challenges:

• NVIDIA (Erin Boyd): Serves a diverse customer base, from biological models to robotics, with extensive training and inference workloads. They observe the convergence of Kubernetes, AI, and HPC, emphasizing the need for efficient GPU management. NVIDIA is increasing its open-source contributions (e.g., Skyhook for on-the-fly NIC parameter updates, NV Sentinel for node health monitoring) to support the community.
• AWS (Vara Bonthu): Sees widespread AI/ML adoption across verticals (finance, retail, supply chain) on Amazon EKS. Customers leverage open-source solutions for custom scheduling, autoscaling, and various AI tasks. Challenges include GPU/CPU optimization, failover for distributed parts, and scaling Kubernetes for bursty batch workloads beyond 5,000-10,000 nodes, which stresses control plane components like etcd and API servers.
• Azure (Sachi Desai): Supports users at all stages of their AI journey, from experimentation and inferencing to advanced model training and RAG (Retrieval Augmented Generation). Focus is on composable architecture, integrating with observability tools, and supporting node reliability for long-running training jobs to reduce cost and ensure progress.
• Google (Dawn Chen): Unique position with its own hardware (TPUs) and large language models (Gemini). Focuses on standardizing hardware and frameworks on GKE to support both internal and external customers, addressing challenges in migration and unified management.

1. GPU Management and Reliability are Paramount: GPUs are expensive, limited resources. The panel highlighted the need for better GPU sharing (including fractional GPU usage), faster detection of GPU hardware problems, and dynamic hot-plugging/detaching of GPUs to improve resource utilization and reduce downtime for training jobs.

1. Standardization is the Future: A recurring theme is the urgent need for standardization—across hardware interfaces, scheduling parameters, auto-scaling signals, and even high-level workflow descriptions. This is seen as crucial to prevent fragmentation within the Kubernetes ecosystem, improve efficiency, and ensure elasticity for customers, allowing them to leverage familiar tools (like Slurm APIs) while benefiting from Kubernetes' underlying orchestration.

Technical Deep Dive

▶ Watch: Azure's support for users across their AI journey (4:50)

The technical discussion centered on the specific mechanisms and projects enabling Kubernetes to handle the unique demands of AI/ML and HPC workloads.

A foundational element is the Device Resource Allocation (DRA) project. As Dawn Chen explained, DRA aims to standardize and abstract specialized hardware like GPUs and TPUs within Kubernetes. This abstraction is crucial for the Kubernetes scheduler to effectively manage and allocate these devices to pods. Instead of direct device plugin interactions, DRA provides a more declarative and extensible framework, allowing for richer device capabilities to be exposed and consumed. While DRA is nearing production readiness, the panel noted ongoing challenges related to migrating existing workloads to this new model and establishing robust monitoring for device health and utilization within the DRA framework.

Scheduling is another area undergoing significant transformation. Traditional Kubernetes scheduling is designed for general-purpose workloads, but AI/ML training jobs often require gang scheduling or co-scheduling, where a group of pods must be scheduled simultaneously, or none at all. This "all-or-nothing" approach prevents deadlocks and ensures efficient resource utilization for distributed applications. To address this, scheduler plugins within the kube-scheduler framework are being developed. Sachi Desai mentioned proposed plugins for gang scheduling, co-scheduling, and topology-aware scheduling. Topology-aware scheduling is vital for HPC and AI/ML to place pods on nodes that optimize data locality, network latency, and inter-GPU communication paths, leading to significant performance gains.

Custom schedulers like Apache Unicorn and Volcano have emerged in the ecosystem to provide these advanced scheduling capabilities. Vara Bonthu highlighted their use for bin packing workloads, optimizing GPU and CPU utilization by efficiently placing multiple smaller workloads onto available resources. NVIDIA has also open-sourced its Kai scheduler, which specifically supports fractional GPU allocation, allowing a single physical GPU to be logically divided and shared among multiple pods, thus improving utilization of expensive GPU hardware.

Resource management for specialized hardware extends beyond simple allocation. The panel stressed the need for fine-grained resource management and better GPU optimization. This includes not just fractional GPU usage but also efficient packing of workloads to minimize idle GPU cycles. Erin Boyd introduced NV Sentinel, an open-source project from NVIDIA focused on monitoring node health, specifically for GPUs. NV Sentinel goes beyond a simple "good or bad" status, providing nuanced insights into GPU health. This is critical for cloud providers to quickly diagnose issues, remove unhealthy nodes from service, and bring them back online efficiently, minimizing disruption to long-running training jobs.

Data management for AI/ML workloads on Kubernetes also presents unique challenges. Vara Bonthu noted that while Kubernetes has improved support for stateful workloads with PVs/PVCs and integration with distributed file systems like Amazon EFS and FSx, there's still a gap in caching capabilities. Customers often duplicate entire datasets across nodes for training, leading to inefficiencies. The need for optimized data sharding and distributed caching clusters was emphasized to streamline data access and reduce training times, especially when streaming data from sources like S3.

Scalability remains a persistent concern. While Kubernetes can scale to thousands of nodes for stateless microservices, stateful data and AI/ML workloads can push the control plane to its limits. Vara Bonthu observed that running 30,000-50,000 pods for bursty batch workloads can introduce bottlenecks in etcd and API servers, as Kubernetes was not originally designed for such intense, high-churn stateful operations. This necessitates significant tuning and ongoing architectural improvements within Kubernetes to handle extreme scales for these specialized workloads.

A pervasive theme throughout the deep dive was the call for standardization. Dawn Chen articulated a vision for standardizing various layers: the hardware layer (via DRA), the scheduling layer (with new parameters for efficient scheduling), and the auto-scaling layer (with primitives that signal cloud providers for node scaling based on workload needs). The goal is to avoid fragmentation of Kubernetes by numerous custom resource definitions (CRDs) and schedulers. While customization can be beneficial, excessive fragmentation can reduce efficiency and elasticity. The panel advocated for standardizing group-of-pods descriptions for workflows, allowing custom schedulers to operate against these common abstractions. This would enable users to leverage familiar tools and APIs, like Slurm, while benefiting from Kubernetes' underlying orchestration, decoupling the core Kubernetes layer from specific workload management details.

Demo / Proof of Concept

▶ Watch: Google's unique position with TPUs and LLMs (5:55)

This panel discussion was a high-level strategic and technical conversation among industry leaders. As such, it did not include a live demonstration or a specific proof of concept of any particular tool or feature. The speakers focused on discussing ongoing projects, challenges, and future directions rather than showcasing implemented solutions.

Defensive Implications

▶ Watch: Where Kubernetes needs to evolve beyond current progress (8:00)

For organizations leveraging or planning to leverage Kubernetes for AI/ML and HPC workloads, the panel discussion provides several critical defensive implications:

1. Prioritize Specialized Hardware Monitoring: Given the expense and criticality of GPUs and TPUs, defenders must implement robust monitoring solutions. Tools like the NVIDIA DCGM exporter (mentioned by Sachi Desai) are essential for gathering detailed metrics on GPU health, utilization, temperature, and performance. This data should be integrated into existing observability stacks (e.g., Prometheus and Grafana) to proactively detect anomalies, predict failures, and optimize resource allocation. The development of tools like NV Sentinel highlights the need for nuanced node health insights beyond simple up/down status.

1. Adopt Advanced Scheduling Strategies: Relying solely on the default Kubernetes scheduler for AI/ML/HPC workloads is insufficient. Defenders should explore and implement advanced scheduling plugins and custom schedulers that support gang scheduling (for distributed training jobs), topology-aware scheduling (to optimize data and network locality), and bin packing (for efficient resource utilization). Solutions like Volcano, Apache Unicorn, and NVIDIA's Kai scheduler (for fractional GPU) are key examples that should be evaluated and integrated.

1. Optimize Data Access and Caching: Data bottlenecks can severely impact AI/ML training performance. Defenders need to ensure that their Kubernetes deployments leverage distributed file systems (e.g., EFS, FSx) effectively with PVs/PVCs. More importantly, investing in or advocating for better data caching capabilities across nodes is crucial to avoid redundant data duplication and accelerate training. This might involve exploring specialized data storage operators or caching layers within the Kubernetes ecosystem.

1. Design for Workload Resilience and Recovery: Long-running AI/ML training jobs are vulnerable to hardware failures. Defenders must architect their applications to be resilient to GPU or node failures. This includes implementing checkpoints, restart mechanisms, and leveraging Kubernetes features that support workload migration. The ability to fast-detect hardware problems (a request from Dawn Chen) and perform smart re-placement of failed GPU/TPU pods is vital to minimize lost progress and associated costs.

1. Stay Abreast of Kubernetes Core Evolution: The Device Resource Allocation (DRA) project and other core Kubernetes primitives for batch workloads are fundamental changes. Defenders should monitor their development, understand their implications, and plan for adoption to benefit from standardized hardware management and improved scheduling capabilities. Early engagement with the Kubernetes community can also help shape these features.

1. Leverage Fractional GPU and Resource Optimization: GPUs are expensive. Implementing fractional GPU allocation (as offered by NVIDIA's Kai scheduler) can significantly improve resource utilization and reduce costs by allowing multiple workloads to share a single GPU. Defenders should also focus on efficient bin packing of CPU and GPU resources to maximize throughput on their clusters.

1. Advocate for and Adopt Standardization: The panel strongly emphasized the need for standardization across hardware, scheduling, and workflow definitions. Defenders should prefer solutions that adhere to emerging standards or contribute to their development, as this will prevent fragmentation, improve interoperability, and ensure long-term elasticity and manageability of their AI/ML/HPC infrastructure on Kubernetes.

Key Takeaways

• Kubernetes is rapidly evolving to become a robust platform for AI/ML and HPC workloads, moving beyond its initial focus on stateless microservices, driven by community efforts and hyperscaler investments.
• Standardization of specialized hardware management, particularly for GPUs and TPUs, is critical, with projects like Device Resource Allocation (DRA) leading the way to enable more intelligent scheduling and abstraction.
• Advanced scheduling capabilities are essential for complex batch workloads, including gang scheduling, topology-aware scheduling, and fractional GPU allocation, which are being addressed by custom schedulers (e.g., Volcano, Apache Unicorn, NVIDIA Kai scheduler) and scheduler plugins.
• Effective data management, including optimized caching and distributed file systems, is crucial to overcome bottlenecks and improve the performance of large-scale AI/ML training jobs on Kubernetes.
• Robust monitoring of specialized hardware health (e.g., using NVIDIA DCGM exporter and NV Sentinel) and designing workloads for resilience and fast recovery from hardware failures are paramount for cost efficiency and successful long-running computations.
• The industry is striving for greater standardization across hardware interfaces, scheduling parameters, and workflow descriptions to prevent fragmentation, improve efficiency, and ensure elasticity for customers deploying AI/ML/HPC on Kubernetes.

About the Speaker(s)

• Rags Srinivas: Served as the moderator for the panel. While his specific role was not detailed beyond "this is us," his presence suggested a role as a community leader or expert in the Kubernetes ecosystem.
• Dawn Chen: A Software Engineer and Tech Lead at Google, Dawn has been involved with Kubernetes since its inception. She currently leads the SIG Node community, focusing on the evolution of Kubernetes to support specialized workloads such as AI/ML and HPC.
• Sachi Desai: A Product Manager on the Azure Kubernetes Service (AKS) team, Sachi's expertise lies in AI, GPU, and HPC deployments on AKS and Kubernetes. She is actively involved in the Kubernetes AI Tool Chain Operator CNCF sandbox project.
• Vara Bonthu: A Principal Open Source Specialist SA at AWS, Vara focuses on scaling data and ML workloads on Amazon EKS. His work involves building data pipelines using frameworks like Spark, Flink, and Trino, and developing ML platforms on Kubernetes.
• Erin Boyd: A Senior Director and Distinguished Engineer at NVIDIA, Erin leads the Kubernetes team for DGX Cloud. She is a long-time contributor to open source, particularly within the CNCF, and is dedicated to advancing GPU management and integration within the Kubernetes ecosystem.