Unlocking How To Efficiently, Flexibly, Manage and Schedule Seven AI Chi... Xiao Zhang & Mengxuan Li

Unlocking How To Efficiently, Flexibly, Manage and Schedule Seven AI Chi... Xiao Zhang & Mengxuan Li

Speakers: Xiao Zhang, Software Engineer, Dynamia Point AI; Mengxuan Li, Software Engineer, Dynamia Point AI

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=VAWw5CujiR8

Overview

This session, presented by Xiao Zhang and Mengxuan Li from Dynamia Point AI, delves into critical challenges faced by organizations leveraging AI workloads in Kubernetes: persistently low GPU utilization and the complexities of managing diverse, heterogeneous AI hardware. The core problem highlighted is that traditional Kubernetes deployments often treat GPUs as non-shareable, leading to significant waste of expensive computing resources. This issue is compounded in environments where multiple vendors' AI chips are used, each requiring its own specialized management solution.

The talk introduces Hami, a heterogeneous AI computing virtualization middleware, as a comprehensive solution to these problems. Hami aims to drastically improve GPU utilization, streamline the management of multi-vendor AI chips, and enhance the overall efficiency and flexibility of AI inference and training workloads within Kubernetes clusters. As a CNCF Sandbox project, Hami represents a community-driven effort to standardize and optimize AI resource management.

The significance of Hami extends beyond mere technical optimization; it addresses a pressing economic and strategic need. With the global GPU market experiencing over 60% year-over-year growth, and specific geopolitical challenges impacting access to high-spec Nvidia cards in regions like mainland China, the efficient utilization of available, often alternative, hardware becomes paramount. Hami enables organizations to maximize their return on investment in AI infrastructure, ensuring that costly GPUs are actively and flexibly consumed by workloads, rather than sitting idle due to inflexible scheduling paradigms.

Background

▶ Watch: Introduction and core problem: improving GPU utilization (0:00)

The landscape of AI computing is currently defined by an insatiable and rapidly accelerating demand for processing power. The global GPU market has witnessed an astounding growth exceeding 60% compared to the previous year, with Nvidia dominating this expansion. Concurrently, the heterogeneous computing market, encompassing a variety of AI accelerators beyond Nvidia, has grown by over 20%. This surge is largely attributed to the emergence and widespread adoption of large language models (LLMs) and other compute-intensive AI applications, pushing the boundaries of existing infrastructure.

A significant challenge arises for regions facing import restrictions on high-spec Nvidia cards, such as mainland China. This has necessitated the adoption of alternative AI chips from various device vendors, including Huawei Ascend, Cambricon, and others. While these alternatives offer a more cost-effective solution and decent performance for production workloads, they introduce a new layer of complexity: heterogeneous cluster management. Each vendor typically implements its own custom scheduler extender to integrate with Kubernetes. In a cluster composed of multiple AI card types, this leads to a convoluted scheduling pipeline where a pod's request must traverse through multiple vendor-specific extenders during both the filter and score phases of Kubernetes scheduling, severely degrading overall scheduling performance.

Beyond the heterogeneous challenge, a fundamental limitation of traditional Kubernetes GPU management is the non-shareability of GPUs. Consider a scenario with five 40GB GPUs, each dedicated to running a small AI model requiring only 2GB of device memory. In a conventional setup, each 2GB model would consume an entire 40GB GPU, leaving the vast majority of its memory unused and preventing any other pods from utilizing that GPU. This results in extremely low GPU utilization across the cluster, as other CPU-bound pods or pods waiting for GPU resources remain in a pending state. This inefficiency translates directly into higher operational costs and underutilized hardware.

One attempt to address these issues within the Kubernetes ecosystem is Dynamic Resource Allocation (DRA). Introduced as a stable API in Kubernetes version 1.32, DRA is designed to facilitate the requesting and sharing of resources between pods and containers. It operates by requiring users to define a ResourceClaim and a ResourceClass, with each device vendor expected to implement their own DRA driver to manage device sharing and allocation via the Kubelet. While conceptually promising, DRA faces several practical limitations: it mandates the latest Kubernetes version (1.32 or newer), has seen slow adoption among device vendors (Nvidia's DRA driver is still under construction and not production-ready), and requires explicit configuration of ResourceClaims and ResourceClasses, adding to operational overhead. Furthermore, the feature is not enabled by default, requiring explicit activation. These restrictions make DRA a future-oriented solution, not yet widely viable for current production environments.

Key Findings

▶ Watch: Kubernetes DRA (Dynamic Resource Allocation) and its limitations (3:00)

The core contribution of this talk is the introduction of Hami, a heterogeneous AI computing virtualization middleware designed to overcome the limitations of traditional Kubernetes GPU management and the complexities of multi-vendor AI chip environments. Hami stands out as a CNCF Sandbox project, signifying its open-source nature, community backing, and potential for broader industry adoption.

Hami's primary and most impactful finding is its ability to enable device sharing for AI accelerators, dramatically improving GPU utilization. The presenters demonstrated that Hami can elevate GPU utilization from less than 50% in traditional setups to nearly 100% by allowing multiple workloads to share the memory and compute resources of a single GPU. This transparency to tasks is a crucial aspect, meaning users do not need to modify their application code, Docker images, or source code; they simply specify the desired device memory.

Beyond sharing, Hami provides advanced scheduling capabilities that are essential for optimizing AI workloads. These include features like explicit device type specification (e.g., A100), task prioritization to ensure critical workloads receive preferential access, dynamic Multi-Instance GPU (MIG) management for Nvidia cards, and topology-aware scheduling to minimize communication overhead in distributed training jobs. It also offers flexible bin pack and spread scheduling policies at both GPU and node levels, allowing users to tailor resource allocation strategies to specific workload requirements.

Finally, Hami addresses the critical need for unified monitoring in complex AI clusters. It introduces a dedicated metrics exporter that provides granular insights into GPU memory allocation, free memory, active workloads, and container details – metrics not typically covered by standard tools like DCGM exporter. This unified monitoring, integrated with tools like Prometheus and Grafana, empowers operators with the visibility needed to manage and troubleshoot their heterogeneous AI infrastructure effectively. Hami's pluggable, non-intrusive, and lightweight design (deployable via Helm) makes it an accessible and practical solution for diverse production environments, supporting a wide range of Nvidia and alternative AI chips.

Technical Deep Dive

▶ Watch: Introducing Hami: heterogeneous AI computing virtualization middleware (4:40)

Hami's architecture is meticulously designed to be both powerful and non-intrusive, integrating seamlessly into the Kubernetes ecosystem. It comprises several key components: a mutating admission webhook, a scheduling extender, device plugins for each supported vendor, and an in-container resource control mechanism. This modular design allows Hami to intercept, modify, and manage resource requests and allocations throughout the Kubernetes scheduling and pod lifecycle.

The cornerstone of Hami's efficiency is its device sharing feature, implemented through an ingenious in-container resource control mechanism. When a pod requesting GPU resources is scheduled, Hami injects a specialized library, named humor, into the container's invocation line. This humor library acts as an interceptor, hijacking calls from the CUDA runtime to the CUDA driver. By doing so, Hami gains precise control and visibility over the exact device memory allocation occurring within each container. If a container attempts to allocate device memory beyond the limit specified in its task definition, the humor library will intercept the call and return an Out-of-Memory (OM) error, effectively enforcing the memory quota. This mechanism is broadly compatible, requiring only CUDA version greater than 10.2 and Nvidia driver version greater than 440. Users specify their requirements using Kubernetes annotations or resource requests, such as nvidia.com/gpu for the number of GPUs and nvidia.com/gpu-memory for the desired device memory per GPU (e.g., "10G"). The effectiveness of this control is verifiable, as running nvidia-smi inside a Hami-managed container will show the device memory capped at the allocated limit, not the physical GPU capacity.

Hami's advanced scheduling capabilities significantly enhance its utility:

1. Device Specification: Users can specify preferred or excluded GPU types via annotations. For instance, use-gpu-type: A100 ensures a pod is only scheduled on an Nvidia A100 GPU, while no-use-gpu-type: A100 prevents it from being scheduled on one. This is crucial for optimizing workloads that benefit from specific hardware generations or avoiding incompatible ones.

1. Task Priority: Hami introduces a simple yet effective task prioritization mechanism using the CUDA_TASK_PRIORITY environment variable (0 for high priority, 1 for low priority). When a high-priority pod begins submitting kernels to a GPU, any low-priority pods running on the same GPU are temporarily suspended. They automatically resume execution once the high-priority pod ceases submitting new kernels. This process is entirely transparent to the applications, ensuring critical workloads maintain performance without manual intervention.

1. Dynamic MIG (Multi-Instance GPU) Support: For Nvidia GPUs that support MIG, Hami simplifies its complex management. Instead of requiring users to specify obscure MIG instance names (e.g., 1g.10gb), Hami allows users to simply request the number of GPUs and the desired device memory. Utilizing nvidia-mig-parted, Hami dynamically generates the most fitting MIG instance on the fly, abstracting away the underlying hardware complexity and making MIG resources easily consumable.

1. NUMA and Topology-Aware Scheduling: For distributed AI training jobs spanning multiple GPUs or nodes, minimizing communication latency is vital. Hami's scheduler observes the NUMA (Non-Uniform Memory Access) topology between GPUs and the underlying network topology. This allows it to intelligently allocate the "nearest" GPUs to a given AI training job, thereby minimizing communication costs and accelerating training times. This feature supports Nvidia, Huawei Ascend, and Metex devices.

1. Bin Pack and Spread Scheduling Policies: Hami provides per-task scheduling policies configurable by users. The Bin Pack policy aims to consolidate workloads by allocating tasks to GPUs that are already running other tasks. This strategy minimizes fragmentation of GPU resources and maximizes overall utilization, especially beneficial for memory-bound inference workloads. Conversely, the Spread policy attempts to allocate tasks to GPUs with no existing workloads, aiming to maximize individual task performance by avoiding resource contention. These policies can be applied at both the GPU and node levels, offering fine-grained control over resource distribution.

Hami's support extends beyond Nvidia to a range of heterogeneous AI chips. It successfully applies its device memory control to Huawei Ascend (e.g., limiting a 64GB Ascend chip to 16GB for a specific container) and Cambricon devices, demonstrating its vendor-agnostic resource virtualization capabilities. The roadmap indicates plans to support additional vendors like Kunlun, AMD, and Intel, further solidifying its role as a universal AI resource manager.

The scheduler implementation itself is realized as a scheduler extender. This design choice was deliberate; instead of adopting the Kubernetes scheduler framework, which would necessitate extensive compatibility testing and maintenance across numerous Kubernetes versions (from 1.16 to 1.32), the extender model allows Hami to be easily inserted into existing Kubernetes scheduling pipelines. The extender handles the additional GPU-specific filtering and scoring processes, ensuring Hami's features are broadly accessible without tight coupling to specific Kubernetes scheduler versions.

Demo / Proof of Concept

▶ Watch: How Hami controls in-container device memory limits (6:20)

While the presentation did not include a live, interactive demonstration of Hami in action, the speakers effectively illustrated its capabilities through detailed explanations, architectural diagrams, and concrete examples. These examples served as compelling proof points for Hami's core functionalities.

A key demonstration of Hami's device sharing involved a scenario where two tasks, each requiring two GPUs, were submitted to a node with four GPUs. Without Hami, this would typically lead to each task consuming all four GPUs, resulting in less than 50% overall utilization. With Hami, the two tasks could effectively share two GPUs, freeing up the remaining two for other workloads and pushing GPU utilization close to 100%. This was visually represented to highlight the efficiency gains.

The in-container resource control was further exemplified by showing the output of nvidia-smi from within a Hami-managed container. The screenshot clearly indicated that a physical 40GB GPU, when allocated only 10GB of device memory by Hami, would report its total memory as 10GB within that container environment. This directly demonstrated how the humor library effectively limits and virtualizes GPU memory, preventing over-consumption and enforcing quotas. Similar examples were provided for Huawei Ascend chips, where a 64GB device was shown to be limited to 16GB within a Hami-managed container.

Configuration examples were provided for various advanced scheduling features. For instance, the simple Kubernetes annotation use-gpu-type: A100 was shown for device specification, and setting the CUDA_TASK_PRIORITY environment variable to 0 or 1 illustrated how to assign task priorities. The dynamic MIG feature was explained by showing how users only need to specify desired GPU count and memory, with Hami handling the complex nvidia-mig-parted operations to create the appropriate MIG instance.

Finally, the talk highlighted Hami's unified monitoring capabilities. Although a live Grafana dashboard was not displayed, the speakers detailed the types of metrics exported by Hami's dedicated metrics exporter. These include granular data on allocated device memory, free memory, the number of workloads running on each GPU, and detailed information about those workloads (e.g., process ID, container name). This demonstrated Hami's ability to provide comprehensive operational visibility, which can then be integrated into standard monitoring stacks like Prometheus and visualized in Grafana. The mention of Hami's integration with Volcano VGPU and Koordinator projects further served as a proof of concept for its interoperability and broader ecosystem adoption.

Defensive Implications

▶ Watch: Hami's advanced scheduling: device type and task priority (8:00)

Hami offers several crucial defensive implications for organizations managing AI infrastructure within Kubernetes, primarily by addressing resource inefficiency, operational complexity, and performance bottlenecks.

Firstly, Hami provides a robust defense against underutilization of expensive GPU resources. By enabling transparent device sharing, it ensures that GPUs, which represent a significant capital expenditure, are maximally utilized. This directly translates to substantial cost savings and a lower Total Cost of Ownership (TCO) for AI infrastructure. Defenders can deploy Hami to stretch their existing GPU investments further, delaying or reducing the need for new hardware purchases even as AI workload demands grow.

Secondly, Hami serves as a unified management layer for heterogeneous AI clusters. In environments where multiple vendors' AI chips (Nvidia, Huawei Ascend, Cambricon, etc.) are present, Hami eliminates the need for installing and managing disparate, vendor-specific scheduler extenders. This simplifies the operational burden, reduces the attack surface associated with multiple third-party components, and improves the overall stability and performance of the Kubernetes scheduler by streamlining its pipeline. From a defensive standpoint, standardizing on Hami reduces complexity and potential points of failure.

Thirdly, the advanced scheduling features inherent in Hami contribute to both performance and resilience. Task prioritization allows critical AI inference or training jobs to receive preferential access to GPU resources, ensuring business-critical workloads are not starved by lower-priority tasks. Topology-aware scheduling minimizes communication latency for distributed jobs, which can prevent timeouts and improve the reliability of large-scale AI training. The bin pack and spread policies enable administrators to optimize resource allocation either for maximum density (bin pack, reducing fragmentation) or maximum individual job performance (spread, reducing contention), adapting to varying workload requirements.

Fourthly, Hami's unified monitoring capabilities provide invaluable visibility for defenders. The detailed metrics on GPU memory allocation, free capacity, and active workloads, exported by Hami's dedicated exporter, enable proactive identification of resource bottlenecks, potential memory leaks within containers, or inefficient workload placement. This granular data, integrated into existing monitoring systems like Prometheus and Grafana, empowers operations teams to quickly diagnose and remediate issues, improving the overall reliability and performance of the AI platform.

Finally, Hami's design as a pluggable, non-intrusive scheduler extender ensures broad compatibility across Kubernetes versions (1.16 to 1.32). This makes it a viable solution for organizations that cannot immediately upgrade to the latest Kubernetes versions required by alternatives like DRA, providing a pragmatic defensive strategy for improving GPU management in existing production environments. Its roadmap towards DRA integration also suggests future-proofing for evolving Kubernetes standards.

Key Takeaways

• Maximized GPU Utilization: Hami significantly improves GPU utilization rates (up to nearly 100%) by enabling transparent device sharing among multiple workloads on a single GPU, addressing the inefficiency of traditional non-shareable GPU allocation in Kubernetes.
• Unified Heterogeneous AI Chip Management: As a CNCF Sandbox project, Hami provides a standardized, pluggable middleware for managing diverse AI accelerators from vendors like Nvidia, Huawei Ascend, and Cambricon, simplifying complex multi-vendor environments and reducing scheduling overhead.
• Transparent In-Container Resource Control: Hami enforces precise device memory limits within containers by injecting the humor library to hijack CUDA runtime calls, effectively virtualizing GPU memory and returning OM errors for over-allocation, all without requiring application code changes.
• Advanced Scheduling for AI Workloads: The solution offers sophisticated scheduling features including dynamic MIG instance creation, NUMA and topology-aware resource allocation, and customizable bin pack/spread policies, optimizing performance and resource distribution for various AI training and inference jobs.
• Broad Kubernetes Compatibility and Operational Visibility: Implemented as a scheduler extender, Hami ensures compatibility across a wide range of Kubernetes versions (1.16-1.32). It also provides a dedicated metrics exporter for granular GPU resource monitoring, integrating with tools like Prometheus and Grafana for enhanced operational insights.
• Cost Efficiency and Future-Proofing: By maximizing the efficiency of expensive AI hardware and offering a clear roadmap for supporting more chip types and integrating with Kubernetes DRA, Hami provides a cost-effective and forward-looking solution for AI infrastructure management.

About the Speaker(s)

The session was delivered by Xiao Zhang (known as Lemon) and Mengxuan Li (referred to as Junga), both serving as software engineers at Dynamia Point AI. Dynamia Point AI is described as a newly founded company, indicating their contribution to cutting-edge solutions in the AI infrastructure space. Both speakers are actively involved in the development and community surrounding Hami, a CNCF Sandbox project, showcasing their expertise in Kubernetes, GPU virtualization, and heterogeneous AI computing. Their work focuses on addressing the practical challenges of deploying and managing AI workloads efficiently in complex, multi-vendor environments.