Enhancing CRI-O With CDI: Streamlining Device Integration in Kubernetes - Harshal Patil, Red Hat

Enhancing CRI-O With CDI: Streamlining Device Integration in Kubernetes

Speakers: Harshal Patil, Red Hat

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=7-JtDLNT0c8

Overview

In the rapidly evolving landscape of container orchestration, integrating specialized hardware devices like GPUs, FPGAs, or custom network cards into Kubernetes pods has historically presented significant challenges. These devices often require intricate setup, including specific driver installations, file system mounts, environment variable configurations, and the execution of pre- or post-container lifecycle hooks. Harshal Patil's KubeCon EU talk, "Enhancing CRI-O With CDI: Streamlining Device Integration in Kubernetes," addresses this complexity head-on by introducing the Container Device Interface (CDI) specification and detailing its integration with CRI-O, the Kubernetes-native container runtime.

This talk illuminates how CDI provides a standardized, vendor-agnostic mechanism for CRI runtimes to expose complex device configurations to containers, abstracting away the low-level intricacies. For developers of accelerators, custom hardware, or device plugins, CDI offers a unified approach to ensure their devices are seamlessly usable within Kubernetes. Patil demonstrates CRI-O's role in leveraging this specification, thereby simplifying device management for both cluster administrators and application developers, and ultimately enhancing Kubernetes' capability to host sophisticated, hardware-accelerated workloads with improved security and efficiency.

Background

▶ Watch: Introduction and talk agenda (0:00)

CRI-O stands out in the container runtime ecosystem due to its singular focus on Kubernetes. Unlike other container runtimes that might serve broader purposes, CRI-O is designed from the ground up to cater exclusively to the Kubernetes Container Runtime Interface (CRI). This specialization allows CRI-O to be highly optimized for Kubernetes environments, from its release cycles and feature development to its internal architecture. For instance, CRI-O includes optimizations like faster pod listing by utilizing caches rather than traversing the cgroup filesystem, and it quickly adapts to new security features such as seccomp artifact support, pulling configurations directly from registries. Its Kubernetes-only target significantly limits its attack surface, making it a more secure choice for container execution within a cluster.

The problem CDI aims to solve stems from the historical lack of standardization in how specialized hardware devices are exposed to containers in Kubernetes. Prior to CDI, device manufacturers or platform providers had to implement bespoke solutions for each device type. This often involved:

• Manual configuration: Specifying device nodes (e.g., /dev/mydevice1, /dev/mydevice2) directly in pod manifests.
• Complex setup scripts: Writing OCI hooks to initialize devices, load drivers, or perform other necessary pre-container actions.
• Environment variables: Setting specific environment variables (e.g., NVIDIA_VISIBLE_DEVICES) to guide device access, often requiring knowledge of vendor-specific conventions.
• Host path mounts: Manually mounting host file systems containing drivers or libraries into the container.

This patchwork approach led to fragmentation, increased operational overhead, and reduced portability. Each device type or vendor might require a different set of steps, making it difficult for Kubernetes users to integrate diverse hardware accelerators consistently. The Container Networking Interface (CNI) provided a successful precedent for standardizing network configuration in containers, and CDI draws inspiration from this model, aiming to bring similar standardization to device integration. The need for a standardized interface became even more pronounced with the advent of advanced features like the Device Resource Allocation (DRA) framework, which requires a robust and consistent way to describe and allocate devices.

Key Findings

▶ Watch: What is the Container Device Interface (CDI)? (2:49)

The central finding presented in the talk is the Container Device Interface (CDI) as a robust, standardized specification for CRI runtimes, like CRI-O, to support third-party devices within Kubernetes. CDI's core contribution is to abstract the complexities of device setup, offering a unified method for device manufacturers to define how their hardware should be prepared and presented to a container.

Key findings and contributions include:

1. Standardized Device Configuration: CDI provides a declarative way to define all necessary configurations for a device to function inside a container. This includes:

• Device nodes: Specifying paths like /dev/mydevice.
• Environment variables: Setting required variables (e.g., NVIDIA_VISIBLE_DEVICES).
• Mount paths: Defining host paths that need to be mounted into the container.
• OCI hooks: Specifying commands to be executed at various container lifecycle stages (e.g., create container, start container) for device initialization or cleanup.
• Multiple device nodes: Supporting scenarios where a device requires more than just a single /dev entry, such as /dev/mydevice1, /dev/mydevice2, etc.

1. Abstraction for Users: CDI shifts the burden of complex device setup from the Kubernetes user to the device manufacturer or driver developer. Users no longer need to know the intricate details of a specific GPU or FPGA's requirements; they simply refer to a fully qualified device name defined in a CDI specification. This greatly simplifies pod manifest creation and promotes consistency across different hardware vendors.

1. Inspiration from CNI: The design of CDI is heavily inspired by the success of CNI, aiming to bring a similar level of plug-and-play capability and standardization to device integration that CNI brought to networking.

1. CRI-O's Role as an Enabler: CRI-O's deep integration with the CDI specification positions it as a key enabler for this standardized approach. By monitoring CDI spec directories, parsing these definitions, and applying the specified configurations to the OCI runtime spec, CRI-O seamlessly bridges the gap between the Kubernetes control plane and the underlying container runtime, ensuring devices are correctly configured before a container starts.

1. Dynamic Device Management: CRI-O's CDI integration supports dynamic addition or removal of device specs without requiring a runtime restart, allowing for more flexible and responsive device management within a cluster.

In essence, CDI elevates device integration from a collection of ad-hoc scripts and manual configurations to a well-defined, portable, and maintainable standard within the Kubernetes ecosystem.

Technical Deep Dive

▶ Watch: Who benefits from CDI: device manufacturers and complex setups (3:22)

The technical foundation of CDI revolves around its specification file, typically a YAML or JSON document, which declaratively describes the requirements for a device. A sample CDI spec file identifies a device on the host using a fully qualified name like vendor.com/device=mydevice. Within this spec, the crucial element is containerEdits. This section outlines all the modifications needed for the container's OCI runtime specification to properly utilize the device.

The containerEdits can include:

• Environment variables: For example, setting NVIDIA_VISIBLE_DEVICES with the UID of a specific GPU.
• Device nodes: Listing specific /dev paths that need to be accessible inside the container.
• Mount paths: Defining host directories that must be mounted into the container, often for drivers or libraries.
• OCI hooks: Specifying commands or executables to run at various stages of the container lifecycle, such as createContainer, startContainer, prestart, poststart, poststop, to initialize or clean up device resources.

A notable feature of containerEdits is the ability to define them at two levels: one for a general device type and another for specific instances of that device type. This neatly avoids duplication when multiple devices of the same type (e.g., several GPUs) require similar base configurations but differ in specific details like their unique device node.

CRI-O's integration with CDI is pivotal. CRI-O continuously monitors specific directories for CDI spec files, by default /etc/cdi and /var/run/cdi, though these paths are configurable via crio.conf. When a Kubernetes component, such as Kubelet, requests a device, it passes the CDI's fully qualified device name to CRI-O. CRI-O then performs the following steps:

1. Locate the spec: It searches the monitored directories for the CDI spec file corresponding to the received fully qualified device name.
2. Read containerEdits: Once located, CRI-O parses the containerEdits defined within the spec.
3. Modify OCI runtime spec: CRI-O applies these edits to the OCI runtime spec (the config.json that describes how a container should run) before passing it to the underlying container runtime, such as C-run or runC. This ensures that all necessary environment variables, device nodes, mounts, and OCI hooks are correctly configured for the container.

The talk elaborates on how CDI fits into both traditional Kubernetes device plugin workflows and the newer Device Resource Allocation (DRA) framework.

With Classical Device Plugins:

1. A user requests device resources (e.g., nvidia.com/gpu: 1) in a pod's resources.limits.
2. The device plugin (e.g., NVIDIA device plugin) receives this request and, in its Allocate method, selects an available physical device.
3. Instead of directly exposing device paths, the device plugin now returns the corresponding CDI device names (e.g., nvidia.com/gpu=GPU1) to Kubelet.
4. Kubelet forwards these CDI device names to CRI-O via the CRI.
5. CRI-O then follows its standard CDI integration flow: locates the spec, reads containerEdits, and modifies the OCI runtime spec.
6. The OCI runtime (C-run/runC) executes the container with the fully configured device.

With the new DRA Driver:

1. Users define device requirements using resource claims (e.g., resourceClaim: gpu-single-gpu) instead of direct resource limits.
2. During pod creation, Kubelet calls the NodePrepareResources method on the DRA plugin.
3. The DRA plugin, similar to the classical device plugin, translates the resource claim into CDI device names and returns them to Kubelet.
4. The subsequent flow is identical: Kubelet forwards CDI names to CRI-O, which modifies the OCI spec, and the runtime runs the container.

This standardization significantly simplifies device management. CRI-O, as a CRI runtime, leverages CDI to abstract away the vendor-specific complexities, providing a uniform interface for device provisioning. The ability to dynamically add or remove CDI specs without restarting CRI-O further enhances operational flexibility, allowing administrators to update device configurations on the fly as hardware changes or drivers are updated.

Demo / Proof of Concept

▶ Watch: How CDI integrates and works with CRI-O (6:09)

Harshal Patil presented a compelling real-world example of CDI integration using Nvidia's DRA driver and A100 GPUs. The demonstration highlighted the practical application of CDI in a Kubernetes environment.

The demo began by inspecting the host system where Nvidia's DRA driver was installed. This driver automatically generates CDI specification files in a designated directory. Patil showed a sample CDI spec file, revealing how it describes the configuration for an A100 GPU. Key elements observed in the spec included:

• Device nodes: Specific /dev paths required for the GPU.
• Environment variables: Crucially, NVIDIA_VISIBLE_DEVICES was set, which Nvidia drivers use to control GPU visibility within a container.
• OCI hooks: The spec included a create container hook, indicating that certain commands need to be executed during container creation to prepare the device.
• Mount paths: Definitions for host paths to be mounted into the container, likely containing essential GPU libraries or drivers.
• Fully qualified name: The spec defined the device with a name like kubernetes.io/gpu.nvidia.com/device=GPU3, illustrating the standardized naming convention.

Next, a sample Kubernetes workload was executed using a resource claim template to request GPU resources via the DRA framework. The resourceClaim specified a GPU single GPU, abstracting the underlying hardware details. Upon the pod's creation, the demonstration showed that the DRA driver dynamically created two additional CDI specs in the monitored directory. These temporary specs, as explained, don't necessarily back a physical device but are used by the DRA system to track the claim and manage resource allocation.

To verify that the container successfully accessed the GPU, Patil displayed the logs from the running pod. The output clearly showed the name of the A100 GPU, confirming that the CDI-driven configuration had correctly exposed the device to the container, enabling the workload to utilize the requested hardware.

Finally, the demo concluded by deleting the pod. As expected, upon deletion, the DRA driver automatically removed the corresponding temporary CDI device specs, demonstrating the lifecycle management and cleanup capabilities inherent in the CDI and DRA integration. This proof of concept effectively illustrated how CDI, in conjunction with CRI-O and DRA, provides a seamless, standardized, and automated mechanism for integrating and managing complex hardware accelerators within Kubernetes.

Defensive Implications

▶ Watch: CDI workflow with Kubernetes Device Plugins (7:44)

The introduction of CDI, while significantly streamlining device integration, also carries important defensive implications for Kubernetes security postures.

1. Reduced Misconfiguration Risk: By standardizing device configuration, CDI inherently reduces the risk of human error and misconfiguration. Instead of administrators or developers manually crafting complex OCI hooks, device node mappings, and environment variables, the device manufacturer provides a tested and validated CDI spec. This consistency makes it less likely for critical security settings to be overlooked or incorrectly applied.

1. Trust in CDI Spec Source: The power of CDI lies in its ability to modify the OCI runtime spec, including injecting environment variables, mounting host paths, and executing arbitrary OCI hooks. This means that the source of the CDI spec files is paramount. Administrators must ensure that CDI specs are generated by trusted device drivers or are carefully reviewed if manually created. An untrusted or malicious CDI spec could potentially lead to container escapes, privilege escalation, or unauthorized access to host resources. As Harshal Patil noted, device drivers typically run with higher privileges to access hardware and thus are the ideal, trusted entity to generate these specs.

1. Existing Security Model Persists: CDI does not bypass existing Kubernetes or Linux security mechanisms. As highlighted in the Q&A, if an administrator was previously responsible for provisioning devices or installing drivers, that responsibility remains. CDI is a way to express device configuration, not a way to provision devices or circumvent privilege requirements. The underlying access control for /dev paths, host mounts, and OCI hook execution still depends on the permissions of the CRI-O process and the container's security context.

1. CRI-O's Limited Attack Surface: CRI-O's Kubernetes-centric design contributes to a more secure baseline. By having a minimal feature set focused solely on Kubernetes, its attack surface is inherently smaller compared to general-purpose container runtimes. This narrow scope means fewer potential vectors for exploitation, which is a beneficial foundation for any new feature like CDI integration.

1. Dynamic Updates and Validation: CRI-O's ability to detect CDI spec changes on the fly without restarts offers operational flexibility but also requires vigilance. Any changes to CDI specs should ideally be part of a controlled, automated deployment pipeline that includes validation and security checks. Uncontrolled modification of CDI specs could introduce vulnerabilities.

1. MIG Slices and Fine-Grained Control: The discussion around MIG (Multi-Instance GPU) slices in the Q&A further illustrates the security benefits of CDI. By allowing a DRA driver to enumerate and create CDI specs for individual MIG slices, CDI enables fine-grained resource allocation. This means containers can be allocated only the necessary fraction of a GPU, reducing the potential impact if a container is compromised, as it would have access to a smaller, isolated hardware resource rather than an entire GPU.

In summary, while CDI offers powerful capabilities, its defensive implications emphasize the need for a robust supply chain for CDI specs, careful management of administrative privileges, and continuous adherence to established security best practices within the Kubernetes cluster.

Key Takeaways

• Standardized Device Integration: CDI provides a vendor-agnostic specification for CRI runtimes to define and expose complex hardware device configurations to containers, abstracting low-level details.
• CRI-O's Kubernetes-Native Advantage: CRI-O's singular focus on Kubernetes allows for highly optimized and secure integration with CDI, leveraging its minimal attack surface and rapid feature adoption.
• Declarative Device Configuration: CDI specs (containerEdits) enable declarative definition of environment variables, device nodes, mount paths, and OCI hooks required for a device, avoiding manual, error-prone setups.
• Seamless Integration with Kubernetes Workflows: CDI supports both traditional device plugins and the newer DRA framework, ensuring a consistent mechanism for device allocation and configuration within Kubernetes pods.
• Dynamic and Flexible Management: CRI-O can dynamically detect and apply changes to CDI specs without requiring a restart, offering greater operational flexibility for device management.
• Enhanced Security through Abstraction (with caveats): CDI reduces misconfiguration risks by centralizing device setup definitions. However, trust in the source of CDI specs (e.g., device drivers) is paramount, as these specs can configure powerful container capabilities.

About the Speaker(s)

Harshal Patil is an expert from Red Hat, a leading contributor to open-source technologies, particularly in the Kubernetes ecosystem. His work focuses on enhancing the capabilities and security of core Kubernetes components like CRI-O. Patil's insights are deeply rooted in the practical challenges of integrating specialized hardware into containerized environments, reflecting Red Hat's commitment to advancing enterprise-grade Kubernetes solutions. His presentation at KubeCon EU underscores his expertise in container runtimes, device management, and the broader Kubernetes infrastructure, particularly from a security and optimization perspective.