Kubeflow Ecosystem: What’s Next for Cloud Native AI/ML and LLMOps

Kubeflow Ecosystem: What’s Next for Cloud Native AI/ML and LLMOps

Speakers: Andre, Staff Software Engineer at Apple & Kubeflow Steering Committee; Yuki, Software Engineer at CyberAgent & maintainer of Kubernetes/Kubeflow/KubeBatch related CNCF tools; Yan, Senior Principal Software Engineer at Red Hat AI, Kubeflow Steering Committee, Argo Project Lead, & Kubernetes Working Group Serving Co-chair; Anie, Technical Director at Nutanix AI, Kubeflow Steering Committee, & lead for AI initiatives (training, AutoML, ML common storage); Valentina Rodriguez, Principal Architect at Red Hat, Kubeflow contributor (release team, platform working groups), & KCD organizer in New York.

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=gGP9QdlNr9Y

Overview

This talk provides a comprehensive update on the Kubeflow ecosystem, a collection of open-source projects designed to make AI and Machine Learning (ML) simple, portable, and scalable on Kubernetes. Presented by a panel of Kubeflow steering committee members and key contributors, the session delves into the latest advancements enabling Generative AI (GenAI) and Large Language Model Operations (LLMOps) within a cloud-native context. The speakers emphasize Kubeflow's role as a crucial bridge between the complex world of ML and the robust cloud ecosystem, offering data scientists and ML engineers simplified interfaces to leverage advanced tools without deep Kubernetes expertise.

The core message of the talk revolves around Kubeflow's adaptability and extensibility, allowing organizations to integrate individual components or deploy a complete end-to-end platform across any Kubernetes cluster—on-prem, in the cloud, or locally. The session highlights how Kubeflow's components now seamlessly address every stage of the GenAI lifecycle, from data processing and model development to fine-tuning, distributed training, optimization, and large-scale inference. The recent release of Kubeflow 1.x marks a significant milestone, bringing a host of new features, performance optimizations, and an enhanced user experience tailored for the demands of modern AI workloads, particularly GenAI.

Background

▶ Watch: Introduction and panelist welcomes (0:00)

Kubeflow originated from the need to run ML workloads reliably and scalably on Kubernetes, the de facto standard for container orchestration. Its fundamental goal has always been to provide an open-source, composable platform that simplifies the deployment and management of ML workflows. This means enabling data scientists to focus on their models and algorithms, while platform engineers can manage the underlying infrastructure with Kubernetes primitives. Before the rise of GenAI, Kubeflow already offered robust solutions for traditional ML tasks, including notebooks for development, operators for distributed training (like TFJob and PyTorchJob), and pipelines for orchestrating complex workflows.

The advent of GenAI and Large Language Models (LLMs) introduced new challenges and requirements for ML platforms. These include handling massive datasets, orchestrating complex fine-tuning processes (such as Reinforcement Learning from Human Feedback - RLHF), optimizing hyper-parameters for LLMs, and serving models that can be gigabytes or even terabytes in size. Kubeflow's community has actively responded to these challenges, evolving its existing components and introducing new features to support the unique demands of LLMOps. The talk frames this as moving into a "new world" where Kubeflow's established architecture provides a powerful foundation for tackling these cutting-edge AI applications, abstracting away the underlying Kubernetes complexity while still leveraging its scalability and resilience.

Key Findings

▶ Watch: Kubeflow GenAI/LLM lifecycle and components overview (2:50)

The talk unveiled a host of significant updates and new features across the Kubeflow ecosystem, demonstrating a strong focus on GenAI and LLMOps capabilities. These advancements collectively aim to enhance user experience, improve performance, and expand the platform's utility for modern AI workloads.

1. Kubeflow 1.x Release: A major announcement was the release of Kubeflow 1.x at KubeCon EU, signifying a culmination of community efforts to deliver a more efficient, optimized, and user-friendly platform.

1. Kubeflow Workspaces (New UI): The notebooks team introduced Kubeflow Workspaces, a new, simplified UI designed to provide interactive development environments for data scientists. This intuitive interface supports various IDEs like RStudio, VS Code, and JupyterLab, making it highly pluggable for both data scientists and platform engineers.

1. Spark Operator 2.1.0: The Spark Operator project, recently donated from Google to the Kubeflow ecosystem, released version 2.1.0. This update brings support for Spark 3.x, integrates Unicorn with gang scheduling for improved resource management, refactors the codebase with control runtime, and enhances interactive sessions for Jupyter notebooks on Spark clusters. A notable benchmark by AWS demonstrated the operator's scalability, running 60,000 Spark applications across 36,000 pods.

1. Katib Enhancements for GenAI: Kubeflow's Katib project, dedicated to hyperparameter optimization (HPO) and architecture search, now simplifies LLM fine-tuning with a single API for creating experiments. It also supports optimizing Retrieval Augmented Generation (RAG) pipelines, allowing users to embed RAG within Katib experiments for performance tuning.

1. Kubeflow Trainer V2: A significant overhaul of the training operator, Kubeflow Trainer V2, was introduced. It features a new resource model with TrainJob and TrainingRuntime CRDs, decoupling infrastructure from training code. This version supports a wide array of ML frameworks, including PyTorch, DeepSpeed, MLX, TensorFlow, with Hugging Face under development, and future plans for Jax, JAX, and XGBoost.

1. Distributed Arrow Cache: An exciting new project involves creating a distributed cache on Kubernetes by connecting Apache Arrow, Iceberg, and PyTorch. This enables zero-copy translation from Arrow format to PyTorch tensors, facilitating distributed streaming and showing significant performance improvements for ML training.

1. Unified SDK & ML Experience Working Group: To simplify the user experience, a new ML Experience working group is developing a unified SDK. This Python interface aims to abstract away Kubernetes complexities, allowing data scientists to process data, train, and optimize models without needing to interact directly with kubectl, Docker, or YAML. Collaborations include integrating with the Llama community for agent building and evaluation, and TorchTune for streamlined PyTorch fine-tuning.

1. Model Registry UI and Storage Integration: The Kubeflow Model Registry, donated by Red Hat, received a new UI, making it easier to manage and version models and their metadata. It also includes updates for integrating with custom storage initializers, improving compatibility with projects like KServe.

1. KServe Advancements for LLMs: KServe, Kubeflow's model serving and inference platform, introduced several critical updates. These include integration with Envoy AI Gateway for supporting multiple LLM providers, token-based rate limiting, and advanced routing. Autoscaling capabilities were improved for large models via KEDA integration and custom metrics. Model caching through PV/PVC was added to reduce spin-up times for large models, and vLLM server support enables multi-node inference for LLMs. Gateway API integration also enhances raw deployment modes.

1. Kubeflow Pipelines Improvements: The orchestration engine, Kubeflow Pipelines, received enhancements for managing resource limits and controlling loop parallelism within workflows. These improvements provide greater control over resource consumption and execution flow, crucial for efficient LLMOps.

Technical Deep Dive

▶ Watch: Exciting updates on Kubeflow Notebooks and Workspaces (3:30)

Kubeflow's latest iteration showcases a mature and evolving ecosystem, deeply integrated with Kubernetes to address the demanding landscape of GenAI and LLMOps. Each component has seen targeted enhancements, often leveraging underlying Kubernetes capabilities or integrating with other cloud-native projects.

Kubeflow Workspaces are at the forefront of improving the developer experience. This new UI layer abstracts the creation and management of interactive development environments (IDEs) on Kubernetes. Data scientists can provision JupyterLab, RStudio, or VS Code instances with specified resources and persistent storage through a few clicks, without needing to write complex Kubernetes manifests. This "snapshot" UI represents a significant step towards making Kubeflow more accessible to users primarily focused on data science tasks.

The Spark Operator, now a core Kubeflow component, is critical for large-scale data processing. Version 2.1.0's support for Spark 3.x ensures compatibility with the latest Spark features. The integration with Unicorn with gang scheduling is a key performance enhancement, allowing Spark applications to request and receive all necessary resources simultaneously, preventing deadlocks and improving overall cluster utilization for resource-intensive jobs. The refactoring with control runtime modernizes the operator's codebase, making it more robust and easier to maintain. Furthermore, interactive sessions enable data scientists to run Spark jobs directly from Jupyter notebooks within Kubeflow, bridging the gap between interactive development and distributed computation. The AWS benchmark, showcasing the operator's ability to handle 60,000 Spark applications across 36,000 pods, validates its enterprise-readiness and scalability for massive data processing needs.

Katib extends its reach into the GenAI domain by simplifying hyperparameter optimization (HPO) for LLM fine-tuning. Previously, setting up HPO experiments for complex models could be arduous. Katib now provides a unified, simple API to define and run these experiments, crucial for finding optimal configurations for LLMs. A novel application demonstrated is its use in optimizing Retrieval Augmented Generation (RAG) pipelines. By plugging the entire RAG pipeline into a Katib experiment, users can optimize parameters related to retrieval, generation, or integration, enhancing the quality and relevance of generated content. This functionality, contributed by GSOC students, highlights the community's innovation.

Kubeflow Trainer V2 represents a significant architectural shift for distributed training. Its new resource model introduces two Custom Resources (CRDs): TrainJob and TrainingRuntime. TrainingRuntime is predefined by DevOps or platform engineers, encapsulating infrastructure details (e.g., GPU types, network configurations). TrainJob is then defined by ML engineers or data scientists, specifying their training code and referencing a TrainingRuntime. This decoupling allows data scientists to focus purely on their training logic and parameters (number of nodes, processes per node) without worrying about Kubernetes specifics. Trainer V2 consolidates various framework-specific job CRDs (like TFJob, PyTorchJob in v1) into a single, internal pipeline framework, reducing maintenance overhead and accelerating the integration of new ML frameworks. Current support includes PyTorch, DeepSpeed, MLX, and TensorFlow, with Hugging Face under active development, and future plans for Jax, JAX, and XGBoost.

The Distributed Arrow Cache is a cutting-edge initiative designed to optimize data loading for distributed ML training. By integrating Apache Arrow, Iceberg, and PyTorch on Kubernetes, this project aims to create a distributed caching layer that enables zero-copy translation of data. Data stored in Apache Iceberg, a table format for large analytics datasets, can be converted directly into PyTorch tensors without intermediate copies, significantly reducing latency and improving throughput for data-intensive training workloads. This is particularly beneficial for streaming data to distributed PyTorch training jobs, a common pattern in GenAI.

The Unified SDK and the new ML Experience working group are tackling the challenge of Kubernetes complexity head-on. The goal is to provide a single Python interface that allows data scientists to orchestrate their entire ML workflow—data processing, training, optimization, and serving—without ever needing to touch kubectl commands, Docker, or YAML files. This SDK aims to abstract the underlying Kubernetes infrastructure, allowing users to "just focus on PyTorch" and scale their models. Key collaborations include working with the Llama community to integrate Kubeflow tools for building LLM agents, post-training evaluation, and prompt engineering. Additionally, integration with TorchTune simplifies PyTorch fine-tuning. The SDK provides a train API that leverages pre-built trainers and runtimes, handling distributed training and resource utilization seamlessly.

The Kubeflow Model Registry fills a critical gap in the MLOps lifecycle by providing a centralized system for managing and versioning models and their associated metadata. The introduction of a new UI significantly improves usability, offering a graphical interface for registering, tracking, and collaborating on models. Updates to support custom storage initializers enhance its flexibility, ensuring models can be stored and retrieved from various backend systems, aligning well with KServe's serving capabilities.

KServe (formerly KFServing) is Kubeflow's solution for highly scalable and cloud-agnostic model serving and inference, now explicitly supporting GenAI. The integration with Envoy AI Gateway is a major step forward, enabling KServe to manage multiple LLM providers, implement token-based rate limiting to control API access, and provide sophisticated routing with traffic shaping, fallback mechanisms, and load balancing. Enhanced autoscaling capabilities for large models are crucial for LLMs, which can have fluctuating demand; KServe now integrates with KEDA to allow autoscaling based on custom metrics, providing fine-grained control. Model caching through Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) addresses the challenge of large model sizes, preventing repeated downloads during autoscaling events and speeding up cold starts. Support for vLLM server enables efficient multi-node inference for LLMs, leveraging specialized runtimes. Finally, Gateway API integration provides a modern, extensible way to manage ingress traffic for KServe deployments, especially in raw deployment modes.

Kubeflow Pipelines remains the backbone for orchestrating end-to-end ML workflows. It stitches together all Kubeflow components into directed acyclic graphs (DAGs). Recent improvements focus on providing better control over resource limits management within pipeline steps, allowing users to precisely define CPU, memory, and GPU requirements. Additionally, enhancements for controlling loop parallelism enable more efficient execution of iterative tasks, such as hyperparameter sweeps or data processing loops, ensuring optimal resource utilization and faster experimentation.

Demo / Proof of Concept

▶ Watch: Katib for LLM hyperparameter optimization and RAG (5:20)

The talk featured a compelling live demo showcasing an end-to-end LLMOps pipeline orchestrated by Kubeflow Pipelines, focusing on fine-tuning a Large Language Model and serving it for inference.

The objective of the demo was to fine-tune a Hugging Face Gemma 2B model for reasoning tasks using Generative Reinforcement Learning from Human Feedback (GRPO). The process leveraged the unsloth library for efficient fine-tuning, Kubeflow Trainer for distributed training, and KServe for deploying the fine-tuned model for inference.

The demonstration began with the Kubeflow dashboard, highlighting its multi-tenancy capabilities where users can switch between profiles. The presenter navigated to the Notebooks section, where a Jupyter notebook was pre-configured with the Kubeflow Pipelines SDK (KFP) and a Hugging Face token to download the Gemma 2B model.

The core of the demo's fine-tuning process involved wrapping custom training code within a KFP DSL component. This code utilized the unsloth library to implement GRPO, which involves defining reward functions. The concept was explained: positive rewards are given for correct or well-formatted answers, and penalties for incorrect ones, guiding the model's learning process. The GRPO trainer from unsloth was configured with various hyperparameters, which could also be tuned by Katib, though this wasn't explicitly shown in the demo.

The crucial part of the training task was the train function, which used Kubeflow Trainer to initiate distributed training. The presenter specified resources for the training job: one node with four CPUs, 64GB of memory, and one GPU. The flexibility to scale by simply changing the number of nodes or GPUs was emphasized. The pipeline was set to wait for the training job to complete before proceeding.

Following the training, the second pipeline task was inference. Once the fine-tuned model was available, KServe was used to create a deployment. The entire workflow was orchestrated using Kubeflow Pipelines, defining a DAG where storage provisioning occurred first, followed by fine-tuning, and then serving. A separate path was also created to serve the original base model for comparison.

The demo then transitioned to the Kubeflow Pipelines UI, showing the triggered run and its DAG visualization. Users could monitor the progress of each step—storage, fine-tuning, and serving—in real-time. Logs and pod details were accessible directly from the UI, abstracting away the need to use kubectl. As the GRPO fine-tuning progressed, the UI displayed improving reward metrics, visually confirming the model's learning.

Finally, after the training pod completed and the KServe inference service was deployed, the presenter demonstrated the fine-tuned model's reasoning capabilities. A query, "What is the area of a triangle for length 3, 4, 4?", was sent to the KServe endpoint. The model's output not only provided the numerical answer but also detailed the step-by-step reasoning process, showcasing the effectiveness of the GRPO fine-tuning. This demonstrated how Kubeflow enables data scientists to rapidly iterate on LLM development, from fine-tuning to scalable serving, all within a unified, cloud-native environment.

Defensive Implications

▶ Watch: Introduction to Kubeflow Trainer for model training (6:50)

The advancements in the Kubeflow ecosystem carry significant implications for platform engineers and security professionals, particularly in managing robust and secure ML environments. Two key areas highlighted are security and resource management.

Security: The talk explicitly mentioned ongoing efforts in security, including CVE reductions and a broader initiative to improve security best practices across all Kubeflow components. This is critical for any cloud-native platform, especially one handling sensitive data and intellectual property in ML models. Defenders should closely monitor Kubeflow's security advisories and incorporate these updates promptly. The shift towards more modular and well-defined components (like Kubeflow Trainer V2's resource model) can simplify security audits and vulnerability management by clearly delineating responsibilities and attack surfaces. As Kubeflow moves towards graduation within the CNCF, adherence to rigorous security standards is expected to strengthen further.

Resource Management: The improvements in Kubeflow Pipelines for controlling resource limits management and loop parallelism are vital for preventing resource exhaustion, ensuring fair usage in multi-tenant environments, and optimizing cloud costs. For platform engineers, this means greater control over how ML workloads consume compute, memory, and GPU resources. Properly configured resource limits prevent runaway jobs from impacting other users or causing cluster instability. The ability to control loop parallelism allows for efficient scheduling of iterative tasks, preventing over-provisioning or under-utilization of expensive resources like GPUs. These controls are essential for building predictable and cost-effective LLMOps platforms, enabling organizations to scale their GenAI initiatives responsibly while maintaining operational stability. The decoupling of infrastructure (TrainingRuntime) from training code (TrainJob) in Kubeflow Trainer V2 also provides a clearer boundary for enforcing resource quotas and policies at the infrastructure level.

Key Takeaways

• Kubeflow 1.x Release & GenAI Focus: The recent release of Kubeflow 1.x signifies a major stride, with the entire ecosystem now explicitly geared towards simplifying and scaling Generative AI (GenAI) and Large Language Model Operations (LLMOps) on Kubernetes.
• Enhanced User Experience: Initiatives like the new Kubeflow Workspaces UI and the Unified SDK (driven by the ML Experience working group) are abstracting Kubernetes complexities, enabling data scientists to focus on model development and training without needing deep cloud-native expertise.
• Advanced Training & Inference Capabilities: Kubeflow Trainer V2 offers a flexible resource model and broad framework support for distributed training, complemented by cutting-edge features like the Distributed Arrow Cache for zero-copy data loading. KServe provides highly scalable LLM serving with Envoy AI Gateway, advanced autoscaling, and model caching.
• Robust MLOps Lifecycle Management: Components like Katib for LLM hyperparameter optimization and RAG pipeline tuning, the improved Model Registry with its new UI, and Kubeflow Pipelines with enhanced resource control, provide a comprehensive platform for managing the entire ML lifecycle.
• Community Growth & Strategic Integrations: The Kubeflow community is actively working towards CNCF graduation, fostering collaboration (e.g., with Llama and TorchTune communities), and integrating valuable projects like the Spark Operator (donated by Google) and potentially Fess as a feature store, expanding its utility and adoption.
• Focus on Security and Efficiency: Ongoing efforts in CVE reductions and improving security best practices, alongside granular resource management features in Pipelines, underscore Kubeflow's commitment to building a secure, efficient, and cost-effective platform for cloud-native AI.

About the Speaker(s)

The panel comprised a diverse group of key contributors and leaders within the Kubeflow community:

• Andre is a Staff Software Engineer at Apple and has been a Kubeflow Steering Committee member for the past eight years, playing a pivotal role in guiding the project's direction.
• Yuki works as a Software Engineer for CyberAgent and is actively involved in maintaining Kubernetes, Kubeflow, and KubeBatch-related CNCF ecosystem tools.
• Yan serves as a Senior Principal Software Engineer at Red Hat AI. He is a Kubeflow Steering Committee member, a project lead for Argo, and co-chair of the Kubernetes Working Group Serving.
• Anie is a Technical Director at Nutanix AI. Within the open-source community, Anie is part of the Kubeflow Steering Committee and leads several AI initiatives, including training, AutoML, and ML common storage.
• Valentina Rodriguez is a Principal Architect at Red Hat. Her contributions to Kubeflow span diverse projects, and she is also part of the release team and works with platform working groups. Additionally, Valentina is an organizer for KCD in New York.