Sponsored Keynote: The Weight of Data: Rethinking Cloud-Native Systems f... V. Caldeira & H. Cummins

The Weight of Data: Rethinking Cloud-Native Systems for AI

Speakers: V. Caldeira, H. Cummins

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=W_EF1HnP4tU

Overview

In a keynote address at KubeCon EU, V. Caldeira and H. Cummins illuminated the profound paradigm shift required within cloud-native architectures, particularly Kubernetes, to effectively support the burgeoning landscape of Artificial Intelligence. The talk, titled "The Weight of Data: Rethinking Cloud-Native Systems for AI," asserts that while Kubernetes has masterfully solved the challenges of distributed, stateless microservices, the inherent statefulness of modern AI workloads—from large language models (LLMs) to multimodal AI agents—presents a fundamentally new set of demands. The core message is clear: the "stateless is good" mantra of traditional cloud-native is directly challenged by AI, necessitating an evolution towards AI-native Kubernetes that can intelligently manage, share, and react to vast, dynamic datasets.

This presentation served as a critical call to action for the cloud-native community, urging a collective effort to adapt existing primitives and foster new solutions for AI's unique requirements. Caldeira and Cummins detailed how the explosion of AI, driven by models like ChatGPT and the increasing complexity of multimodal interactions, has led to an unprecedented "weight of data." This data, far from being transient, is central to AI agent behavior, demanding sophisticated solutions for data locality, synchronization, and fault tolerance that go beyond current cloud-native capabilities. The speakers highlighted how the CNCF ecosystem, with its robust foundation, is uniquely positioned to lead this transformation towards a future where Kubernetes is not just cloud-native, but truly AI-native and intelligent state-aware.

Background

▶ Watch: Introduction: AI's impact and the stateful challenge for Kubernetes (0:00)

The advent of Generative AI and large language models (LLMs) like ChatGPT two and a half years prior fundamentally reshaped the technological landscape, bringing sophisticated AI capabilities into mainstream organizational adoption. This new era of AI is characterized by several key trends:

1. Model Scale: Next-generation models ingest increasingly vast quantities of human knowledge, growing exponentially in size and complexity.
2. Multimodal AI: Interactions have evolved beyond simple text prompts to encompass multiple data types, requiring more intricate data handling.
3. Context State: AI agents now demand enormous volumes of "context state" – the historical data and ongoing conversational context needed for coherent and intelligent responses.

This explosion of data, termed "the weight of data" by the speakers, quickly exposed the limitations of traditional monolithic architectures. The industry's prior solution to monoliths was to decompose them into smaller, independent services, a pattern perfectly aligned with the cloud-native paradigm championed by Kubernetes. Kubernetes thrives on orchestrating these smaller, often stateless components, enabling scalability and resilience through horizontal scaling and ephemeral instances. The core tenet of this paradigm has historically been that "stateless is good," simplifying deployment, scaling, and failure recovery.

However, AI workloads fundamentally invert this principle. AI is inherently stateful. Models themselves are large, persistent states. AI agents continuously share, modify, and react to dynamic data, making state management central to their operation. This clash between AI's statefulness and cloud-native's stateless bias creates significant challenges:

• Data Locality: AI models and agents require high-speed access to massive datasets, making data proximity to compute resources critical.
• Synchronization: Distributed AI agents frequently need to synchronize their understanding of shared state, a complex task at scale.
• Performance Bottlenecks: The sheer volume and dynamic nature of AI data can overwhelm traditional storage and networking solutions.

While Kubernetes already offers strong primitives for state management, such as Persistent Volumes (PVs) for durable storage, StatefulSets for stable identities and access to storage for pods, and DaemonSets for consistent node-level deployments, these are often insufficient for the unique demands of AI. The ecosystem has also seen growth in data-on-Kubernetes projects like VEST (cloud-native distributed MySQL at scale) and Rook (orchestration for storage systems like Ceph), alongside event-driven architectures leveraging projects like KNative and Strimzi (enabling Kafka on Kubernetes). These tools facilitate dynamically linked applications and microservices. Yet, AI workloads operate at a totally different scale and dynamism, requiring an evolution of these constructs to truly make Kubernetes "AI-native."

Key Findings

▶ Watch: Challenges for AI workloads: dynamic state, scale, bottlenecks (2:50)

The central finding of the talk is that the prevailing cloud-native paradigm, optimized for stateless microservices, is fundamentally inadequate for the demands of modern AI workloads. AI's intrinsic statefulness and dynamic data interaction necessitate a significant re-evaluation and evolution of how Kubernetes manages resources and orchestrates applications. The speakers identified several critical areas where existing systems fall short and proposed key evolutions:

1. AI's Statefulness vs. Cloud-Native's Statelessness: The core conflict is that AI agents don't just store state; they actively share, modify, and react to data in high-volume, high-throughput, and highly dynamic ways. This goes beyond the capabilities of current durable storage and stable identity primitives, leading to issues like data locality, synchronization overhead, and performance bottlenecks at scale.

1. Need for AI-Native Scheduling: Traditional Kubernetes schedulers are not optimized for the highly specific and intensive resource requirements of AI. The talk highlights the necessity for:

• NUMA-aware AI scheduling: To optimize memory access patterns for memory-intensive AI workloads.
• GPU and topology-aware AI scheduling: To ensure efficient utilization of specialized hardware accelerators and their interconnectivity, crucial for compute-intensive model training and inference.

1. LLM Gateways for Efficient Inference: As LLMs become ubiquitous, efficient routing of inference requests to these "very high consuming" models is paramount. This requires dedicated LLM gateways capable of intelligently managing and distributing workload to optimize resource usage and latency.

1. A New Model for Fault-Tolerant Management: This is perhaps the most novel and critical finding. AI agents do not fail in the traditional sense of a system crash or unresponsiveness. Instead, they can "fail by producing a wrong output." Recovering from such semantic failures in a distributed system is far more complex than restarting a crashed container. A completely new model for fault tolerance is required to ensure the reliability and trustworthiness of AI systems.

1. CNCF Ecosystem as a Strong Foundation: Despite these challenges, the speakers emphasized that the existing CNCF ecosystem provides a robust foundation for building AI-native Kubernetes. Projects like Q for workload management, Envoy AI Gateway, KServe, and vLLM for scalable inference, Dapr for distributed intelligent architectures, and OpenTelemetry for correlated observability are already paving the way. The collective strength of the community and these projects can drive the necessary shift towards decentralized, intelligent, state-aware AI within cloud-native environments.

Technical Deep Dive

▶ Watch: New challenge: Fault tolerance for AI agent failures (4:00)

The technical deep dive into rethinking cloud-native systems for AI centers on addressing the "weight of data" and the unique operational characteristics of AI workloads. The speakers detailed how existing Kubernetes primitives, while foundational, require significant evolution and augmentation to meet these new demands.

Evolution of State Management for AI:

While Persistent Volumes (PVs) provide durable storage and StatefulSets offer stable identities for pods, AI agents operate with a more dynamic and interactive form of state. They don't just store data; they actively share, modify, and react to it in real-time, often across multiple distributed components. This necessitates:

• High-Throughput, Low-Latency Data Access: AI models, especially during training and inference, require incredibly fast access to vast datasets. Traditional network-attached storage, while flexible, can introduce latency bottlenecks. Solutions must prioritize data locality and efficient data movement.
• Distributed State Synchronization: In multi-agent systems or complex inference pipelines, maintaining a consistent view of shared state across distributed agents is critical. This involves complex synchronization mechanisms that go beyond simple data replication.
• Dynamic Data Lifecycle Management: AI data often has a complex lifecycle, from raw ingestion to feature engineering, model training, and serving. Managing this lifecycle efficiently within Kubernetes requires sophisticated data planes.

The talk acknowledged the growing data on Kubernetes ecosystem with projects like VEST (providing cloud-native distributed MySQL at scale) and Rook (orchestrating storage systems like Ceph). These projects are vital for managing traditional databases and block/object storage within Kubernetes. Similarly, event-driven architectures using KNative and Strimzi (for Kafka on Kubernetes) enable dynamic, loosely coupled microservices. However, AI workloads push the boundaries of these systems, demanding even greater scale, dynamism, and specialized handling for memory, compute, and fault tolerance.

AI-Native Scheduling:

The intensive memory and compute consumption of AI workloads necessitate specialized scheduling capabilities:

• NUMA-aware AI Scheduling: Non-Uniform Memory Access (NUMA) architectures are common in modern servers. Accessing memory local to a CPU core is significantly faster than accessing memory on a remote NUMA node. For memory-intensive AI tasks, NUMA-aware scheduling ensures that pods are placed on nodes where their allocated CPU and memory resources are physically co-located, minimizing memory access latency and improving overall performance. This is crucial for large models that frequently access vast amounts of data in memory.
• GPU and Topology-aware AI Scheduling: Graphics Processing Units (GPUs) and other accelerators are indispensable for AI. GPU and topology-aware scheduling goes beyond simply allocating a GPU. It considers the physical layout of GPUs within a node, their interconnects (e.g., NVLink), and their proximity to CPU cores. This ensures that compute-intensive AI tasks are scheduled on nodes and to specific GPUs that offer the most efficient data transfer paths and minimal communication overhead, maximizing accelerator utilization and throughput.

LLM Gateways:

To efficiently route inference requests to "very high consuming large language models," LLM gateways are proposed. These gateways act as intelligent proxies, capable of:

• Load Balancing: Distributing inference requests across multiple LLM instances or different models based on load, cost, or performance characteristics.
• Request Optimization: Potentially batching requests, caching common responses, or performing early request validation.
• Resource Management: Interfacing with the underlying Kubernetes infrastructure to scale LLM instances up or down based on demand, ensuring efficient use of expensive GPU resources.

Fault-Tolerant Management for AI Agents:

This is a paradigm shift in reliability engineering. Traditional fault tolerance focuses on detecting and recovering from system crashes, network failures, or resource exhaustion. For AI agents, the challenge is different: "they may fail by producing a wrong output." This implies a semantic failure rather than a system-level one. Recovering from such issues in a distributed system requires:

• Output Validation and Monitoring: Mechanisms to detect incorrect or undesirable outputs from AI agents. This could involve anomaly detection, statistical analysis, or comparison with known-good outputs.
• Rollback and Re-execution Strategies: If a wrong output is detected, the system needs to be able to revert to a previous valid state, potentially re-execute the agent's logic with different parameters, or trigger alternative pathways.
• Human-in-the-Loop Interventions: For complex or critical AI systems, human oversight and intervention might be necessary to diagnose and correct semantic failures.
• Distributed State Consistency: Ensuring that all distributed agents maintain a consistent understanding of the "correct" state, even after a semantic failure and recovery operation.

Leveraging CNCF Projects for AI-Native Kubernetes:

The speakers highlighted several CNCF projects that provide a strong foundation for addressing these challenges:

• Q: An "extremely interesting workload management capability on Kubernetes." While not explicitly detailed, such a project would likely provide advanced scheduling, resource allocation, and job orchestration for complex AI workflows, going beyond standard Kubernetes scheduling.
• Envoy AI Gateway, KServe, and vLLM: These projects collectively address scalable inference management for Generative AI models.
• Envoy AI Gateway would likely provide the intelligent routing and load balancing capabilities mentioned for LLM gateways.
• KServe (formerly KFServing) is a Kubernetes-native platform for deploying, serving, and managing machine learning models. It provides serverless inference, auto-scaling, canary rollouts, and multi-model serving.
• vLLM is an open-source library for fast LLM inference, often integrated into serving platforms like KServe, to optimize the performance of serving large models.
• Dapr (Distributed Application Runtime): Helps build "highly distributed intelligent architecture and systems." Dapr provides building blocks for microservices, such as state management, service invocation, publish/subscribe, and resource bindings, which are crucial for orchestrating complex, distributed AI agents. Its state management capabilities could be particularly relevant for managing the shared, dynamic state of AI agents.
• OpenTelemetry: Described as a "very very big dependency" for correlating observability across dynamic AI agents. OpenTelemetry provides a standardized way to collect traces, metrics, and logs from distributed systems. For AI, this is critical for diagnosing performance bottlenecks, understanding the flow of data through complex pipelines, and crucially, for detecting and debugging semantic failures ("wrong output") by tracing the agent's decision-making process and data interactions.

The synergy of these projects, combined with ongoing efforts from the CNCF AI working group (which has produced a white paper on its progress), forms the bedrock for shifting Kubernetes towards a truly AI-native, intelligent, and state-aware platform.

Demo / Proof of Concept

▶ Watch: CNCF projects addressing AI state and inference management (4:40)

The transcript mentions a "live Red Hat demo this afternoon at 3:30 in the demo theater" and that "Vincent has a talk tomorrow about AI on Cube." While this keynote itself did not include a live demonstration within its allotted time, the reference indicates that practical implementations and demonstrations of the concepts discussed were available at the conference.

Given the technical content of the keynote, a likely demonstration or proof of concept would showcase:

• AI-Native Scheduling in action: Illustrating how specific AI workloads (e.g., an LLM inference service) are scheduled onto nodes with optimal GPU and NUMA topology, perhaps showing performance improvements compared to non-optimized scheduling.
• Scalable LLM Inference: A demonstration of deploying and scaling an LLM using tools like KServe and vLLM on Kubernetes, highlighting the role of an Envoy AI Gateway in routing requests and managing traffic. This could involve simulating varying loads and showing how the system dynamically scales to meet demand while maintaining performance.
• Distributed AI Agent Orchestration: A simple multi-agent system built with Dapr, demonstrating how agents share and modify state, and how OpenTelemetry provides end-to-end observability across these distributed components, potentially even showing how a "wrong output" could be detected and traced.

The absence of a specific demo detailed in this transcript means we cannot elaborate on its exact mechanics, but the context strongly suggests a practical illustration of leveraging the identified CNCF projects to build and operate stateful, AI-intensive applications on Kubernetes.

Defensive Implications

▶ Watch: Vision: Building intelligent, state-aware cloud-native AI together (5:20)

While the talk was not explicitly focused on cybersecurity, the challenges of managing "the weight of data" and the unique failure modes of AI agents have profound defensive implications for the reliability, integrity, and operational security of cloud-native AI systems. Defenders must consider:

1. Data Integrity and Provenance for Stateful AI: With AI agents dynamically sharing and modifying state, ensuring data integrity becomes paramount. Defenders need robust mechanisms to verify the origin and correctness of data throughout its lifecycle, from ingestion to model training and inference. Any compromise in data integrity can lead to biased models or "wrong outputs," which are a form of system failure. Solutions must incorporate strong access controls, encryption for data at rest and in transit, and immutable logging of data modifications.

1. Robust Fault Tolerance for Semantic Failures: The concept of AI agents "failing by producing a wrong output" introduces a new attack surface and operational challenge. Defenders must evolve beyond traditional system monitoring to include AI-specific monitoring that detects anomalous or incorrect outputs. This requires:

• Output Validation Systems: Implementing automated checks (e.g., using rule-based systems, statistical anomaly detection, or secondary AI models) to validate the output of primary AI agents.
• Explainability and Interpretability: Leveraging explainable AI (XAI) techniques to understand why an AI agent produced a particular output, aiding in debugging and identifying malicious manipulation or unintended biases.
• Recovery Strategies for Semantic Errors: Developing automated or semi-automated recovery plans for wrong outputs, which might involve reverting to previous states, re-running tasks with different parameters, or escalating to human review.

1. Secure Orchestration of Distributed AI: As AI systems become more distributed and leverage projects like Dapr for inter-service communication, the attack surface expands. Defenders must ensure:

• Secure Service-to-Service Communication: Implementing mutual TLS (mTLS) and strong authentication/authorization policies between AI agents and supporting services.
• Supply Chain Security: Verifying the integrity of all components, from base container images to model weights and dependencies, especially with the increased use of open-source AI models and tools.
• Resource Isolation and Quotas: Properly segmenting AI workloads and enforcing resource quotas to prevent denial-of-service attacks or resource exhaustion, particularly for GPU-intensive tasks.

1. Enhanced Observability for Trust and Security: OpenTelemetry's role in providing correlated observability across dynamic AI agents is critical for security. Defenders need this detailed telemetry to:

• Detect Malicious Activity: Identify unusual patterns in data access, compute utilization, or inter-service communication that could indicate an attack.
• Forensic Analysis: In the event of a breach or semantic failure, detailed traces and logs are essential for post-mortem analysis to understand the root cause and scope of the incident.
• Compliance and Auditing: Provide verifiable records of AI system behavior for regulatory compliance and internal auditing purposes.

1. Secure AI-Native Scheduling: The specialized scheduling (NUMA-aware, GPU-aware) for AI also has security implications. Misconfigurations could lead to:

• Resource Starvation: Malicious actors attempting to monopolize scarce GPU resources.
• Information Leakage: Improper isolation of workloads on shared accelerators or NUMA nodes.
• Defenders must ensure that these advanced scheduling mechanisms are configured securely, with proper multi-tenancy isolation and resource governance.

In essence, the shift to AI-native cloud systems demands a parallel evolution in defensive strategies, moving beyond infrastructure security to encompass the unique challenges of AI model integrity, data trustworthiness, and semantic reliability in distributed environments.

Key Takeaways

• AI fundamentally challenges the "stateless is good" paradigm of traditional cloud-native systems, necessitating a shift towards intelligent state-aware architectures.
• Existing Kubernetes primitives for state management (PVs, StatefulSets) are foundational but require significant evolution to handle the dynamic, high-volume, and high-throughput data interactions of modern AI agents.
• AI-native scheduling is critical for optimizing performance, requiring NUMA-aware and GPU/topology-aware capabilities to efficiently utilize memory and specialized hardware accelerators.
• A new model for fault tolerance is essential for AI systems, as agents can fail by producing "wrong outputs" rather than just crashing, demanding sophisticated detection and recovery mechanisms.
• CNCF projects like Q, Envoy AI Gateway, KServe, vLLM, Dapr, and OpenTelemetry provide a strong foundation for building scalable, resilient, and observable AI-native Kubernetes environments.
• The cloud-native community must collaborate to build this future, evolving Kubernetes to be truly AI-native and capable of managing the "weight of data" effectively and securely.

About the Speaker(s)

V. Caldeira and H. Cummins are prominent voices at KubeCon EU, advocating for the evolution of cloud-native systems to meet the demands of artificial intelligence. While specific titles were not provided in the transcript, the context of the talk, including references to a Red Hat demo and booth, strongly suggests their affiliation with Red Hat. Their expertise lies in guiding the cloud-native community through the challenges of integrating stateful AI workloads into distributed Kubernetes environments, emphasizing the need for innovative solutions in state management, scheduling, and fault tolerance. V. Caldeira was noted to have an additional talk scheduled on "AI on Cube," further highlighting their deep involvement in this critical area.