Sponsored Keynote: Hand Meet Glove: Why Kubernetes Will Become the Platform o... I. Levine & K. Babo

Hand Meet Glove: Why Kubernetes Will Become the Platform of AI

Speakers: I. Levine, K. Babo (Solo.io)

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=-k1CdrRAGMM

Overview

This keynote presentation, delivered by I. Levine and K. Babo, outlines a significant evolution in the cloud-native landscape, asserting that the foundational components for modern networking are now complete and optimized, paving the way for Kubernetes to become the definitive platform for Artificial Intelligence (AI) and agentic workloads. The talk details several key open-source contributions and donations to the Cloud Native Computing Foundation (CNCF), emphasizing a strategic pivot to address the emergent demands of AI infrastructure. It highlights how existing cloud-native paradigms, particularly around API gateways and service meshes, are being adapted and extended to manage the unique challenges posed by Large Language Models (LLMs) and autonomous agents.

The speakers argue that the decade-long community effort in building Kubernetes infrastructure has culminated in a robust, scalable, and efficient ecosystem perfectly poised to support the next wave of technological transformation driven by AI. The presentation introduces new projects and features designed to bridge the gap between traditional microservices architectures and the requirements of AI-driven applications, covering everything from secure LLM integration and model serving to the deployment and management of AI agents. This talk is crucial for platform engineers, architects, and developers seeking to understand how Kubernetes will underpin the future of AI development and deployment.

Background

▶ Watch: Cloud-native networking complete; K Gateway CNCF sandbox (0:00)

The journey to establish Kubernetes as a robust platform has been characterized by continuous innovation, particularly in the realm of cloud-native networking. Initially, the complexity of inter-service communication, traffic management, and security in distributed microservices architectures led to the widespread adoption of solutions like API gateways and service meshes. Projects such as Envoy Proxy, Istio, and Gloo emerged as critical components, addressing challenges related to ingress, egress, traffic routing, policy enforcement, and observability. The concept of a sidecar proxy, where a proxy runs alongside each application container, became a prevalent pattern for service mesh implementations, offering powerful features but often incurring overheads in terms of resource consumption and operational complexity.

Recognizing the need for further optimization and standardization, the cloud-native community has been working towards refining these foundational layers. The announcement of ambient mesh in Istio represented a significant step forward, aiming to provide service mesh capabilities without the traditional sidecar model, thereby reducing operational overhead and improving performance. Concurrently, efforts to standardize API gateway configurations led to the development of the Gateway API within the Kubernetes ecosystem, providing a more expressive and extensible way to manage ingress traffic.

This mature and optimized cloud-native infrastructure now faces its next grand challenge: the rise of Artificial Intelligence, particularly Generative AI and Large Language Models (LLMs). The integration of LLMs into applications, the serving of fine-tuned models, and the emergence of agentic workloads (applications where LLMs drive business logic and interact with tools) introduce new requirements. These include specialized routing, prioritization, security guardrails for LLM interactions, cost management, and the need for new protocols to facilitate agent-to-tool communication. The existing cloud-native stack, while powerful, required specific extensions and adaptations to meet these novel demands, setting the stage for the innovations presented in this keynote.

Key Findings

▶ Watch: Is existing infrastructure ready for AI workloads? (2:20)

The keynote made several significant announcements and revealed key findings, signaling a concerted effort to complete the cloud-native networking stack and extend it for AI workloads:

1. K Gateway as a CNCF Sandbox Project: The open-source Gloo API gateway, an Envoy-based solution, was officially donated to the CNCF and accepted as a sandbox project under the new name K Gateway. This move standardizes and open-governs a critical ingress component, building on feedback from thousands of users.
2. Ambient Mesh General Availability (G.A.) and Migration Tools: The Istio ambient mesh architecture, offering a sidecar-less approach to service mesh, reached General Availability. To facilitate adoption, two new tools were announced at ambientmesh.io: a cost analysis tool to predict savings from migrating to ambient and an automated zero-downtime migration tool for seamless transitions from sidecar-based meshes.
3. Open-Sourcing of AI Gateway Features: Features from Solo.io's commercial Gloo AI Gateway product, designed for integrating applications with hosted LLM providers (e.g., adding guard rails, semantic analysis, cost controls), were open-sourced and integrated into the K Gateway project.
4. Gateway API Inference Extensions Implementation: Recognizing the unique ingress requirements for hosting LLM serving on private infrastructure (e.g., request routing, prioritization, load balancing for GPU-backed models), the CNCF community developed the Gateway API inference extensions spec. Solo.io announced its full open-source implementation of this spec within the K Gateway project.
5. MCP Gateway for Agentic Workloads: A new component, MCP Gateway, was announced and upstreamed to K Gateway. This gateway addresses the challenges of discovery, security, and observability for a large number of tools used by AI agents communicating via the Agent Communication Protocol (MCP), federating multiple tool servers into a single endpoint.
6. K Agent Project Donation to CNCF: A brand new open-source project, K Agent, designed for building, deploying, and managing AI agents within Kubernetes, was announced and immediately donated to the CNCF. K Agent builds upon Microsoft's Autogen framework, adding an MCP-based tooling layer, example agents, and a declarative API with a controller for Kubernetes-native agent management. This donation was performed live during the keynote.

These findings collectively demonstrate a dual focus: solidifying the core cloud-native networking components under open governance and rapidly innovating to meet the infrastructure demands of the emerging AI paradigm, all within the Kubernetes ecosystem.

Technical Deep Dive

▶ Watch: K Gateway open-sources AI gateway features for LLM integration (3:45)

The keynote presented a comprehensive technical vision for Kubernetes as the platform for AI, building upon and extending existing cloud-native infrastructure. The discussion spanned several critical areas, from optimizing the core networking stack to introducing entirely new components for AI agent management.

Cloud-Native Networking Optimization

The foundation of this vision rests on a highly optimized cloud-native networking stack. The speakers highlighted Gloo, an Envoy-based API gateway, as a cornerstone for ingress. Its donation to the CNCF as K Gateway signifies a commitment to open governance and community-driven development, ensuring its continued evolution for scale and performance. K Gateway is positioned as the unified entry point for both traditional microservices and emerging AI workloads.

Concurrently, the service mesh layer has seen significant advancements with Istio ambient mesh reaching General Availability. Ambient mesh offers a sidecar-less architecture, which is a crucial optimization. Traditional sidecar-based service meshes, while powerful, introduce resource overhead (CPU, memory) due to a proxy running alongside every application pod. Ambient mesh addresses this by moving proxy functions to a shared node-level component (the ztunnel), significantly reducing the cost, improving performance, and enhancing the user experience. The speakers emphasized its lightweight nature, intuitive operation, and effortless multicluster functionality, boasting support for "100 million pod scale with incredible performance." The release of tools like the cost analysis tool and the automated zero-downtime migration tool underscores the practical considerations for enterprises transitioning from existing sidecar deployments to the more efficient ambient model.

AI and Agentic Workload Infrastructure

The core of the keynote's technical deep dive focused on adapting and extending this optimized cloud-native stack to meet the unique demands of AI and agentic workloads.

LLM Integration and Egress Control

The initial phase of integrating AI into applications often involves making egress calls to hosted LLM providers (e.g., OpenAI, Google Cloud AI). This scenario introduces critical requirements beyond basic connectivity:

• Guard Rails: Preventing harmful or inappropriate content from being sent to or received from LLMs.
• Semantic Analysis: Understanding the meaning and intent of requests and responses for policy enforcement.
• Cost Management: Monitoring and controlling API usage to external LLM services.
• End-to-End Governance Controls: Ensuring compliance and security across all LLM interactions.

These features, initially developed within Solo.io's commercial Gloo AI Gateway, have now been open-sourced and integrated directly into the K Gateway project. This means that K Gateway can act as a centralized control point for all LLM egress traffic, applying necessary policies and observability.

Model Serving and Ingress for LLMs

As organizations mature in their AI journey, they often transition to hosting LLM serving on their own infrastructure, particularly for fine-tuned models that require specialized hardware like GPUs. This shift places new and specific demands on the ingress tier:

• Request Routing: Directing requests to specific model versions or instances.
• Prioritization: Ensuring critical AI inference requests receive higher precedence.
• Load Balancing: Efficiently distributing inference requests across GPU-backed model servers, often considering GPU utilization and latency.

The CNCF community recognized this gap and responded by implementing the Gateway API inference extensions. This new specification and API provides a standardized way to configure these advanced ingress functionalities for AI model serving. Solo.io highlighted its significant contribution to this spec and announced a full open-source implementation within the K Gateway project, making it readily available for Kubernetes users. This allows K Gateway to intelligently route and manage traffic to internal LLM endpoints, optimizing resource utilization and ensuring reliable inference.

Agentic Workloads and Tooling

The most transformative aspect discussed was the evolution of applications from traditional microservices to agentic workloads. In this paradigm, the LLM itself takes charge of the business logic, interacting with various "tools" (e.g., databases, APIs, external services) to achieve its goals. Core components like tooling and function calling become central to the application's operation.

A significant challenge arises when agents need to interact with a large number of tools. The Agent Communication Protocol (MCP), notably used by Anthropic, defines the communication between agents and tools. While MCP works well for a few tools, scaling to hundreds or thousands of tools introduces familiar problems:

• Discovery: How do agents find available tools?
• Security: How are tool interactions authenticated and authorized?
• Observability: How can tool usage and performance be monitored?

These are precisely the problems that API gateways were designed to solve for microservices. Leveraging this parallel, the keynote introduced MCP Gateway. This new component, now upstreamed to K Gateway, federates all tool servers into a single endpoint. It then seamlessly multiplexes traffic from agents to the appropriate upstream tool, providing centralized control over discovery, security, and observability for agent-tool interactions. This effectively creates a "service mesh for agents and tools," bringing robust enterprise-grade capabilities to the agentic paradigm.

K Agent: Building and Managing Agents in Kubernetes

Finally, addressing the complete lifecycle of AI agents, the keynote unveiled K Agent. This project tackles the problem of building, deploying, and managing agents themselves within Kubernetes. K Agent is built on a core agent framework layer based on Microsoft's Autogen, a popular framework for multi-agent conversations. K Agent extends Autogen with three crucial components:

1. MCP-based Tooling Layer: An extensible layer that ships out-of-the-box with tool server implementations for popular CNCF projects, enabling agents to easily interact with the cloud-native ecosystem.
2. Example Agents: A set of pre-built agents that leverage these tools to support traditional application development and platform engineering workflows in Kubernetes, demonstrating practical use cases.
3. Declarative API and Controller: A Kubernetes-native declarative API and controller implementation to simplify the development and deployment of custom agents, allowing users to define agents and their behaviors using standard Kubernetes manifests.

The vision for K Agent is profound: to do for platform engineering what Cursor (an AI-powered code editor) did for software engineering, by simplifying and automating complex tasks through intelligent agents. The project was open-sourced two weeks prior to the keynote and, in a live demonstration, was formally donated to the CNCF, underscoring its intended role as a crucial piece of future AI infrastructure.

In essence, the technical deep dive demonstrates a strategic and comprehensive effort to evolve Kubernetes into the primary platform for AI, leveraging and extending its mature networking capabilities while introducing specialized components for LLM and agent management.

Demo / Proof of Concept

▶ Watch: Introducing K Agent for building and managing Kubernetes agents (6:55)

While the keynote did not feature a live, interactive technical demonstration of code or system execution in the traditional sense, it did include a significant "proof of concept" moment related to the K Agent project.

The speakers announced that K Agent, a new open-source project designed for building, deploying, and managing AI agents in Kubernetes, had been open-sourced two weeks prior and was now ready for deeper community involvement. To underscore its significance and commitment to the open-source ecosystem, I. Levine performed a live donation of K Agent to the CNCF during the presentation. This involved navigating to a pull request on a public repository and physically pressing a button to initiate the donation process. This act, while seemingly simple, served as a powerful symbolic demonstration of the project's readiness and the speakers' confidence in its value to the broader cloud-native community. The spontaneous applause and the speaker's humorous relief ("That always makes me nervous. I expected to time out or something. That's awesome.") highlighted the real-time nature and impact of this live action.

Furthermore, the speakers encouraged attendees to visit their booth for "hands-on labs for everything we've discussed today," indicating that practical demonstrations and opportunities for direct engagement with the technologies (K Gateway, ambient mesh tools, MCP Gateway, K Agent) were available immediately after the talk. This suggests that while a full live coding demo wasn't part of the keynote stage presentation, the underlying technologies were robust enough for immediate hands-on exploration by the community.

Defensive Implications

▶ Watch: Live donation of K Agent project to CNCF (8:05)

The advancements outlined in this keynote carry significant defensive implications for organizations operating cloud-native environments and those integrating AI. By centralizing control and standardizing practices, these new tools and projects enhance security, governance, and operational resilience.

1. Enhanced LLM Egress Security and Governance (K Gateway AI Features): The open-sourcing of AI Gateway features within K Gateway provides critical defensive capabilities for interactions with hosted LLMs. Guard rails can prevent the leakage of sensitive data or the injection of harmful prompts (prompt injection attacks). Semantic analysis allows for deeper inspection of LLM requests and responses, enabling the detection of anomalous behavior or policy violations. Cost controls not only manage budgets but can also serve as an early warning system for runaway or malicious LLM usage. By centralizing these controls at the API gateway level, organizations gain a single enforcement point for all LLM egress traffic, simplifying auditing and compliance.

1. Secure and Controlled LLM Ingress (Gateway API Inference Extensions in K Gateway): For organizations hosting their own LLM models, the Gateway API inference extensions implemented in K Gateway offer robust control over ingress. This allows for precise request routing, prioritization, and load balancing, which can be critical for maintaining the availability and performance of sensitive AI services. More importantly, it provides a dedicated layer to enforce access control, rate limiting, and other security policies specific to AI inference endpoints, protecting against denial-of-service attacks or unauthorized model access.

1. Centralized Agent-Tool Communication Security (MCP Gateway): The MCP Gateway addresses a crucial security gap for agentic workloads. By federating all tool servers behind a single endpoint, it provides a centralized point to enforce security policies for agent-tool interactions. This includes:

• Authentication and Authorization: Ensuring only legitimate agents can access specific tools.
• Observability: Providing a clear audit trail of which agents accessed which tools, when, and with what parameters, which is vital for forensic analysis and compliance.
• Traffic Inspection: Allowing for deep packet inspection of MCP traffic to detect malicious commands or data exfiltration attempts by compromised agents.

This pattern mitigates the complexity of securing numerous direct agent-tool connections, reducing the attack surface.

1. Simplified and Secure Agent Deployment (K Agent): K Agent provides a Kubernetes-native way to define and manage AI agents, which inherently brings the benefits of Kubernetes' security model. By leveraging a declarative API and controller, K Agent promotes secure configuration practices, ensuring agents are deployed consistently and adhere to defined policies. The MCP-based tooling layer within K Agent, when combined with MCP Gateway, ensures that agents are designed to interact with tools through a controlled and observable channel. This framework helps prevent the proliferation of ad-hoc agent deployments with inconsistent security postures.

1. Optimized and Resilient Infrastructure (Ambient Mesh): The widespread adoption of ambient mesh contributes to a more resilient and efficient infrastructure. By reducing the resource overhead of the service mesh, it frees up resources that can be allocated to security monitoring tools or to handle larger traffic volumes more robustly. The simplified operational model also reduces the potential for misconfigurations that could introduce security vulnerabilities. Its "jaw-dropping multicluster functionality" extends these benefits to complex, distributed environments, ensuring consistent security policies across all clusters.

In summary, these innovations provide defenders with a more comprehensive and integrated toolkit to secure the entire cloud-native and AI application lifecycle, from external LLM interactions to internal agent-tool communications and the deployment of AI agents themselves. They promote centralized control, standardized policies, and enhanced observability, which are fundamental pillars of a strong security posture.

Key Takeaways

• Cloud-Native Networking Maturity: The core cloud-native networking stack, including API gateways and service meshes, is now considered complete and highly optimized, with projects like K Gateway and Istio ambient mesh providing scalable and efficient solutions.
• Kubernetes as the AI Platform: Kubernetes is strategically positioned as the foundational platform for AI and agentic workloads, with significant new open-source contributions designed to bridge the gap between traditional microservices and AI requirements.
• AI Gateway Features for LLM Governance: K Gateway now incorporates open-source AI Gateway features, offering essential guard rails, semantic analysis, cost controls, and end-to-end governance for applications interacting with hosted LLMs.
• Gateway API for AI Model Serving: The Gateway API inference extensions, fully implemented in K Gateway, provide specialized ingress capabilities (routing, prioritization, load balancing) crucial for hosting and serving fine-tuned LLMs on Kubernetes, particularly on GPU-backed infrastructure.
• MCP Gateway for Agent-Tool Communication: MCP Gateway, integrated into K Gateway, solves critical discovery, security, and observability challenges for agentic workloads, federating numerous tool servers into a single, manageable endpoint for Agent Communication Protocol (MCP) traffic.
• K Agent for Kubernetes-Native Agent Management: The new K Agent project, donated to the CNCF, simplifies the building, deploying, and managing of AI agents in Kubernetes, extending Microsoft's Autogen with an MCP-based tooling layer, example agents, and a declarative API for platform engineers.

About the Speaker(s)

I. Levine and K. Babo are prominent figures in the cloud-native community, representing Solo.io, a key contributor to the projects discussed in this keynote. As presenters of a sponsored keynote at KubeCon EU, they showcased Solo.io's significant open-source contributions and strategic vision for the future of cloud-native infrastructure, particularly in the context of AI. Their expertise lies in architecting and developing advanced networking solutions, API gateways, and service meshes, with a recent focus on extending these capabilities to support the emerging demands of AI and agentic workloads within the Kubernetes ecosystem. Their presentation highlighted their deep involvement in community-driven initiatives, including contributions to K Gateway, Istio ambient mesh, Gateway API, and the new K Agent project, demonstrating their commitment to advancing open-source cloud-native technologies.