Green AI in Cloud Native Ecosystems: Strategies for Sustainability... Vincent Caldeira & Tamar Eilam

Green AI in Cloud Native Ecosystems: Strategies for Sustainability... Vincent Caldeira & Tamar Eilam

Speakers: Vincent Caldeira, CTO for Asia Pacific, Red Hat; Tamar Eilam, IBM Fellow and Chief Scientist for Sustainable Computing, IBM Research

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=fznzH-gf9h8

Overview

In an era defined by the explosive growth of artificial intelligence, the environmental impact of AI systems has emerged as a critical concern. This talk, delivered by Vincent Caldeira of Red Hat and Tamar Eilam of IBM Research at KubeCon EU, addresses the burgeoning AI sustainability crisis and presents comprehensive strategies for achieving Green AI within cloud-native ecosystems. The speakers meticulously dissect the energy consumption patterns of modern AI, highlighting the disproportionate impact of AI inference and proposing a multi-faceted approach to optimize resource utilization across the entire machine learning lifecycle.

The session underscores that green AI is intrinsically linked to efficient AI, advocating for a paradigm shift in how AI systems are designed, deployed, and operated. With global regulations like the EU AI Act beginning to mandate energy consumption disclosure, the imperative for sustainable AI is no longer merely ethical but also a pressing business and compliance requirement. Caldeira and Eilam showcase how strategic optimizations, particularly at the platform level for inference workloads, can lead to dramatic reductions in both operational energy consumption and the embodied carbon footprint of hardware, ensuring that AI's transformative potential can be harnessed responsibly for the benefit of humanity.

This article delves into the core arguments, technical mechanisms, and community efforts discussed in the talk, providing a detailed exploration of the challenges and opportunities in building environmentally conscious AI infrastructure. It serves as a guide for engineers, architects, and decision-makers navigating the complexities of optimizing AI workloads for sustainability in a cloud-native context.

Background

▶ Watch: Introduction and the emerging AI sustainability crisis (0:00)

The journey towards an AI sustainability crisis began subtly around 2010 with the widespread adoption of deep neural networks. Since this "deep learning era," the energy consumption required to train AI models has seen an average growth of four to five times per year, a staggering increase over a 15-year period. This exponential demand is projected to escalate further, with predictions indicating that by 2028, nearly 19% of data center power demand will be attributed solely to AI workloads. This rapid consumption surge has not gone unnoticed, prompting the emergence of regulatory frameworks globally. The EU AI Act, for instance, stands as the first regulation worldwide to mandate the disclosure of energy consumption for AI systems, signaling a future where understanding and reporting AI's environmental impact will be a compulsory business requirement.

Ironically, despite the immense computational resources poured into AI, a significant operational inefficiency persists. A 2023 report by the State of AI Infrastructure revealed that 74% of companies utilizing AI struggle to effectively leverage their compute and acceleration infrastructure, pointing to widespread operational problems in optimizing resource use. This inefficiency exacerbates the sustainability challenge, as underutilized hardware contributes to both increased energy consumption and a larger embodied carbon footprint from manufacturing.

However, the narrative of AI's environmental impact is not solely one of consumption; AI also serves as a powerful tool for driving sustainability initiatives. The speakers highlighted several impactful applications, many of which are pioneered by IBM Research in collaboration with entities like NASA. These include:

• Climate Science: Developing foundation models that process 250,000 terabytes of earth observation data, enabling four times faster interpretation of satellite imagery for predicting extreme climate events.
• Material Science: Using foundation models to simulate and model new material structures, aiding in the identification of green and safe alternatives for the hundreds of thousands of consumer products currently monitored for toxic substances by agencies like the EPA.
• Environmental Science: Partnering with national science agencies to identify and remediate "forever chemicals" that persist in the environment for millennia.
• Health Sciences: Revolutionizing early disease detection through AI-powered medical imaging.

To harness AI's benefits responsibly, a holistic approach to sustainability is essential, encompassing the entire machine learning (ML) life cycle and supply chain. This involves shifting left in the ML development process, much like in DevOps or security, to integrate efficiency from the earliest stages. For instance, data distillation—creating more meaningful and concise datasets—can reduce model training by up to 70% by processing less data. This initial optimization cascades throughout the life cycle, leading to smaller models, faster testing, and reduced resource consumption. The optimization efforts can be categorized into three pillars: data optimization, model optimization (making models as small as possible), and system-centric optimization (efficient operations). Ultimately, the goal is to reduce the usage of compute, accelerators, networking, and storage resources.

A particular emphasis was placed on AI inference due to its disproportionate impact on overall energy consumption. Citing figures from Facebook AI, the speakers revealed that approximately 65% of an AI system's operational carbon footprint is spent on inference. For many enterprises, this figure is likely even higher than Facebook's, given their less frequent model retraining cycles. Furthermore, optimizing the platform level for inference can yield an astonishing 800 times improvement in consumption. Beyond operational energy, the manufacturing of hardware represents a significant portion of AI's environmental cost, with 50% of embodied carbon originating from hardware production. By optimizing inference, the need for less hardware is reduced, directly contributing to a substantial cut in embodied carbon. This dual benefit—reducing both operational energy and embodied carbon—solidifies inference as the critical battleground for Green AI.

Key Findings

▶ Watch: AI's growing data center power demand and new regulations (2:10)

The talk presented several pivotal insights into the pursuit of sustainable AI in cloud-native environments:

• Green AI is Efficient AI: The fundamental premise is that efforts to improve efficiency in AI systems directly translate to reduced resource consumption, including energy. By making AI workloads run more efficiently, organizations inherently contribute to environmental sustainability.
• Inference Dominates AI's Carbon Footprint: Contrary to common assumptions that training is the primary energy culprit, the speakers highlighted that AI inference accounts for the majority (65%) of an AI system's operational carbon footprint. Moreover, 50% of the embodied carbon in AI hardware also stems from manufacturing, which can be mitigated by reducing the need for excess hardware through inference optimization. This makes inference optimization a high-leverage area for sustainability efforts.
• The Critical Role of the Platform: The platform, particularly in a cloud-native context, is identified as the linchpin for achieving Green AI. Its responsibility lies in intelligently bridging diverse AI models and requests (with varying characteristics like latency and throughput requirements) with heterogeneous infrastructure, ensuring the use of fit-for-purpose accelerators to avoid wasteful resource allocation.
• Intersectionality of Optimization Techniques: A crucial finding is that various optimization techniques—such as GPU slicing, intelligent routing and queuing, caching, and autoscaling—are not isolated but deeply interdependent. For optimal results, these control mechanisms must operate coherently, with an "intelligent awareness" of each other, to ensure that their combined effect is synergistic ("1 + 1 = 3, not -1").
• Rapid Evolution Demands Modular and Open Approaches: The AI landscape, especially concerning optimization algorithms and techniques, is evolving at "light speed." To keep pace, architectural principles like modularity, separation of concerns, data-driven optimization, and leveraging best-of-breed open-source technologies are essential. This allows for quick testing of new algorithms and fostering community collaboration without reinventing the wheel.
• Standardization and Community are Key: Addressing the complexity of sustainable AI platforms requires collective effort. Initiatives within the CNCF (Cloud Native Computing Foundation), such as the AI working group, the TAG Environmental Sustainability, and projects like Kepler for energy observability and Gateway API extensions for LLM inference, are vital for standardizing practices and sharing best practices across the industry.

Technical Deep Dive

▶ Watch: How AI can drive sustainability improvements (3:30)

The technical deep dive of the talk focused on a comprehensive suite of optimization techniques, spanning both model architecture and sophisticated platform-level strategies tailored for cloud-native AI inference. The overarching goal is to maximize throughput, minimize latency, and drastically reduce the energy footprint while maintaining performance and respecting Service Level Objectives (SLOs).

Model Architecture and Runtime Optimizations:

The journey to efficient AI begins with the model itself. Several techniques were highlighted:

• Laura Adapters (Low-Rank Adaptation): This method addresses the challenge of fine-tuning large language models (LLMs) for specific tasks without retraining the entire model. By freezing most of the original model's weights and introducing small, trainable "adapters" (Laura modules), it significantly reduces the number of parameters that need to be updated. This "breaking the monolith" approach leads to smaller, more efficient models for specific tasks, reducing computational demands during fine-tuning and inference.
• Mixtures of Experts (MoE): An architectural paradigm where different "expert" sub-networks specialize in processing different types of input. During inference, a "router" network selectively activates only a subset of these experts for a given input, leading to more efficient computation compared to running a monolithic model.
• Quantization: This technique reduces the precision of model weights and activations (e.g., from 32-bit floating-point to 8-bit integers). Lower precision requires less memory and faster computation, making models smaller and quicker to infer, often with minimal impact on accuracy.
• Speculative Decoding: A runtime inference technique for LLMs where a smaller, faster "draft" model optimistically generates a sequence of tokens. A larger, more accurate "verifier" model then checks these tokens in parallel. If the draft's predictions are correct, they can be accepted in a batch, significantly speeding up token generation compared to the standard auto-regressive process.

Platform-Level Optimizations in Cloud-Native Environments:

The core of the talk emphasized the critical role of the platform in orchestrating efficient inference, especially given the diverse nature of AI workloads and heterogeneous hardware.

1. Heterogeneous Hardware and Fit-for-Purpose Accelerators:

The speakers stressed the importance of not running all AI workloads on GPUs. While GPUs are essential for large-scale model training, inference for smaller models or specific tasks can be more efficiently handled by other types of accelerators (e.g., CPUs, TPUs, custom ASICs). The platform's role is to intelligently match the workload to the most suitable and energy-efficient hardware, avoiding the significant cost and energy waste of over-provisioning GPUs.

1. Addressing Diverse AI Patterns:

Different AI use cases present unique performance requirements, necessitating tailored platform strategies:

• Chatbots: Require low latency but can tolerate slightly higher latency than other interactive applications, as users are accustomed to human-like response times. Strategies like sticky management (routing requests from the same user to the same instance) can improve efficiency.
• Code Assist: Demands much lower latency for features like autocompletion, as developers expect instant feedback. A large context window is often needed to understand the entire codebase. Context truncation (eliminating irrelevant code) helps manage this.
• LLM Service: Characterized by high volume, multiple tenants, and numerous interactive requests. Requires SLO-based routing and queuing to prioritize requests according to their latency or throughput requirements.
• Document Processing: Primarily focused on throughput, processing large volumes of data efficiently.
• LLM Powered Search (RAG): A hybrid workload combining real-time LLM inference with transactional database lookups, demanding careful management of both latency and data freshness.
• Identic: Involves end-to-end latency, context awareness across transactions, and topology awareness (understanding the physical layout of resources).

1. Core Platform Optimization Techniques:

• Right Sizing and GPU Slicing: To prevent smaller LLMs from monopolizing entire GPUs, GPU slicing allows sharing a single GPU among multiple models. Techniques like NVIDIA's Multi-Instance GPU (MIG) partitions or Multi-Process Service (MPS) enable creating isolated GPU instances or contexts. Right sizing techniques, whether through profiling or analytical methods, are crucial to determine the optimal slice size for each model. The speakers highlighted Insta Slice, an open-source project (IBM Research, Red Hat, Nvidia collaboration) for dynamically slicing GPUs, coupled with right-sizing methods, to achieve significant energy and cost savings.

• Routing and Queuing: Beyond traditional load balancing, intelligent routing and queue management are vital for inference. The unpredictable nature of LLM inference (variable token generation, mixed batch/interactive requests) necessitates:
• Intelligent Load Balancing: Distributing requests based on real-time server load, model characteristics, and request SLOs.
• Intelligent Queue Management: Avoiding the head-of-the-line problem (where a long-running request blocks others), potentially through eviction policies or queue reordering based on priority or expected completion time. IBM Research's work with UIC on the VLM open-source project explores various policies for this.

• Caching (KV Cache Management): Due to the auto-regressive nature of LLMs (generating tokens one by one, where each new token depends on previous ones), caching intermediate computations is critical. The Key-Value (KV) cache stores the key and value states of attention layers, preventing redundant recomputations.
• Node-level KV Cache: Essential for each inference node to store its own cache.
• Cross-node KV Cache Sharing: A more complex problem, as load balancing might route requests from the same user or session to different nodes. Solutions like Project Mooncake (IBM Research) explore techniques such as disaggregated prefill and decoding to enable efficient sharing of KV caches across nodes, ensuring optimal reuse even in distributed environments.

• Laura Management: With potentially thousands of dynamic Laura adapters in a multi-tenant environment, efficient management is crucial. This involves intelligent caching of adapters based on anticipated demand and integrating adapter requirements into routing and queuing decisions to ensure the correct adapter is available where and when needed.

• Autoscaling and Placement: Dynamically adjusting the number of VLM instances based on real-time load is fundamental. This involves:
• Scaling Decisions: Based on metrics like queue size and request patterns.
• Optimal Placement: Deciding where to start new VLM instances, considering factors like existing cache locations, hardware heterogeneity, and network topology to minimize latency and resource consumption.

Architectural Principles for Coherent Optimization:

The speakers emphasized that these techniques are not isolated but form an intersectionality where their combined effect must be maximized. This requires:

• Modularity and Separation of Concerns: Well-defined APIs, clear control flow, and data flow between optimization components (e.g., what routing needs to know about caching).
• Data-Driven Optimization: Robust benchmarking tools to evaluate the benefits of individual techniques and, crucially, their combinations.
• Leveraging Best-of-Breed Open Source Technologies: Fostering a community approach to accelerate innovation and allow for rapid testing of new algorithms (e.g., combining a favorite router with a new distributed KV cache algorithm).
• Support for Heterogeneity: Designing platforms that can seamlessly integrate and manage diverse hardware and software components for maximum efficiency and cost reduction.

Demo / Proof of Concept

▶ Watch: Strategies for optimizing the AI life cycle and supply chain (6:00)

While the talk did not feature a live demonstration, the speakers extensively referenced several active open-source projects and research initiatives that serve as practical proof-of-concept implementations for the discussed Green AI strategies. These projects embody the principles of modularity, community collaboration, and data-driven optimization.

Key examples include:

• Insta Slice: An open-source project developed in collaboration between IBM Research, Red Hat, and NVIDIA. Insta Slice enables the dynamic slicing of GPUs, allowing smaller LLM models to utilize only a fraction of a GPU's resources rather than the entire unit. This directly addresses the GPU underutilization problem, leading to significant energy and cost savings by maximizing hardware efficiency. The talk highlighted its combination with IBM Research's work on right-sizing techniques as a powerful duo for resource optimization.
• VLM Open Source Project: This project, resulting from collaboration between IBM Research and UIC, focuses on intelligent routing and queuing policies for large language model workloads. It tackles challenges like avoiding the head-of-the-line problem, managing diverse request types (batch vs. interactive), and dynamically reordering queues to optimize throughput and latency.
• Project Mooncake: An IBM Research initiative exploring advanced KV cache management techniques, particularly focusing on how to share KV caches across multiple nodes in a distributed inference environment. This project delves into concepts like disaggregated prefill and decoding to overcome the complexities of maintaining cache coherence and availability across a cluster, which is crucial for efficient auto-regressive LLM inference.

These projects demonstrate the feasibility and impact of the proposed technical optimizations, providing tangible examples of how research and community efforts are translating into deployable solutions for sustainable AI infrastructure. They are presented as ongoing contributions to the open-source ecosystem, inviting further technical contributions and reviews from the community.

Defensive Implications

▶ Watch: Why AI inference is a critical focus for optimization (7:20)

The insights from this KubeCon talk carry significant implications for organizations and security professionals responsible for designing, deploying, and maintaining AI systems. The shift towards Green AI is not just an environmental imperative but a strategic necessity for operational resilience, cost efficiency, and regulatory compliance.

1. Prepare for Regulatory Compliance: The most immediate defensive implication is the need to prepare for forthcoming regulations like the EU AI Act, which mandates the disclosure of AI system energy consumption. Organizations must establish robust mechanisms for monitoring, measuring, and reporting the energy usage of their AI workloads, especially inference. This requires implementing observability tools like Kepler (a CNCF project for energy observability) to gain visibility into the environmental footprint of cloud-native AI deployments. Failure to comply could result in significant penalties and reputational damage.

1. Optimize AI Infrastructure for Cost and Resilience: Inefficiency in AI infrastructure translates directly into higher operational costs and increased susceptibility to resource bottlenecks. By adopting the platform-level optimization techniques discussed (e.g., GPU slicing, intelligent routing, KV cache management, autoscaling), organizations can:

• Reduce operational expenditure: Lower energy consumption means lower utility bills and reduced cloud costs.
• Improve resource utilization: Maximize the return on investment for expensive AI accelerators by ensuring they are used efficiently, reducing the need for premature hardware upgrades.
• Enhance system resilience: Efficient resource allocation can prevent overload scenarios and ensure consistent performance under varying loads, directly impacting SLOs.

1. Strategic Hardware Procurement and Management: The talk highlighted the substantial embodied carbon in hardware manufacturing. Defenders should advocate for a "fit-for-purpose" approach to hardware procurement, avoiding the default assumption that all AI workloads require top-tier GPUs. Instead, evaluate the specific needs of inference workloads and consider heterogeneous infrastructure that includes more energy-efficient accelerators or even CPUs for less demanding tasks. This reduces both embodied carbon and acquisition costs, while extending the lifespan of existing hardware.

1. Integrate Sustainability into MLOps and DevSecOps: Green AI principles must be integrated throughout the entire ML lifecycle, from data preparation to model deployment. This means:

• "Shift Left" for Sustainability: Incorporate energy efficiency considerations early in the model development process, for example, by prioritizing data distillation to reduce training data volume or exploring smaller, more efficient model architectures like Laura adapters or Mixtures of Experts.
• Automate Green Practices: Leverage cloud-native tools and automation to implement and enforce efficient deployment strategies, such as dynamic GPU slicing and intelligent autoscaling.
• Security and Sustainability Synergy: Just as security is "baked in" during DevSecOps, sustainability should be a core design principle for AI systems, not an afterthought.

1. Leverage Open-Source and Community Standards: The complexity of AI optimization requires collaborative solutions. Organizations should actively engage with and contribute to open-source projects and community initiatives like those within the CNCF AI working group and TAG Environmental Sustainability. Adopting standards like Gateway API extensions for LLM inference and best practices from the Sustainable AI white paper can accelerate the adoption of proven, efficient architectures, reducing the burden on individual teams to reinvent solutions. This collaborative approach also helps in staying abreast of rapidly evolving optimization techniques.

By proactively addressing these defensive implications, organizations can transform the challenge of AI sustainability into an opportunity for innovation, cost savings, and a stronger, more responsible AI posture.

Key Takeaways

• AI's Energy Footprint is a Crisis: The exponential growth of AI's energy consumption, with 19% of data center power projected for AI by 2028, necessitates urgent action and is driving new regulations like the EU AI Act.
• Inference is the Primary Carbon Culprit: AI inference, not just training, accounts for the majority (65%) of an AI system's operational carbon footprint and significantly impacts embodied carbon in hardware. Optimizing inference offers an 800x improvement potential.
• Green AI is Synonymous with Efficient AI: Reducing resource consumption (compute, network, storage) directly translates to lower energy usage and a smaller environmental impact, aligning sustainability goals with operational efficiency and cost savings.
• Platform-Level Optimizations are Critical: Intelligent platform strategies—including GPU slicing (e.g., Insta Slice), smart routing and queuing (e.g., VLM project), KV cache management (e.g., Project Mooncake), and autoscaling—are essential for bridging diverse AI workloads with heterogeneous hardware efficiently.
• Optimizations are Interdependent and Require Coherence: Various techniques are not isolated but form an "intersectionality" where they must work together synergistically to maximize overall efficiency, demanding modular architectures and data-driven approaches.
• Community and Standardization are Key Enablers: Collaborative efforts through open-source projects and initiatives within organizations like the CNCF (e.g., Kepler, Gateway API extensions, Sustainable AI white paper) are vital for sharing best practices, standardizing solutions, and accelerating the adoption of Green AI.

About the Speaker(s)

Vincent Caldeira serves as the CTO for Asia Pacific at Red Hat, based in Singapore. His role involves leading technological strategy and innovation across the region. Vincent is actively involved in the cloud-native community, particularly as a member of the Cloud Native AI working group within the CNCF, where he contributes to shaping best practices for AI in cloud-native environments.

Tamar Eilam is an IBM Fellow and the Chief Scientist for Sustainable Computing at IBM Research. Her work focuses on advancing the state of the art in energy-efficient and sustainable computing solutions. Tamar's expertise lies in developing innovative approaches to optimize resource utilization across various computing paradigms, including AI, with a strong emphasis on practical, research-backed solutions.