Cloudy With a Chance of Kubernetes: Going From One To Three... Laurent Bernaille & Maxime Visonneau

Cloudy With a Chance of Kubernetes: Going From One To Three...

Speakers: Laurent Bernaille, Infrastructure Engineer, DataDog; Maxime Visonneau, Infrastructure Engineer, DataDog

Conference: KubeCon EU

YouTube: <https://www.youtube.com/watch?v=iCAFXF5ECto>

Overview

This talk, "Cloudy With a Chance of Kubernetes: Going From One To Three...", presented by Laurent Bernaille and Maxime Visonneau from DataDog, delves into the intricate journey of migrating and operating a massive-scale observability platform across not one, but three major public cloud providers: AWS, GCP, and Azure. The speakers share DataDog's evolution from a single-cloud, VM-centric infrastructure to a multi-cloud, Kubernetes-native architecture, highlighting the strategic decisions, technical challenges, and innovative solutions developed along the way. The core theme revolves around leveraging Kubernetes as a powerful abstraction layer to manage infrastructure and application deployments consistently across diverse cloud environments.

The presentation is a candid exploration of the benefits and limitations of using Kubernetes for true cloud agnosticism at an extreme scale. DataDog, processing over 10 trillion events daily and operating hundreds of Kubernetes clusters with tens of thousands of nodes and hundreds of thousands of pods, offers invaluable insights into the practicalities of multi-cloud operations. This talk is particularly relevant for platform engineers, SREs, and architects grappling with multi-cloud strategies, large-scale Kubernetes deployments, or the complexities of abstracting underlying cloud primitives while maintaining performance, reliability, and cost efficiency.

The speakers meticulously detail how DataDog navigated the "leaky abstractions" of cloud providers, from compute and storage to networking and higher-level services. They present a compelling case for building custom tooling and platforms on top of Kubernetes to achieve operational consistency, even when faced with significant differences in cloud provider implementations. The talk underscores that while Kubernetes offers a compelling vision for unified infrastructure, realizing that vision at scale requires deep technical expertise, strategic partnerships, and a willingness to build bespoke solutions for critical components.

Background

▶ Watch: DataDog's multi-cloud journey and rationale (0:00)

Until 2018, DataDog's infrastructure was rooted in a classic, single-region setup, running exclusively on AWS in the US. This environment relied heavily on virtual machines (VMs), managed by Chef, with applications deployed using Capistrano. This traditional approach served its purpose for a time, but as DataDog's global footprint and customer base expanded, the limitations of a single-cloud strategy became apparent.

The impetus for DataDog's multi-cloud journey was multifaceted. Firstly, there was a strong customer demand for proximity, necessitating deployments in diverse geographical regions. Secondly, DataDog embraced a philosophy of "dog fooding"—experimenting with technologies and operational models internally to better understand and serve their customers' needs, particularly as customers began adopting Kubernetes. Thirdly, expanding to multiple locations facilitated establishing strategic partnerships with various cloud providers.

This strategic shift commenced in 2018 with the establishment of DataDog's first region outside the US, on GCP in Europe. Over the subsequent years, DataDog embarked on a significant diversification, adding more main regions and 29 edge locations globally, including a dedicated region for government agencies in the US (2021) and an expansion into Asia (Tokyo). As of early 2025, DataDog operates across six main large regions and 29 edge locations, spanning AWS, GCP, and Azure.

The challenge was clear: how to manage this rapidly expanding and diversifying infrastructure without rewriting everything for each cloud. DataDog sought an infrastructure abstraction that would allow them to treat different regions and providers in a unified manner. The answer, which became evident at the time, was Kubernetes. Kubernetes offered a consistent API for interacting with infrastructure, a standardized way to package software via containers, and a unified approach to defining application rollout strategies and resource definitions that were largely cloud-agnostic.

Beyond abstraction, DataDog's decision to adopt Kubernetes was also influenced by its growing customer base using Kubernetes, making it a natural choice for enhancing their own integrations and support. The vibrant and expanding Kubernetes community provided a rich resource for problem-solving and collaboration. Furthermore, running on Kubernetes made it significantly easier to attract and hire talent in the infrastructure space, a crucial advantage for a rapidly growing company.

However, the initial landscape for managed Kubernetes services in early 2018 was nascent. EKS and AKS were still in beta, and GKE, while generally available, was limited to 500 nodes per cluster, far below DataDog's anticipated scale requirements. Managed offerings were also perceived as overly opinionated, with differing implementations for networking, operating systems, and supported Kubernetes versions, making a truly unified abstraction difficult. Crucially, they offered limited observability into the control plane components (API servers, etcd), which DataDog deemed essential for operating at their scale. This led DataDog to make the strategic decision to self-manage their Kubernetes clusters.

Key Findings

▶ Watch: Infrastructure evolution: from single region to global presence (2:00)

DataDog's journey reveals several pivotal findings regarding multi-cloud Kubernetes operations at extreme scale:

• Kubernetes as a Leaky Abstraction: While Kubernetes offers a powerful API for infrastructure abstraction, it does not entirely eliminate the underlying differences between cloud providers. Critical primitives like compute, storage, and networking often "leak" their provider-specific characteristics, necessitating deep custom integrations and a nuanced understanding of each cloud's nuances.
• Self-Hosting Control Planes within Kubernetes: A significant innovation was the shift to running Kubernetes control plane components as pods within "parent" Kubernetes clusters. This allowed DataDog to leverage the same Helm charts and the Software Delivery Platform used for applications to manage and update their Kubernetes clusters, dramatically simplifying the operational overhead for hundreds of clusters compared to traditional Infrastructure as Code (IaC) tools like Terraform and Ansible for every cluster.
• Custom Software Delivery Platform for Multi-Cloud Deployments: To manage application deployments across numerous regions and clusters with varying configurations, DataDog built a custom Software Delivery Platform. This platform orchestrates complex Temporal workflows, enabling phased rollouts, automated health checks, and conditional progression, significantly de-risking deployments and standardizing the process for application teams.
• Evolving Compute Abstraction: DataDog transitioned from a one-pod-per-node model with cloud provider autoscaling groups to a more sophisticated system using custom Node Group Custom Resource Definitions (CRDs). This allowed users to declare resource needs abstractly, while an infrastructure controller and the Cluster Autoscaler dynamically provisioned and optimized underlying instances across clouds, accommodating diverse CPU architectures and cost-effective hardware generations.
• Standardization with CSI and CNI: The Container Storage Interface (CSI) has been crucial for abstracting storage, providing a consistent API for block devices across providers. Similarly, adopting Cilium as a unified Container Network Interface (CNI) plugin enabled native pod IP addressing, robust network policies, and enhanced routing capabilities across multi-cloud environments, overcoming provider-specific networking complexities.
• Higher-Level Services Require Scrutiny: While object storage services (S3, GCS, Blob Storage) proved largely consistent across providers, higher-level services like L7 load balancers and managed databases (Postgres, Redis) exhibited sufficient behavioral differences and inconsistencies. This often led DataDog to either abstract to a lower layer (L4 load balancing with Envoy for L7) or self-manage these services at a platform level to ensure consistent behavior and control.
• Non-Technical Challenges are Paramount: The success of multi-cloud Kubernetes is not solely technical. It demands developing deep expertise in both Kubernetes and individual cloud providers, fostering strong relationships with providers for support and issue resolution, and investing heavily in internal training and platform adoption to empower application teams. Addressing the "last 20%" of edge cases remains an ongoing challenge for platform teams.

Technical Deep Dive

▶ Watch: Why Kubernetes was chosen for infrastructure abstraction (3:50)

DataDog's journey began with a traditional infrastructure model. Until 2018, everything ran on AWS in a single region, utilizing virtual machines (VMs) managed by Chef, with application deployments handled by Capistrano. This setup, while functional, lacked the flexibility and scalability required for a multi-cloud expansion.

The decision to adopt Kubernetes as the core abstraction layer led to an initial phase where Kubernetes control planes were deployed on VMs, and worker nodes were also VMs. This self-managed approach utilized familiar tools like Terraform, Ansible, Packer, and shell scripts for orchestration. However, as the number of regions and clusters grew—with some regions requiring multiple clusters due to a 5,000-node cluster limit—managing updates to the control planes became increasingly complex and time-consuming.

To address this, DataDog implemented a significant shift: running Kubernetes control plane components as pods within other Kubernetes clusters. This architecture involves a "parent" Kubernetes cluster, whose control plane is still managed on VMs (though the goal is to eventually run this on a managed Kubernetes offering). The worker nodes of this parent cluster then host the control planes (API server, etcd, controller manager, scheduler) of "child" Kubernetes clusters, running them as standard pods. This innovation allowed DataDog to use Helm charts and their existing application deployment primitives to update and roll out control plane components, streamlining a previously arduous process across dozens or hundreds of clusters.

For application deployments, DataDog developed a sophisticated Software Delivery Platform. This platform orchestrates complex, multi-region, multi-cluster deployments using Temporal workflows. When an application team initiates an update (e.g., from version blue to green), the platform executes a predefined workflow. This typically involves deploying to one region (e.g., Asia), pausing to perform verification checks using monitors and metrics, and waiting for stabilization. If successful, it proceeds to the next region (e.g., GCP), potentially iterating over multiple clusters within that region, before finally moving to the last region (e.g., AWS). This phased, automated approach ensures consistency, reduces risk, and provides a standardized deployment experience. Crucially, this same platform was later leveraged to manage updates for the Kubernetes control planes themselves, demonstrating the power of unified tooling.

The abstraction of compute resources evolved significantly. Initially, to ease migration from VMs, DataDog adopted a single pod per node model, relying on cloud provider autoscaling groups (e.g., AWS ASGs). While simple, this approach was inefficient and still exposed provider-specific instance types to application teams. DataDog then introduced Custom Resource Definitions (CRDs) for Node Groups. Users would declare their desired node configurations (e.g., instance types, regions) through these CRDs. A custom controller operating on each cluster would reconcile these CRDs with the underlying cloud provider APIs, creating and managing the respective autoscaling groups. The Cluster Autoscaler was then used to dynamically adjust the number of nodes within these groups based on pending pod schedules, with out-of-the-box support for all major cloud providers. The ultimate goal was to abstract away node types entirely, allowing application teams to specify only resource requests and limits for their pods, while the infrastructure team managed the underlying node groups, leveraging diverse CPU architectures, accelerators, and cost-effective hardware generations without application team intervention.

Storage abstraction centered around providing pods with access to block devices. While cloud providers offer local and remotely managed block storage, the key to multi-cloud consistency was the Container Storage Interface (CSI). CSI drivers, implemented by providers, standardize how Kubernetes interacts with different storage systems. This provided out-of-the-box support for DataDog's chosen providers and enabled advanced capabilities like volume snapshots. However, DataDog noted inconsistencies in offerings, varying maturity of CSI drivers across providers, and challenges with migration efforts. Furthermore, Kubernetes' pre-CSI internal volume plugins necessitated a later migration, highlighting the importance of adopting industry standards early.

Networking was another critical area. DataDog opted for native Software Defined Networking (SDN), directly addressing pods with IPs on the cloud provider's network, rather than using overlay networks. This choice was driven by a desire to avoid performance overhead and increase reliability by reducing moving parts in the data path. Native SDN also enabled cross-network connectivity between pods in different clusters. The initial implementation required deep integration with each cloud provider's SDN and necessitated different Container Network Interface (CNI) plugins for different providers. To unify this, DataDog eventually transitioned to Cilium, which provided out-of-the-box multi-cloud support, network policy enforcement, improved routing via eBPF, and replaced kube-proxy. Scalability remains an ongoing challenge for Cilium at DataDog's operational scale.

For higher-level services, DataDog found varied success with cloud provider offerings. Object storage (S3, GCS, Blob Storage) proved largely consistent in interfaces and behavior, making it a successful multi-cloud primitive. Load balancers, however, presented more challenges. While L4 load balancing was sufficiently similar, L7 load balancing varied significantly across providers. DataDog responded by standardizing on L4 load balancing and managing L7 properties themselves using Envoy. Crucially, for managed databases like Postgres and Redis (initially RDS and Cloud SQL), DataDog encountered too many subtle behavioral differences and operational pain points. These inconsistencies forced application teams to understand provider-specific quirks, undermining the abstraction. As a result, DataDog made the strategic decision to implement these database services themselves at a platform level, gaining full control and ensuring consistent behavior across all clouds.

Demo / Proof of Concept

▶ Watch: Challenges with managed Kubernetes offerings in 2018 (5:30)

The talk primarily focused on architectural descriptions and workflow illustrations rather than a live code demonstration or proof-of-concept execution. The speakers utilized detailed diagrams to explain the "Software Delivery Platform" workflow for application updates across multiple regions and the "parent-child Kubernetes cluster" model for self-hosting control planes. These visual aids effectively conveyed the complex systems and processes DataDog developed to achieve their multi-cloud Kubernetes objectives.

Defensive Implications

▶ Watch: DataDog's initial approach to self-managing Kubernetes clusters (7:00)

Operating Kubernetes at DataDog's scale across multiple cloud providers introduces unique security challenges and opportunities for robust defense. The strategies and tools discussed in the talk have significant defensive implications:

• Standardized Security Policies via Kubernetes Abstraction: By using Kubernetes as a unified abstraction layer, security teams can define and enforce consistent network policies (e.g., NetworkPolicy via Cilium), access controls (e.g., RBAC), and resource configurations across AWS, GCP, and Azure. This reduces the complexity of managing disparate security controls for each cloud, leading to a more predictable and auditable security posture.
• Enhanced Control Plane Observability and Security: Self-managing Kubernetes control planes, especially when running them as pods within other clusters, provides DataDog with unparalleled observability into critical components like API servers and etcd. This direct access is vital for monitoring for suspicious activity, auditing configurations, and rapidly responding to potential compromises of the cluster's brain. It also allows for hardening these components beyond what managed services might offer.
• Automated, Secure Deployment Workflows: The Software Delivery Platform with its Temporal workflows is a powerful defensive tool. By enforcing phased rollouts, automated health checks, and conditional progression, it minimizes the blast radius of misconfigurations or security regressions. If a deployment introduces a vulnerability or breaks a security control, the platform can halt the rollout, preventing wider exposure. Integrated monitoring and metrics provide immediate feedback on the security impact of changes.
• Least Privilege and Optimized Compute Security: The evolution of compute abstraction, from dedicated nodes to managed node groups driven by pod resource requests/limits, enables better security. Security teams can work with infrastructure to define hardened instance types, enforce least privilege by ensuring pods only receive necessary resources, and leverage specific hardware features (e.g., confidential computing, secure enclaves) where available and appropriate, abstracting this complexity from application teams.
• Consistent Storage Security with CSI: The Container Storage Interface (CSI) standardizes how volumes are provisioned and managed. This allows for consistent application of storage security controls such as encryption at rest, access control policies, and regular snapshotting for data recovery and integrity checks, regardless of the underlying cloud provider's storage offering. Defenders must, however, be aware of the varying maturity and potential vulnerabilities in provider-specific CSI drivers.
• Granular Network Policy and Visibility with Cilium: Adopting Cilium for network management provides advanced defensive capabilities. Its eBPF-powered architecture enables highly granular NetworkPolicy enforcement, allowing precise control over inter-pod communication and micro-segmentation across multi-cloud environments. Cilium also offers deep visibility into network flows, aiding in threat detection, incident response, and compliance auditing.
• Strategic Self-Management of Critical Services: DataDog's decision to self-manage databases and L7 load balancing due to provider inconsistencies highlights a crucial defensive strategy. For critical services where provider differences could lead to security gaps or non-compliance, taking ownership allows the platform team to implement uniform security best practices, auditing, and hardening measures, ensuring consistent protection across all cloud environments.
• Expertise, Relationships, and Training: Finally, the non-technical challenges identified—deep expertise, strong provider relationships, and internal training—are foundational for defense. A well-trained team with a deep understanding of both Kubernetes and cloud specifics is better equipped to identify and mitigate vulnerabilities. Strong relationships with cloud providers facilitate faster resolution of platform-level security issues and access to early security advisories. Training application teams to use the platform securely is paramount for overall organizational resilience.

Key Takeaways

• Kubernetes is a Powerful but Leaky Abstraction: While Kubernetes provides a unified API for multi-cloud infrastructure, underlying cloud primitives (compute, storage, network) inevitably "leak" their provider-specific differences, necessitating custom abstractions and deep technical understanding.
• Self-Hosting Kubernetes Control Planes Streamlines Operations: Running Kubernetes control plane components as pods within other Kubernetes clusters allows for consistent, automated updates using existing application deployment tools (like Helm and DataDog's Software Delivery Platform), significantly reducing the operational burden of managing hundreds of clusters.
• Custom Platforms are Essential for Multi-Cloud Consistency: Building bespoke tools like DataDog's Software Delivery Platform for phased, automated deployments with integrated health checks is crucial for achieving consistent and reliable application and infrastructure rollouts across diverse cloud environments.
• Standardization with CSI and Unified CNI (Cilium) is Key: Adopting industry standards like the Container Storage Interface (CSI) and a unified Container Network Interface (CNI) like Cilium helps abstract storage and networking complexities, enabling consistent management and robust policy enforcement across multiple cloud providers.
• Evaluate Higher-Level Cloud Services Critically: While some higher-level services (e.g., object storage) are sufficiently consistent, others (e.g., L7 load balancers, managed databases) may exhibit enough behavioral differences to warrant abstracting to a lower layer or self-managing them at a platform level for true multi-cloud consistency and control.
• Success Requires Deep Expertise and Strategic Partnerships: Effective multi-cloud Kubernetes operations at scale demand not only advanced technical skills in Kubernetes and individual cloud providers but also strong relationships with providers and continuous internal training to empower teams and address complex challenges.

About the Speaker(s)

Laurent Bernaille and Maxime Visonneau are both experienced Infrastructure Engineers at DataDog. They are part of the infrastructure engineering group, responsible for designing, building, and operating DataDog's massive-scale observability platform across a multi-cloud environment. Their work focuses on leveraging Kubernetes to manage hundreds of clusters, tens of thousands of nodes, and hundreds of thousands of pods across AWS, GCP, and Azure. They are active in the Kubernetes community and can be found on the Kubernetes Slack for further discussion.