Building a 5* Kubernetes Hotel - Dean Fuller & Rachael Wonnacott

Building a 5* Kubernetes Hotel

Speakers: Dean Fuller, Lead Developer Platform Engineering; Rachael Wonnacott, Technical Product Owner, Kubernetes Platform

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=ahANKkTT-yo

Overview

In "Building a 5\* Kubernetes Hotel," Dean Fuller, Lead of Developer Platform Engineering, and Rachael Wonnacott, Technical Product Owner for the Kubernetes platform at Fidelity International, delve into their organization's ambitious journey to revolutionize its internal developer experience. The talk outlines Fidelity's evolution from a "cloud house" model, where application teams managed their own infrastructure, to a "Kubernetes hotel" paradigm, where a centralized platform team provides a fully managed, opinionated Kubernetes environment. This shift aims to empower Fidelity's 2,500 developers to focus solely on delivering business value, abstracting away the complexities of underlying infrastructure.

Fidelity International, a global financial services giant established in 1969, managing $950 billion across 27 countries, embarked on its cloud journey in 2013, initially with Cloud Foundry and later expanding to AWS and Azure. The core challenge addressed in this presentation is the deceleration of innovation caused by developers being overburdened with infrastructure management. The "hotel" model represents a strategic pivot to a platform-as-a-service (PaaS)-like experience on Kubernetes, promising standardization, resilience, and enhanced security, while navigating the inevitable "hype cycle" of product adoption and managing the cognitive load inherent in such a transformation. This talk provides invaluable insights for any large enterprise grappling with developer productivity and platform engineering challenges in a highly regulated environment.

Background

▶ Watch: Speakers introduce Fidelity's initial 'Cloud House' DevOps model (0:00)

Fidelity International's extensive cloud journey began in 2013 with an on-premise Cloud Foundry implementation, which proved highly successful. This was followed by a strategic move to public cloud providers like AWS and Azure over the last seven years. Initially, Fidelity adopted a purist DevOps approach, framed as the "cloud house" model. In this model, the central cloud platform team built the foundational infrastructure—analogous to providing the water mains to a house—while individual application teams (customers) were responsible for constructing and maintaining everything within their "house," including taps, bathrooms, and kitchens. This approach fostered significant innovation, leading to over 600 such "houses" across the organization.

However, over time, the initial acceleration of cloud adoption began to slow. Feedback from app developers revealed a growing dissatisfaction: they preferred to "focus on core business delivery and not worry too much about the underlying infrastructure." Managing infrastructure, while initially seen as empowering, became a significant burden, diverting valuable time and expertise from core business logic. This realization prompted a strategic re-evaluation, leading to the conceptualization of the "cloud container hotel." In this new model, the platform engineering team would provide fully furnished, standardized "hotel rooms"—a managed Kubernetes environment—taking on the responsibility for infrastructure maintenance and issues. This shift aimed to liberate application developers from infrastructure concerns, allowing them to concentrate entirely on their applications.

The decision to pivot specifically to Kubernetes was driven by several factors. The Cloud Foundry community was diminishing, and industry trends clearly indicated Kubernetes' dominance, with many third-party vendors expecting its presence and providing Helm charts. Furthermore, adopting Kubernetes offered better career vitality for engineers and was a prerequisite for certain business-critical applications, such as those interacting with the Bloomberg API. Fidelity, as a regulated company, also had stringent guardrails, particularly around network security and network isolation, which became non-negotiable constraints for the Minimum Viable Product (MVP) of the new platform. This necessitated the platform team's deep involvement in Kubernetes' core functionality to meet these enterprise-grade requirements.

Key Findings

▶ Watch: Explaining the original shared responsibility model of the 'Cloud House' (2:00)

Fidelity's journey revealed several critical insights into platform adoption and engineering within a large, regulated enterprise:

1. The "Hotel Hype Cycle": The initial vision of a "five-star Kubernetes hotel" created a "peak of expectation" among developers. However, the reality of the Minimum Viable Product (MVP), which the platform team accurately delivered, was perceived as a "one-star motel." This gap led to a "trough of disillusionment." Crucially, this initial feedback, though difficult to hear, was vital. It acted as an "engineering trigger" and drove the platform towards a "path of enlightenment," allowing the team to iteratively add features that truly transformed the "motel" into a more viable "hotel." The speakers emphasized that going live with the MVP, even if imperfect, was essential to gather this invaluable, real-world feedback.

1. Conservation of Cognitive Load: The talk introduced the concept of the "conservation of cognitive load." While the "hotel" model successfully abstracted infrastructure complexity from application developers, that load does not disappear; it shifts. The platform engineering team, despite building on managed services like Amazon EKS, found themselves heavily burdened by the operational overhead of managing numerous open-source CNCF projects and ensuring the platform's stability. This highlighted that abstracting infrastructure for one group necessitates a dedicated and skilled team to absorb that complexity.

1. Balancing Priorities in a Regulated Environment: For Fidelity, a financial institution managing $950 billion, security was a "total non-negotiable." This, along with resiliency and scaling, took precedence during the initial build. Significant time and effort were invested in these areas, including upskilling internal security teams on Kubernetes. This prioritization, while essential for compliance and trust, meant that capability and developer experience (DevX) initially lagged. The speakers acknowledged this trade-off, stressing that while DevX is a key selling point, business value, audit, and compliance are paramount in their context.

1. Open Source as a Foundation Requires Significant Investment: The platform heavily leverages open-source projects like Cilium, Kyverno, and Carpenter. While these tools are foundational for a robust Kubernetes platform, the team discovered the substantial time and effort required to integrate, maintain, and support these plugins within their enterprise environment. This included dealing with "test flakes" – intermittent test failures indicative of an unstable environment – and lengthy CI/CD pipelines due to destroying and rebuilding clusters for every change.

1. The Need for a Multi-Cluster Strategy: As the platform grew, the team encountered challenges with blast radius and cyclical dependencies (or "chicken-and-egg dependencies"). Hosting all services, including critical infrastructure components like authentication or scanning software, on a single cluster increased the risk of widespread outages. To address this and further manage cognitive load, the team is exploring a multi-cluster approach, dedicating separate clusters for different use cases (e.g., core infrastructure services vs. application workloads). This strategy aims to distribute responsibility and enhance resilience.

Technical Deep Dive

▶ Watch: Why initial cloud journey slowed: developers wanted to focus on business (2:40)

Fidelity's Kubernetes platform is built upon Amazon EKS, leveraging its managed service capabilities to reduce some of the underlying operational burden. However, to transform EKS into an enterprise-grade "hotel," the platform team integrated several crucial CNCF projects and open-source tools.

For the initial Minimum Viable Product (MVP), the focus was heavily on security, resilience, and cost-effectiveness, particularly for a regulated multi-tenant environment. The three core CNCF projects chosen were:

• Cilium: Selected as the Container Network Interface (CNI). Cilium is critical for providing advanced network security, enforcing network policies, and ensuring network isolation between different tenants (applications) within the shared Kubernetes clusters. This was a non-negotiable requirement for Fidelity's regulated environment.
• Kyverno: Utilized for policy enforcement. Kyverno allows the platform team to define and apply policies as code, ensuring that all deployed resources adhere to Fidelity's security, compliance, and operational standards. This provides automated guardrails, preventing misconfigurations and enforcing best practices across the "hotel rooms."
• Carpenter: Employed for intelligent cluster scaling to zero. This tool significantly contributes to cost savings by efficiently scaling down nodes when not in use, ensuring that resources are only consumed when needed. This is particularly important for managing a large, dynamic environment with varying workloads.

Beyond these core components, the platform supports application deployment through a centralized Argo CD service. This GitOps-driven approach enables declarative configuration for application deployments, often utilizing Helm charts. While Argo CD is the preferred method for platform-managed deployments, application teams retain the flexibility to use other CI/CD tools like Jenkins for their build pipelines.

The operational reality of managing this platform, however, presented significant challenges. The small platform team, consisting of approximately 15 engineers, faced a high cognitive load due to the sheer number of open-source add-ons and the need to maintain their stability. A notable issue was the prevalence of "test flakes"—tests that intermittently pass or fail without code changes—which indicated an underlying instability in the environment. This instability directly impacted their Continuous Integration (CI) pipeline, which involved destroying and rebuilding development clusters for every Pull Request (PR), leading to excessively long run times. As more add-ons were introduced, the number of tests and flakes increased, further exacerbating the problem.

To address the growing cognitive load and mitigate risks, the team is actively exploring a multi-cluster approach. The initial design, where Kubernetes was envisioned to handle all infrastructure services, led to a single large cluster potentially hosting both application workloads and critical platform components like authentication or scanning software. This concentrated too much responsibility on the platform team and increased the blast radius if a cluster failure occurred. The proposed multi-cluster strategy aims to segment workloads, potentially creating dedicated clusters for specific infrastructure services, thereby distributing the cognitive load among multiple teams and reducing the impact of any single cluster outage. This also helps address "chicken-and-egg dependencies," where essential platform services might inadvertently depend on the very cluster they are meant to support.

Furthermore, the platform offers ancillary services, such as a declarative database-as-a-service. Application teams can specify their database requirements (flavor, size) within their declarative configuration, which then triggers an automated pipeline, managed by a separate team, to provision the database within their namespace. This extends the "hotel" concept beyond just compute, providing a more comprehensive PaaS-like experience.

Demo / Proof of Concept

▶ Watch: The new 'Cloud Container Hotel' model to address developer needs (4:00)

The talk primarily focused on Fidelity's journey, strategic decisions, and the operational realities of building their Kubernetes platform rather than a live demonstration or proof of concept of the platform itself. While the speakers described the evolution from a "motel" (MVP) to a more capable "hotel," the presentation did not include a technical walkthrough or live demo of the platform's features or its underlying components in action. The emphasis was on the architectural and organizational challenges and solutions encountered throughout their implementation.

Defensive Implications

▶ Watch: Technical reasons for choosing Kubernetes for the new platform (5:40)

For a regulated entity like Fidelity International, security is paramount, and the "Kubernetes hotel" model incorporates several defensive strategies:

1. Security as a Non-Negotiable Baseline: From the outset, security was deemed a "total non-negotiable." This meant that corners could not be cut, and significant investment was made in securing the platform. This included not only implementing robust technical controls but also dedicating time to upskill internal security teams on Kubernetes specifics, ensuring alignment and understanding across the organization. Any organization in a regulated industry must adopt a similar mindset, prioritizing security even if it initially impacts other areas like developer experience or feature velocity.

1. Robust Policy Enforcement with Kyverno: The use of Kyverno for policy enforcement is a critical defensive measure. It allows Fidelity to define and automatically enforce security policies, compliance rules, and best practices across all Kubernetes clusters and namespaces. This prevents unauthorized configurations, ensures consistent security posture, and provides an auditable trail of policy adherence, which is crucial for meeting regulatory requirements.

1. Network Security and Isolation with Cilium: Cilium as the CNI provides advanced network security capabilities essential for a multi-tenant environment. It enables fine-grained network policies, ensuring strict network isolation between different application teams and preventing lateral movement in case of a breach in one "hotel room." This micro-segmentation capability significantly reduces the blast radius of potential security incidents.

1. Managing Blast Radius through Multi-Cluster Architecture: The realization that hosting all services (including critical infrastructure like authentication or container scanning) on a single cluster increases the blast radius led to the exploration of a multi-cluster approach. By isolating different types of workloads or critical platform services into separate clusters, Fidelity can limit the impact of a security event or operational failure to a contained segment, preventing a cascading failure across the entire organization.

1. Addressing Cyclical Dependencies for Resiliency: The concept of "chicken-and-egg dependencies" highlights a defensive concern regarding platform recovery. If core services required for cluster bootstrapping or authentication are hosted on the very cluster that needs to be rebuilt, it creates a circular dependency that could impede recovery efforts. Defensive planning requires careful architectural design to ensure such critical components are either externalized or hosted on independent, highly resilient infrastructure.

1. Focus on Platform Stability and Observability: The presence of "test flakes" and long CI/CD pipelines indicates an underlying instability that can mask security vulnerabilities or operational issues. Investing in platform stability, robust testing, and comprehensive observability is a defensive necessity. A stable environment ensures that changes are deployed predictably and securely, reducing the likelihood of introducing vulnerabilities or operational outages.

1. Enablement and Education for Secure Adoption: Fidelity's dedicated enablement function plays a crucial defensive role. By actively working with application developers, this team guides them on best practices for designing and deploying secure applications on Kubernetes, troubleshooting, and understanding the platform's security model. This proactive education minimizes the risk of insecure application deployments and fosters a shared responsibility for security within the "hotel" model.

Key Takeaways

• Embrace the MVP, Manage Expectations: Launching an MVP, even if it feels like a "motel" rather than a "five-star hotel," is crucial for gathering real-world customer feedback. This feedback, though potentially critical, is invaluable for guiding the platform's evolution and ensuring it meets actual user needs.
• Open Source is Powerful but Demands Investment: Leveraging CNCF projects and other open-source tools is fundamental for building a robust Kubernetes platform. However, be prepared for significant operational overhead in terms of integration, maintenance, testing, and supporting these components within an enterprise environment.
• Cognitive Load Shifts, It Doesn't Disappear: Abstracting infrastructure complexity from application developers means that the platform engineering team absorbs that load. Plan for this increased responsibility and resource allocation for the platform team.
• Prioritize Security, Resiliency, and Scaling in Regulated Environments: For financial institutions and other regulated industries, these pillars are non-negotiable and must be prioritized from day one, even if it means initially compromising on capability or developer experience. Upskilling internal security teams on new technologies is vital.
• Consider Multi-Cluster Strategies: To manage blast radius, distribute cognitive load, and address complex dependencies (e.g., core infrastructure services), a multi-cluster approach with specialized clusters for different use cases can be a necessary architectural evolution.
• Invest in Enablement and Education: A dedicated enablement function that guides developers and provides feedback to the platform team is critical for successful adoption, fostering best practices, and ensuring a smooth transition for customers migrating to the new platform.

About the Speaker(s)

Rachael Wonnacott serves as the Technical Product Owner for the Kubernetes platform at Fidelity International. With a background in physics, Rachael is responsible for translating the strategic vision into an actionable product roadmap, defining the platform's MVP, and balancing the demands of feature development with the critical need for operational stability and platform maintenance. She plays a pivotal role in navigating the complexities of delivering a managed Kubernetes experience to Fidelity's extensive developer base.

Dean Fuller leads Developer Platform Engineering at Fidelity International. He is instrumental in shaping the strategic direction of Fidelity's developer platform, including the significant shift from a purist DevOps "cloud house" model to the "Kubernetes hotel" vision. Dean focuses on empowering Fidelity's 2,500 developers across 27 countries to deliver business value efficiently, overseeing the evolution of the platform to meet the needs of a large, regulated financial organization.

Fidelity International, where both speakers are employed, is a global financial services company with a rich history dating back to 1969. It manages $950 billion in assets, operating in 27 countries, and has been on a cloud transformation journey since 2013, initially with Cloud Foundry and later expanding into public cloud environments like AWS and Azure.