A Journey To Modernizing a Regulated Cloud Control Plane - Pranita Praveen & Steven Borrelli

A Journey To Modernizing a Regulated Cloud Control Plane

Speakers: Pranita Praveen, Director of Engineering, Macquarie; Steven Borrelli, Principal Solutions Architect, Upbound

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=TELnK0PrKHU

Overview

This talk, presented by Pranita Praveen of Macquarie Group and Steven Borrelli of Upbound, details Macquarie's ambitious journey to modernize its cloud control plane within a highly regulated financial services environment. Facing challenges with a decade-old, cumbersome infrastructure pipeline across a multicloud estate (AWS, GCP, Azure), Macquarie sought to drastically improve developer experience and accelerate the enablement of new cloud services. The presentation outlines their strategic adoption of a Kubernetes-native, universal control plane leveraging Crossplane, Argo CD, and GitOps principles.

The core of Macquarie's transformation involved moving away from disparate, manual, and pipeline-centric deployment models to a unified, automated, and continuously reconciled control plane. This shift was critical not only for operational efficiency and reducing platform engineer toil but also for embedding stringent security and compliance guardrails earlier in the development lifecycle. The speakers highlight the technical decisions, architectural patterns, and custom integrations required to achieve this, particularly focusing on how they built a "bot of bots" workflow to provide immediate feedback on infrastructure deployments.

Ultimately, this talk provides a compelling case study for large, regulated enterprises navigating complex multicloud strategies. It demonstrates how modern cloud-native technologies, particularly Crossplane's ability to extend Kubernetes' declarative management to external cloud resources, can be harnessed to overcome legacy infrastructure challenges, enhance security posture, and significantly boost developer productivity in a highly controlled environment.

Background

▶ Watch: Introduction to Macquarie Group and platform challenges (0:00)

Macquarie Group, a global financial services organization with over 20,000 employees, embarked on its public cloud journey approximately 10 years ago. Starting with AWS, they have since expanded to GCP and Microsoft Azure, achieving over 70% cloud adoption, predominantly on IaaS. While this initial migration successfully moved a significant portion of their technology stack to the cloud, it inadvertently created new operational challenges.

Over the decade, the infrastructure pipelines used for cloud deployments became increasingly cumbersome to maintain, leading to significant toil for platform engineers. A major pain point was the existence of different deployment solutions for each cloud provider (AWS, GCP, Azure), resulting in an operations-heavy model. This fragmented approach hindered the rapid consumption of new cloud services, a critical capability for innovation in a fast-paced industry. Furthermore, as a highly regulated entity, Macquarie faced stringent requirements for security boundaries and guardrails around any new cloud service, even for development and testing, making "service enablement" a slow, manual process.

To address these issues, Macquarie sought to uplift its cloud strategy with several core principles: deploying a single multicloud deployment solution, embracing GitOps for automation, adhering to cloud-native standards for security and maintainability, and leveraging the continuous reconciliation capabilities inherent in control planes. This led them to explore solutions that could provide a universal interface for managing infrastructure across their diverse cloud estate, ultimately guiding them to Crossplane. Crossplane, an open-source Kubernetes add-on, extends Kubernetes' declarative API to manage external infrastructure, aligning perfectly with Macquarie's vision for a modern, automated, and secure control plane.

Key Findings

▶ Watch: Defining key principles: control planes, GitOps, cloud-native (3:10)

Macquarie's journey revealed several key findings and contributions crucial for modernizing a regulated cloud control plane:

1. The Efficacy of a Universal Control Plane for Multicloud: The primary finding was the successful implementation of a single, unified control plane based on Kubernetes and Crossplane, capable of managing infrastructure across AWS, GCP, and Azure. This eliminated the need for disparate tools and processes, significantly reducing operational overhead and developer toil.
2. Accelerated Service Enablement in Regulated Environments: By automating controls and removing manual steps, Macquarie achieved super-fast service enablement. Developers could provision and test new cloud services within a day or two, a dramatic improvement over previous timelines, while still adhering to strict security and compliance requirements through automated guardrails.
3. Enhanced Developer Experience through Shift-Left Feedback: A significant contribution was the development of the "bot of bots" workflow. This innovative GitOps-driven process provides immediate, pre-deployment feedback on security, compliance, and configuration directly within a pull request. This shift-left approach drastically improved developer experience by catching issues in seconds rather than waiting for cloud-side deployments to fail.
4. Successful Brownfield Integration Strategy: Recognizing the need to integrate with a decade of existing cloud infrastructure, Macquarie opted for a "brownfield" control plane. They developed custom Golang APIs (Identity, Access Management, Compliance) to bridge the new Crossplane-based control plane with their existing AWS pipelines and resources, ensuring seamless interoperability and avoiding a disruptive greenfield-only approach.
5. Leveraging Cloud-Native and Community-Driven Tools: The project demonstrated the benefits of building on a foundation of CNCF projects and community-supported tools like Crossplane, Argo CD, Argo Workflows, and Gatekeeper. This approach simplified maintenance, improved security posture (e.g., easier CVE management), and provided access to a rich ecosystem of features and expertise.

Technical Deep Dive

▶ Watch: Steven introduces Crossplane as a universal control plane (4:00)

Macquarie's modern control plane is a sophisticated integration of cloud-native technologies, designed to provide a unified, automated, and secure infrastructure management solution across their multicloud environment.

The Control Plane Concept

Steven Borrelli introduced the foundational concept of a control plane, tracing its origins to the rise of the internet in the '90s for managing millions of network routes. Key characteristics emerged: a northbound interface for client interaction and a southbound interface for sending commands to devices. Kubernetes is presented as a modern, popular control plane, with its API server acting as the northbound interface and controllers as the southbound interface. Crossplane extends this paradigm, aiming to create a universal control plane by using Kubernetes' core technologies – Custom Resource Definitions (CRDs) and controllers – to manage almost any external resource.

Crossplane Fundamentals

At the heart of Macquarie's solution is Crossplane, which provides several key abstractions:

1. Managed Resources (MRs): These are CRDs that map one-to-one with a specific object in a cloud provider's API (e.g., an AWS S3 bucket). Crossplane providers expose these CRDs, and a dedicated controller continuously reconciles the desired state defined in the Kubernetes API server with the actual state in the cloud provider. Every aspect of the cloud provider's API is exposed in the corresponding CRD.
2. Compositions (XRDs - Composite Resource Definitions): For building complex infrastructure, Crossplane allows combining multiple managed resources into a higher-level, opinionated API. An XRD defines the API presented to end-users (analogous to a restaurant menu item), while the Composition defines the underlying "recipe" – the set of managed resources and their configurations – that fulfills that API request. This enables platform teams to curate secure, compliant, and standardized infrastructure patterns for application developers.
3. Functions: To configure complex compositions, Crossplane leverages functions. These are serverless-like components that can be written in any programming language (e.g., KCL, Go templating, Go, Python, C# SDK) to generate the desired state. Crossplane passes all relevant data into the function, which then outputs the configured resources. This provides immense flexibility for dynamic and conditional resource generation.

Crossplane's architecture emphasizes specialized, single ownership components: functions define desired state, providers interact with APIs, compositions combine resources, and XRDs define Kubernetes-native APIs. Its continuous reconciliation, CRD-based everything (allowing integration with tools like Kyverno or CEL), and robust operational features like metrics emission and workload identity (leveraging OIDC trust for pods to prevent static secrets) were critical factors in its selection.

Macquarie's Control Plane Architecture

Macquarie's deployed control plane is built on an Amazon EKS Kubernetes cluster. The key components are:

• Infrastructure as Code (IaaS) Tool: Crossplane for managing cloud resources.
• Continuous Integration (CI): Argo Workflows for orchestrating automation tasks.
• Continuous Delivery (CD): Argo CD for GitOps-driven deployment and continuous synchronization.
• Security Constraints/Policy Enforcement: Gatekeeper (an OPA-based admission controller) for enforcing policy-as-code.
• Observability Stack: Grafana and Loki for logging and monitoring.
• Custom Provider: A specific provider developed to integrate with Macquarie's existing AWS pipeline.
• Custom APIs: Several Golang and Python APIs to interface with internal Macquarie systems like their developer portal and change management tools.

Key Design Decisions

1. IaaS Tool Selection: Macquarie evaluated existing Terraform usage and KCC (Kubernetes Config Connector) for GCP. Crossplane was chosen because it offered a Kubernetes-native resource model (KRM), easy integration with their existing infrastructure, built-in continuous reconciliation, and crucial multicloud coverage from a single tool, ticking boxes that the others didn't.
2. CI/CD Tool Selection: Argo CD and Argo Workflows were selected due to their CNCF backing, strong community support, familiarity within the Kubernetes ecosystem (especially with Crossplane documentation), and flexibility for integrating with various tools and the "bot of bots" workflow.
3. Greenfield vs. Brownfield: This was the most significant design decision. A greenfield approach (isolated, new applications only) would have been faster to deliver but limited in impact. Macquarie opted for a brownfield approach, integrating the new control plane with their existing AWS foundational pipeline. This allowed new services deployed via Crossplane to seamlessly interact with existing applications and resources managed by the legacy pipeline, maximizing leverage across the organization.

Brownfield Integration Services

To achieve seamless brownfield integration with their existing AWS foundational pipeline (which manages VPCs, subnets, account bootstrapping, IAM, and existing service enablement), Macquarie developed three custom Golang integration services:

1. Identity API: This API retrieves resource identifiers for infrastructure managed by both the existing pipeline and the new control plane, enabling cross-system resource awareness.
2. Access Management API: This service defines and manages policy requirements for new resources, ensuring they are aware of and compliant with the policy requirements of existing resources.
3. Compliance API: This critical API provides feedback on the compliance status of the target cloud environment. It acts as a gate, preventing the deployment of new changes if the environment is non-compliant, unless an explicit exemption process is followed.

These integration services ensure that the new Crossplane-managed resources operate within the established, regulated framework of Macquarie's existing cloud footprint, creating a single, cohesive environment for application teams.

Demo / Proof of Concept

▶ Watch: Understanding Crossplane's Managed Resources concept (5:20)

A central demonstration of Macquarie's modernized control plane is their innovative "bot of bots" workflow, which significantly enhances the developer experience and shifts security and compliance feedback left.

This workflow is triggered by a pull request (PR) containing a claim (the declarative deployment specification for infrastructure) or an update to an existing PR in a Git repository. A webhook then calls an Argo Workflow, which orchestrates a series of ephemeral Argo bots. These bots are short-lived pods that perform specific checks and terminate upon completion, conserving cluster resources.

The workflow proceeds as follows:

1. Policy as Code Bot: This bot is the first to run. It checks the proposed claim against Macquarie's defined security requirements and policies. This leverages tools like Gatekeeper to ensure that the configuration specified in the claim adheres to all security best practices and internal standards before any resources are provisioned.
2. Compliance Bot: Following the policy check, the compliance bot assesses the target environment where the claim is intended to be deployed. It ensures that the environment itself meets all governance rules and compliance standards, providing a crucial check on the operational context.
3. Hydrator Bot: This bot focuses on developer experience. It automatically "hydrates" the claim specification with additional, often complex, application-specific identifiers and details that developers shouldn't have to manually input. Examples include VPC IDs, DNS-specific information, and Route 53 IDs. This allows developers to write much simpler and cleaner claim specifications.

The output of these bots is directly posted as comments on the pull request in the Git repository. If all bots are successful, the PR comment indicates this. If any bot fails, the comment clearly states which bot failed and the specific issue, allowing the developer to quickly identify and rectify the problem, raise a new PR, and re-trigger the workflow.

Once all bots are successful and the reviewer approves the PR, the commit is signed using Git's commit signing feature. This signed commit is then passed to Argo CD, which in turn passes it to Crossplane for the actual deployment. The commit signing acts as an extra security gate, ensuring that the "bot of bots" checks have been successfully run and not bypassed.

This workflow has received overwhelmingly positive feedback from developers. Previously, they had to wait for deployments to hit the cloud platform to discover security or configuration issues. Now, feedback is provided within seconds, directly in their familiar Git environment, drastically accelerating the development cycle and reducing frustration.

Defensive Implications

▶ Watch: Crossplane Composition: combining resources for complex infrastructure (6:18)

The modernization journey undertaken by Macquarie Group offers several critical defensive implications for organizations operating in regulated environments:

1. Shift-Left Security and Compliance: The "bot of bots" workflow is a prime example of shift-left security. By integrating automated policy-as-code and compliance checks directly into the pull request review process, security and governance issues are identified and remediated in seconds, long before any infrastructure is provisioned in the cloud. This proactive approach significantly reduces the risk of deploying non-compliant or insecure resources.
2. Automated Guardrails and Service Enablement: The control plane's design emphasizes automated security boundaries and guardrails for new cloud services. This means that even experimental or newly adopted services are automatically provisioned with the necessary security controls, minimizing the attack surface and ensuring regulatory adherence from day one.
3. Continuous Reconciliation for Drift Detection: Crossplane's inherent continuous reconciliation mechanism is a powerful defensive tool. It constantly monitors the actual state of deployed infrastructure against the desired state defined in Git. Any unauthorized changes or configuration drift in the cloud environment are automatically detected and corrected, preventing malicious tampering or accidental misconfigurations from persisting.
4. Standardized, Policy-Enforced APIs: By exposing infrastructure as Kubernetes-native CRDs, Macquarie can leverage existing Kubernetes security tools like Gatekeeper (for policy enforcement) and potentially Kyverno or CEL for more granular access control and validation. This standardizes how security policies are applied across diverse cloud resources, making them easier to manage and audit.
5. Enhanced Supply Chain Security with Git Commit Signing: The requirement for Git commit signing before deployment via Argo CD adds an important layer of security to the infrastructure supply chain. It verifies the authenticity and integrity of the code being deployed, ensuring that changes have been reviewed, passed automated checks, and originated from a trusted source, preventing unauthorized or unvetted code from reaching production.
6. Reduced Manual Toil and Human Error: Automating infrastructure provisioning and compliance checks drastically reduces the need for manual intervention, thereby minimizing the potential for human error. This frees up platform engineers to focus on higher-value security enhancements and architectural improvements rather than repetitive, error-prone tasks.
7. Workload Identity and Secret Management: Crossplane's support for workload identity (leveraging OIDC trust for pods) for its providers reduces the reliance on long-lived, static secrets. This significantly lowers the risk associated with secret exposure and simplifies credential management, a common security vulnerability.

Key Takeaways

• Embrace Universal Control Planes for Multicloud: Crossplane, built on Kubernetes, provides a powerful and unified declarative API for managing infrastructure across multiple cloud providers, simplifying operations and accelerating service enablement.
• Prioritize Developer Experience with Shift-Left Security: Implementing automated pre-deployment checks, such as Macquarie's "bot of bots" workflow, directly within the Git pull request process drastically improves developer velocity by providing immediate feedback on security and compliance.
• Strategic Brownfield Integration is Crucial for Enterprises: For organizations with existing cloud footprints, designing a control plane that seamlessly integrates with legacy systems through custom APIs (Identity, Access Management, Compliance) is essential for broad adoption and impact.
• Leverage Cloud-Native and Community-Driven Tools: Building on CNCF projects like Kubernetes, Crossplane, and Argo, along with tools like Gatekeeper, provides robust, maintainable, and secure foundations, reducing custom development and easing vulnerability management.
• Automate Everything for Regulation and Efficiency: Full automation, from infrastructure provisioning to security and compliance checks, is key to meeting stringent regulatory requirements at speed, reducing manual toil, and ensuring continuous reconciliation of desired states.
• GitOps as the Operational Backbone: Using Git as the single source of truth for infrastructure definitions, combined with tools like Argo CD and Git commit signing, enforces robust change management, auditability, and security in a regulated environment.

About the Speaker(s)

Pranita Praveen is a Director of Engineering at Macquarie Group. With over 10 years of experience in platform engineering and cloud platforms, she brings extensive expertise in building and evolving complex cloud infrastructure in large, regulated environments. Her background is centered on enabling developers and modernizing technological stacks to meet business demands.

Steven Borrelli is a Principal Solutions Architect at Upbound, the company behind Crossplane. He is also an active member of the Crossplane community, contributing to its development and advocating for its adoption. His insights into Crossplane's architecture and capabilities are foundational to understanding its application in modern cloud control planes.