Building a Platform Framework: Lessons Learned From Developing a Multi-Cl... Cat Morris & Jake Klein

Building a Platform Framework: Lessons Learned From Developing a Multi-Cl... Cat Morris & Jake Klein

Speakers: Cat Morris, Product Manager, Centasto; Jake Klein, Engineer, Centasto

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=AHY4IDlBhzE

Overview

In their KubeCon EU talk, "Building a Platform Framework: Lessons Learned From Developing a Multi-Cluster Kubernetes Operator," Cat Morris and Jake Klein from Centasto delve into the complex journey of developing Kredics, an open-source platform framework designed to simplify multi-cluster Kubernetes management. The speakers, drawing from their extensive experience in platform engineering and product development, articulate the escalating challenges faced by organizations as their infrastructure footprints grow from single Kubernetes clusters to sprawling, multi-cloud, multi-region, and even on-premise environments. They highlight how this increasing complexity often leads to operational overhead, inconsistent deployments, and a bottleneck for application development teams.

The core of their presentation focuses on the practical lessons learned while building Kredics, offering a blueprint for other organizations navigating similar waters. Morris and Klein emphasize the importance of leveraging the existing rich Kubernetes ecosystem while strategically innovating on top of it. They introduce the concept of "Promises" as a powerful abstraction layer, enabling platform teams to deliver "anything as a service" to their developers, thereby decoupling application teams from the underlying infrastructure intricacies. Their insights are particularly valuable for platform engineers, DevOps practitioners, and architects grappling with the realities of scaling Kubernetes deployments and striving for a more efficient, self-service developer experience.

Ultimately, the talk serves as a candid reflection on the triumphs and pitfalls encountered during Kredics' development, providing actionable advice for building robust, scalable, and user-centric platform frameworks. By sharing their "dos and don'ts," Morris and Klein aim to help others avoid common mistakes, reduce complexity, and prioritize effectively to deliver genuine value to their users in the ever-evolving landscape of cloud-native infrastructure.

Background

▶ Watch: Introduction: Building a platform framework with Kredics (0:00)

The journey into platform complexity, as vividly illustrated by Morris and Klein, often begins innocently enough. A single Kubernetes cluster suffices for initial development, easily managed with cloud provider UIs or basic tooling. However, as a product matures and a company scales, this simplicity quickly evaporates. The need for separate production and development environments necessitates multiple clusters, each with distinct requirements for scale and compute power. This initial multiplication soon escalates further with the introduction of availability zones for resilience, doubling the cluster count yet again.

The true inflection point arrives with business expansion: new regions for global reach and regulatory compliance, demanding entirely separate data environments. This pushes platform teams towards sophisticated cluster management tools like Terraform or open-source projects such as Cluster API (CAPI). The complexity compounds when diverse teams within the organization adopt different cloud providers – for instance, an ML team utilizing Azure credits while the main infrastructure resides on AWS. This introduces the challenge of managing disparate authentication models and subtle cloud-specific differences. The ultimate test of complexity comes with acquisitions, bringing in on-premise infrastructure, often with demands for complete control over hardware and data. This results in a highly heterogeneous environment encompassing multiple clouds, on-premise data centers, virtual machines, databases, and even edge compute, orchestrated by a mix of tools like Terraform, Puppet, and Ansible. This "crazy" landscape, as described by the speakers, is the problem Kredics was built to address.

Kredics emerged as an open-source project from Centasto, aiming to simplify the construction of robust platforms. A foundational principle for Kredics' design was "don't reinvent the wheel; use the ecosystem." The speakers highlighted the sheer volume of projects in the Kubernetes ecosystem—over 215 by their count—spanning observability, deployments, and various other domains. These tools, having solved existing problems and endured hard-won lessons, offer a rich foundation. However, the challenge lies in selecting the right tools and frameworks. Cat Morris referenced Evan Botcher's definition of a digital platform: "a foundation of self-service APIs, tools, services, knowledge and support which are arranged as a compelling internal product." This definition underscores the need for a holistic approach, focusing on what services to provide and how to deliver them effectively to internal users. The alternative, often leading to a "Gremlin after midnight" scenario, involves attempting to build every component oneself—custom GUIs, CI/CD systems, infrastructure as code—or relying on a single, all-encompassing product that inevitably fails to meet diverse needs. Kredics sought to avoid this pitfall by innovating on top of existing solutions, focusing its specialized efforts where they could add the most unique value.

Key Findings

▶ Watch: The ultimate problem: managing diverse infrastructure at scale (3:40)

The central contribution of Kredics, and a key finding from its development, is the concept of a Promise. This is a fundamental abstraction within the Kredics framework, designed to encapsulate "anything as a service" that a platform team wishes to provide to its users. A Promise is defined using standard YAML and Custom Resource Definitions (CRDs), making it instantly familiar to anyone working within the Kubernetes ecosystem. It comprises an API for user interaction, defined dependencies, workflows to orchestrate actions, and explicit rules governing where and how the service is deployed. This approach allows Kredics to integrate seamlessly with the broader cloud-native landscape.

Kredics' unique value proposition lies in its specialization in platform orchestration. Rather than attempting to replicate the functionality of cloud providers or core Kubernetes, Kredics focuses its energy on stitching together lower-level infrastructure components and presenting them as cohesive, self-service offerings to end-users. This strategic innovation on top of existing "wheels" allows Kredics to leverage powerful community projects like Backstage, Port, and Headlamp for user interfaces, rely on Kubernetes for its robust orchestration capabilities, and utilize the unparalleled infrastructure services of AWS, Azure, and GCP.

However, a significant finding from their experience was the profound "don't underestimate the complexity of going multicluster." What initially appears straightforward quickly escalates into a labyrinth of challenges related to connectivity, security, consistency, and scheduling across diverse environments. Kredics' architectural evolution was driven by the necessity to address these complexities head-on, particularly through its innovative adoption of a generalized GitOps model for multi-directional communication and scheduling, and the strategic use of cluster labeling to enable intelligent workload placement. This approach allowed them to manage distributed state without requiring direct, persistent connectivity to every target cluster, proving crucial for supporting edge and air-gapped environments.

Technical Deep Dive

▶ Watch: Lesson 1: Don't reinvent the wheel, use the ecosystem (4:15)

The technical journey of Kredics began by observing a common pattern: developers gaining direct access to Kubernetes clusters, deploying operators like Kubeflow, and deriving value. While initially effective, this decentralized approach quickly led to multi-tenant chaos. Different teams introduced conflicting requirements, versioning issues, and accidental resource deletions. Kredics' core design philosophy emerged from this challenge: abstracting direct Kubernetes access from developers. Instead, developers interact with a higher-level platform interface, while the platform—powered by Kredics—manages the underlying Kubernetes complexity.

A platform engineer, using Kredics, defines a service (e.g., "Kubeflow as a service") as a Promise. This Promise captures the company-specific configuration and deployment logic for Kubeflow. Once installed on the Kredics platform, the Promise is responsible for scheduling Kubeflow to all target Kubernetes clusters where it needs to run. This scheduling is not arbitrary; it adheres to specific requirements defined within the Promise, such as a particular Kubernetes version, the presence of GPU nodes, or specific regional deployments. The goal is to allow users to request a service (e.g., "train this model for me"), and the platform intelligently determines the optimal cluster based on these requirements, abstracting the cluster selection entirely.

The initial design considered giving the Kredics platform direct API access to all managed clusters, similar to existing tools like Komodor or Argo CD. However, this approach presented significant drawbacks, particularly for air-gapped environments, edge compute, or clusters across different cloud providers with unique authentication models. Maintaining a central repository of credentials for every cluster and ensuring persistent connectivity was deemed unrealistic and complex.

This led to the adoption of a GitOps-first approach. Instead of direct API calls, Kredics orchestrates the writing of desired state configurations—Kubernetes manifests, Terraform files, etc.—to designated Git repositories or S3 buckets. Remote clusters then utilize their own GitOps tools, such as Argo CD or Flux, or even custom loops, to pull these files and converge their state. This model offers immense flexibility: it decouples the central platform from direct cluster connectivity, supports disconnected operations (even a USB stick transfer for air-gapped scenarios), and leverages the inherent benefits of Git for version control, auditability, and collaboration.

To enable intelligent scheduling, Kredics introduced a system of cluster labeling, drawing inspiration from how nodes are labeled within a single Kubernetes cluster. Each managed cluster is assigned labels describing its attributes—e.g., region: eu-west-1, gpu-enabled: true, k8s-version: 1.27, or operator: cubeflow-installed. When a Promise needs to deploy a service, it specifies the required cluster labels. Kredics then filters the available clusters, identifies those matching the labels, and writes the necessary configuration files to their respective Git repositories. For example, a "Kubeflow as a service" Promise might target clusters with gpu-enabled: true, ensuring Kubeflow is only deployed where GPU resources are available.

A significant evolution in Kredics' design was the shift from scheduling "one document at a time" to multi-document scheduling. Early designs struggled with complex workloads requiring different components to be deployed to different clusters. For instance, an ML model might need to run on an expensive GPU cluster, while its associated dashboard should reside on a cheaper, general-purpose cluster. Kredics adapted to support scheduling multiple documents to multiple distinct destinations concurrently. This enables intricate scenarios like deploying an agent to one cluster and another agent to a separate cluster for a multi-cluster database setup, or even sending an ML model to a cluster, a dashboard to another, and an S3 bucket request to a Terraform Git repository. This generalized GitOps approach extends beyond Kubernetes, treating any Git-converged system (Terraform, Ansible Tower, Backstage) as a potential target.

Finally, Kredics also addressed the challenge of getting information back from the clusters (e.g., health checks, status updates). Initial ideas involved deploying agents to push data back to the platform or using remote databases. However, these solutions would reintroduce the very connectivity and architectural complexity the GitOps model sought to eliminate. The ingenious solution was to use the same GitOps mechanism for bidirectional communication. Clusters write their status or health information back to a designated Git repository, which the Kredics platform then reads. This "pull-based" status reporting maintains the decoupled architecture, significantly reducing complexity and accelerating development cycles, as evidenced by features being developed in "days rather than weeks and months." This iterative, feedback-driven approach to technical design, prioritizing simplicity and leveraging existing patterns, proved critical in Kredics' development.

Demo / Proof of Concept

▶ Watch: Defining a digital platform (Evanbotcher's definition) (4:45)

While the talk did not feature a live, step-by-step demonstration of Kredics in action, the speakers detailed the architectural evolution and design principles that underpin its functionality. The entire presentation can be considered a conceptual proof of concept, illustrating how Kredics addresses the challenges of multi-cluster management through its core components and methodologies. The discussion around Promises, GitOps-based scheduling, cluster labeling, and bidirectional communication via Git serves to explain the operational mechanics of the framework, demonstrating its capability to manage diverse infrastructure landscapes.

Defensive Implications

▶ Watch: Innovate on the wheel: the concept of 'promises' (6:00)

The design principles and architecture of Kredics offer several significant defensive implications for organizations managing complex Kubernetes environments:

• Reduced Attack Surface through Abstraction: By abstracting direct Kubernetes access from application developers and providing a higher-level platform interface, Kredics inherently reduces the potential attack surface. Developers interact with well-defined "Promises" rather than raw Kubernetes APIs, limiting their ability to misconfigure or inadvertently expose sensitive resources. This also centralizes policy enforcement at the platform level.
• Enhanced Security via GitOps: The adoption of a GitOps model is a powerful security control. Git serves as the single source of truth for the desired state of all infrastructure and applications. This provides:
• Auditability: Every change is recorded in Git, creating an immutable audit trail of who changed what, when, and why.
• Version Control: Easy rollback to previous stable states in case of misconfigurations or security incidents.
• Immutability: Configuration drift is minimized as clusters continuously converge to the state declared in Git. Unauthorized manual changes are detected and reverted.
• Code Review: All infrastructure changes can undergo rigorous code review processes before being merged and deployed, acting as a critical gate for security vulnerabilities.
• Decoupled and Resilient Architecture: Eliminating direct API connectivity between the central Kredics platform and remote clusters enhances security by reducing the blast radius. If the central platform were compromised, it would not automatically grant an attacker direct access to all managed clusters. Similarly, a compromise of an edge or air-gapped cluster would not necessarily expose the central platform or other clusters in the fleet. This design improves overall system resilience and isolation.
• Intelligent and Policy-Driven Deployment: The use of cluster labeling ensures that workloads are deployed only to environments that meet specific criteria. This can be leveraged for security purposes, ensuring sensitive workloads only run on clusters with appropriate compliance certifications, specific security configurations (e.g., hardened OS, specific network policies), or in designated secure regions. This prevents accidental deployment to less secure or non-compliant environments.
• Standardized and Secure Service Delivery: "Promises" enforce company-wide best practices for deploying services. Security configurations, such as network policies, resource quotas, secret management integrations, and container security settings, can be pre-baked into these Promises. This standardizes secure deployments, reducing the likelihood of individual teams making security oversights.
• Simplified Operational Security: By reducing the overall complexity of managing a multi-cluster, multi-cloud environment, Kredics indirectly improves operational security. Simpler systems are easier to understand, audit, monitor, and troubleshoot. This reduces the cognitive load on security and operations teams, allowing them to focus on higher-value tasks and respond more effectively to incidents.
• Bidirectional Status for Security Monitoring: Using the GitOps mechanism for clusters to report status back to the platform enables continuous security monitoring. Clusters can commit logs of security events, compliance status, or detected anomalies to Git, which the central platform can then ingest and analyze. This provides a decentralized yet unified view of the security posture across the entire fleet.

Key Takeaways

• Embrace the Ecosystem, but Innovate Strategically: The Kubernetes ecosystem is vast and powerful (over 215 projects mentioned). Leverage existing tools and projects to avoid reinventing the wheel, but identify your unique value proposition and innovate on top of established solutions to create a specialized product or platform.
• Never Underestimate Multicluster Complexity: Managing multiple Kubernetes clusters across different regions, clouds, and on-premise environments introduces significant challenges in connectivity, state management, security, and scheduling. Plan for this complexity from the outset rather than hoping it will remain simple.
• GitOps as a Foundational Paradigm: GitOps is not just for Kubernetes deployments; it's a flexible and powerful model for orchestrating state across diverse systems (Terraform, Backstage, S3). It enables decoupled, auditable, and resilient operations, even for air-gapped or edge environments, and can facilitate bidirectional communication.
• Prioritize User Needs with Strategic Vision: Avoid over-engineering based on isolated feedback or immediate requests. Instead, prioritize user needs by aligning them with broader organizational goals and remembering the core problem your platform aims to solve. This ensures that development efforts deliver impactful features at the right time.
• Abstract Kubernetes Expertise: Do not assume your users possess deep Kubernetes knowledge. Design platform frameworks like Kredics to provide a simplified, unified interface that abstracts away the underlying complexity, making it easier for application teams to consume services without needing to be Kubernetes experts.
• Reduce Complexity in Your Control: While external factors (like multiple clouds) introduce complexity, focus on simplifying the aspects of the platform that are within your control. This includes choosing efficient communication patterns (e.g., GitOps for bidirectional communication) and avoiding unnecessary architectural overhead.

About the Speaker(s)

Cat Morris is a Product Manager at Centasto, where she has been working for approximately a year and a half to two years. Prior to her role at Centasto, Cat spent five to six years in the platform and Kubernetes domain, working on platform products at companies such as Thoughtworks. Her experience spans both the product management and technical aspects of building robust platform solutions.

Jake Klein is an Engineer at Centasto, having been with the company for nearly three years. Before joining Centasto, Jake was involved with WeWorks, contributing to their open-source projects. His background in engineering and open-source contributions provides a deep technical perspective on the challenges and solutions discussed in the talk.