Breaking Free From the Cloud: Banking on Self... Kārlis Akots Gribulis & Per Hedegaard Christiansen

Breaking Free From the Cloud: Banking on Self-Hosted Kubernetes

Speakers: Kārlis Akots Gribulis, Engineer; Per Hedegaard Christiansen, Head of Container Platform Engineering, Saxo Bank

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=Zfp94fOMcwE

Overview

In this insightful KubeCon EU presentation, Kārlis Akots Gribulis and Per Hedegaard Christiansen from Saxo Bank shared their ambitious journey of migrating critical components from a public cloud environment to a self-hosted Kubernetes platform. The talk, titled "Breaking Free From the Cloud: Banking on Self-Hosted Kubernetes," delved into the motivations, technical challenges, and significant benefits experienced by a regulated financial institution in making such a substantial architectural shift.

The core of Saxo Bank's decision was multifaceted, driven by a combination of stringent compliance requirements, perceived reliability issues with public cloud offerings, and, most significantly, a substantial opportunity for cost reduction. By meticulously detailing their experiences, including the evolution of their infrastructure choices and the custom solutions developed, the speakers provided a compelling case study for large enterprises contemplating similar moves. This article explores the intricate details of Saxo Bank's migration, offering valuable lessons for organizations seeking greater control, enhanced security, and improved cost efficiency in their cloud-native operations.

The presentation underscored that while public cloud offers undeniable advantages, particularly for smaller entities or those requiring vast global distribution, large, established organizations with existing data centers and internal expertise can achieve profound benefits by strategically embracing self-hosted solutions. Saxo Bank's story is a testament to the power of internal competence and a well-executed strategy in navigating the complexities of modern cloud infrastructure.

Background

▶ Watch: Speakers and Saxo Bank introduction (0:00)

Saxo Bank, an online trading platform founded in 1992, manages over 100 billion USD in client assets for 1.2 million customers across a wide array of financial instruments. As a "too big to fail" financial institution, compliance, reliability, and security are paramount. While the core banking systems predominantly run on traditional Windows servers, a significant portion of critical components, essential for client-facing operations and internal processes, had been running on Kubernetes in a public cloud environment for four years.

The initial motivation for Saxo Bank to embark on this self-hosting journey stemmed from several key factors. Firstly, as a regulated entity, they are mandated to have a robust exit plan from any public cloud provider, ensuring business continuity and data sovereignty. Secondly, after four years of operating in the public cloud, they began to experience notable reliability issues and encountered certain limitations that hindered their operational efficiency. However, the most significant impetus, and the primary driver behind the project, was the escalating cost associated with public cloud services. This economic pressure catalyzed the strategic decision to explore an on-premises Kubernetes solution.

The idea for this ambitious undertaking was initially inspired by a keynote presentation at KubeCon Valencia 2022, which showcased the management of over 1,000 Kubernetes clusters using Cluster API (CAPI). Having already accumulated four years of experience running Kubernetes in the public cloud and possessing a strong internal engineering team, Saxo Bank felt confident in their capabilities to manage and operate such a complex infrastructure themselves. This existing competence and the prohibitive public cloud costs laid the groundwork for their transition to a self-hosted, cloud-agnostic Kubernetes platform.

Key Findings

▶ Watch: Key motivations for self-hosted Kubernetes (2:30)

Saxo Bank's migration to self-hosted Kubernetes yielded several transformative findings and benefits, directly addressing their initial motivations:

• Massive Cost Savings: The most significant finding was an approximate 85% cost saving on compute and storage by moving workloads on-premises. This calculation factored in all data center operational costs, including manpower, network, cooling, and power, amortized over a five-year period. To illustrate, a single server costing €20,000, written off over five years, amounts to €333 per month, providing 32 physical cores and 1.5 TB of RAM—a capacity far more cost-effective than comparable public cloud VMs.
• Accelerated Cluster Creation: The speed of creating new Kubernetes clusters dramatically improved, increasing by roughly 15 to 20 times on-premise. In contrast, basic cluster deployments in their previous managed cloud environment could take up to 30 minutes, even without applications. This was particularly problematic for numerous short-lived test clusters that needed daily rebuilding. The public cloud's "stop and start" feature, intended to save costs and speed up restarts, proved "reliably unreliable," often leading to state mismatches, ghost pods (pods marked as running on non-existent nodes), and policies blocking themselves due to missing workloads.
• Enhanced Security Benchmarks and Transparency: Moving on-premise provided full control over the Kubernetes control plane, enabling Saxo Bank to almost double their CIS benchmark score in just two to three weeks. In the public cloud, many control plane settings were abstracted or immutable, preventing them from addressing reported security issues. On-prem, they could implement critical security measures like etcd encryption (which was not enabled by default in their cloud provider) and gain granular control over service account certificates, significantly improving their security posture and compliance.
• Increased Reliability and Resilience: By owning the infrastructure, Saxo Bank could develop custom solutions to address specific reliability issues. For instance, they created a cron job called "ghost pod destroyer" and implemented a node termination handler to combat the ghost pod problem and ensure proper cleanup of resources.
• Strategic Flexibility and Growth: The decision to separate compute (hosts) from storage (SAN) rather than adopting a hyper-converged infrastructure (HCI) provided crucial flexibility. This allowed them to scale compute and storage independently, adding more hard drives to the SAN or more host machines as needed, without being constrained by fixed ratios.
• Internal Competence Validation: Having successfully operated Kubernetes in the public cloud for four years, Saxo Bank's engineering team had already developed significant expertise, often resolving critical issues internally rather than relying on the cloud provider. This self-sufficiency provided the confidence that they possessed the necessary competence to manage a fully self-hosted environment.

Technical Deep Dive

▶ Watch: Revealing 85% cost savings on-prem (3:20)

Saxo Bank's transition to self-hosted Kubernetes involved a meticulous re-architecture and selection of technologies to meet their specific requirements for control, scalability, and security.

At the heart of their new infrastructure is Cluster API (CAPI), used for declarative management and automation of Kubernetes cluster lifecycle operations. This allows them to deploy Kubernetes on raw VMware virtual machines, a choice acknowledged as potentially controversial but well-suited to their existing organizational infrastructure and expertise. The platform leverages GitOps principles extensively, meaning all infrastructure and application configurations are stored in Git repositories, and changes are applied automatically to the clusters.

For storage, Saxo Bank initially explored MinIO for S3-compatible object storage. However, they encountered significant challenges with its AGPLv3 license, which presented liability concerns for an enterprise environment due to its strong copyleft nature. The enterprise version of MinIO was deemed too costly. Consequently, they transitioned to Rook Ceph, an open-source storage orchestrator that provides S3 compatibility and integrates well with Kubernetes for persistent storage. To manage Rook, they developed a custom Crossplane provider from scratch. This was necessary because an existing Ceph provider for Crossplane did not support private registries, a critical requirement for their secure environment. Previously, they had also developed a custom Crossplane provider for MinIO (forking and extending a slightly dead community provider) to manage users, buckets, and policies.

Identity management presented another critical area for re-engineering. In the public cloud, they utilized Workload Identity with Managed Identity, where one managed identity was shared across all clusters within an environment. This setup had a significant limitation: a maximum of 20 attached federated credentials. This cap proved problematic for their dynamic test environments, which frequently spun up more than 20 short-lived clusters, leading to contention for identity slots. By moving on-premise, Saxo Bank gained direct control over service account certificates. This allowed them to generate and reuse the same certificate across all dev and test clusters, effectively eliminating the 20-credential limit and simplifying identity management at scale.

Secret management also saw a significant shift. While HashiCorp Vault was used in the public cloud, its integration into a pure GitOps workflow proved challenging due to the need for manual sealing and unsealing operations. Although tools like Bank-Vaults existed to automate this, they were found to be slightly outdated and not keeping pace with HashiCorp's cadence. Furthermore, cost reduction was a factor. For general application secrets, Saxo Bank replaced Vault with Sealed Secrets. This solution encrypts secrets into a Kubernetes Secret that can be safely stored in Git, then decrypted only by the controller running in the target cluster. They acknowledged that Vault still has essential use cases for key management and automatically generated secrets that developers should never directly access.

The platform architecture includes Harbor for container image registry, CoreDNS for internal DNS resolution, and STEX for single sign-on. For high availability in production, the control plane is spread across three data centers, with application workloads running in two of them. This multi-data center approach introduced networking challenges, particularly concerning routing external traffic, handling non-shared subnets, and managing floating IPs, which is a common feature in public clouds but requires custom solutions on-prem.

Cluster management with Cluster API starts from a temporary kind cluster, which creates a "registry cluster" (dubbed the "grandmother cluster"). This registry cluster then becomes self-managed and is responsible for provisioning and managing all subsequent workload clusters. The speakers noted that this self-managed management cluster setup is a potential single point of failure and is still under monitoring, acknowledging the "chicken-egg problem" of managing the manager.

Finally, to address the reliability issues experienced in the public cloud, Saxo Bank developed custom tooling. This included a cron job to identify and delete "ghost pods" (pods reported as running by the API server but on non-existent nodes) and a node termination handler to prevent such state mismatches during node lifecycle events. For their dev/test environments, which are hosted in a single data center but use Helm charts with topology spread constraints for zones, they implemented a workaround by faking zone labels on nodes to ensure deployments functioned correctly.

Demo / Proof of Concept

▶ Watch: Public cloud stop/start feature unreliability (5:30)

While the talk didn't feature a live, interactive demo in the traditional sense, the speakers provided compelling demonstrations of their new platform's capabilities and highlighted how their architectural choices manifested in tangible benefits.

One key demonstration involved a video showcasing the cluster creation speed in their on-premises setup. This visual evidence underscored the dramatic improvement—a 15 to 20 times increase—compared to their previous public cloud environment. The swift provisioning of a basic Kubernetes cluster on-prem, in stark contrast to the 30-minute public cloud equivalent, served as a powerful proof of concept for the efficiency gains achieved through direct infrastructure control.

Another illustrative "demo" occurred somewhat serendipitously during an internal presentation by Per Hedegaard Christiansen. He intended to showcase a workload migration to the on-premises environment but inadvertently forgot to adjust the application's configuration settings from pointing to public cloud resources to the new on-prem endpoints. To his surprise, the demo still worked flawlessly. This unexpected success highlighted the effectiveness of their GitOps-driven configuration management. By using config maps within each namespace to store environment variables (such as ingress URLs or Kafka locations), developers were largely abstracted from the underlying infrastructure. This design ensured that applications could seamlessly transition between environments with minimal, if any, code changes, validating their strategy of making the migration as smooth as possible for development teams.

These demonstrations, though not formal live coding sessions, effectively conveyed the core message: Saxo Bank's self-hosted Kubernetes platform delivers on its promises of speed, reliability, and developer experience, demonstrating that the technical deep dive translated into practical, operational advantages.

Defensive Implications

▶ Watch: Ghost pods issue and custom solutions (6:10)

Saxo Bank's migration to self-hosted Kubernetes carries significant defensive implications, primarily stemming from the increased control and transparency gained over their infrastructure:

• Enhanced Security Posture through Direct Control: By owning the entire Kubernetes stack, Saxo Bank gained the ability to directly implement and enforce security best practices at every layer. This was vividly demonstrated by their ability to nearly double their CIS benchmark score in a matter of weeks. In the public cloud, many control plane configurations (e.g., etcd settings, API server flags) are immutable or abstracted, preventing organizations from fully hardening their clusters. On-prem, Saxo Bank could explicitly enable critical features like etcd encryption, which was not a default in their previous cloud provider, significantly reducing the risk of data exposure at rest within the cluster's key-value store.
• Improved Identity Management and Reduced Attack Surface: The shift from public cloud Workload Identity with its 20-credential limit to direct control over service account certificates offers a more robust and scalable identity management solution. This eliminates the need for complex workarounds or fighting for limited slots, leading to a more streamlined and less error-prone configuration for workload identities. By centralizing certificate generation and reuse for dev/test environments, they reduce the complexity and potential misconfigurations that could lead to unauthorized access.
• Granular Secret Management with Sealed Secrets: The adoption of Sealed Secrets for general application secrets improves the security of secrets in transit and at rest within Git repositories. Developers can commit encrypted secrets to version control without exposing sensitive information, aligning with GitOps principles. While HashiCorp Vault remains crucial for key management and highly sensitive, auto-generated secrets, offloading less critical secrets to Sealed Secrets reduces the operational overhead and potential attack surface associated with a full Vault deployment for every application.
• Proactive Reliability and Resilience Measures: The development of custom solutions like the "ghost pod destroyer" and the node termination handler directly addresses reliability issues that could otherwise lead to service disruptions or security vulnerabilities. Ghost pods, for instance, could represent stale entries that might be exploited or simply cause operational headaches. By actively managing node and pod lifecycle events, Saxo Bank enhances the overall resilience and predictability of their environment, reducing the window for potential exploitation due to inconsistent states.
• Supply Chain Control and Compliance: Operating on-premises provides greater control over the software supply chain. Saxo Bank's decision to switch from MinIO (AGPLv3 license concerns) to Rook Ceph (for S3 compatibility) demonstrates a proactive approach to managing legal and security risks associated with open-source licenses. This level of control is crucial for financial institutions subject to stringent compliance and regulatory requirements, including the need for robust exit strategies from external providers.
• Transparency and Auditability: The increased visibility into the underlying infrastructure components allows for more thorough auditing and monitoring. Security teams can directly inspect configurations, logs, and network flows, which can be more challenging in highly abstracted public cloud environments. This transparency is invaluable for demonstrating compliance and responding to security incidents effectively.

In essence, Saxo Bank's move is a strategic defensive play, trading the convenience of managed cloud services for the profound security benefits of complete ownership and granular control, tailored precisely to the demanding requirements of a regulated financial entity.

Key Takeaways

• Significant Cost Reduction: Large enterprises with existing data centers and internal expertise can achieve substantial cost savings (Saxo Bank reported 85% on compute and storage) by moving critical workloads from public cloud to self-hosted Kubernetes.
• Unparalleled Control and Security: On-premises Kubernetes offers granular control over the control plane, enabling organizations to enforce stringent security benchmarks (Saxo Bank nearly doubled their CIS score) and implement custom hardening measures like etcd encryption and precise service account certificate management.
• Strategic Component Selection is Crucial: Careful evaluation of open-source licenses (e.g., MinIO's AGPLv3) and the operational fit of tools (e.g., HashiCorp Vault vs. Sealed Secrets for different secret management needs) is vital for long-term sustainability and compliance.
• Robust Automation and Custom Solutions are Essential: Managing Kubernetes at scale on-prem requires heavy reliance on automation tools like Cluster API and GitOps, alongside developing custom solutions for unique challenges such as "ghost pods," unreliable cloud features, and multi-data center networking complexities.
• Context-Dependent Decision: The decision to "break free from the cloud" is not universal. It is best suited for organizations with significant existing infrastructure, a mature internal engineering team, and specific drivers like high public cloud costs, strict compliance requirements, or unique reliability demands.

About the Speaker(s)

Kārlis Akots Gribulis is an engineer at Saxo Bank, having joined the company last year. Prior to his role at Saxo Bank, Kārlis gained valuable experience building Kubernetes clusters for other prominent companies such as VUX and Accenture. He is a highly certified Kubernetes professional, holding several certifications (CKS, CKA, CKAD) and is actively pursuing further "golden cubesornot" certifications, demonstrating his deep expertise in the cloud-native ecosystem. Originally from Riga, Latvia, Kārlis currently resides in Denmark.

Per Hedegaard Christiansen serves as the Head of Container Platform Engineering at Saxo Bank, a position he has held for approximately six years. His extensive experience in container technologies and platform management has been instrumental in leading Saxo Bank's ambitious journey to self-hosted Kubernetes. Beyond his corporate role, Per is a prominent figure in the Danish cloud-native community, acting as the chairman of Cloud Native Denmark (formerly KCD Denmark), an organization that hosts conferences and donates profits to charity to foster young talent in IT. He resides in Copenhagen, Denmark, and has a passion for global travel.