Dancing With the Pods: Live Migration of a Database Fleet While Serving... Jayme Bird & Manish Gill

Dancing With the Pods: Live Migration of a Database Fleet While Serving... Jayme Bird & Manish Gill

Speakers: Jayme Bird, Software Engineer, ClickHouse; Manish Gill, Software Engineer, ClickHouse

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=ufY_JFPpzRI

Overview

This technical article delves into the intricate process of performing live, zero-downtime migrations for a large fleet of ClickHouse databases running on Kubernetes. Presented by Jayme Bird and Manish Gill from ClickHouse, the talk at KubeCon EU outlines the journey of transitioning thousands of production clusters from a traditional single-StatefulSet orchestration model to an advanced multi-StatefulSet architecture. This shift was necessitated by the need to implement "make before break" vertical autoscaling, a technique crucial for cloud-native elasticity that current Kubernetes StatefulSets do not natively support for stateful workloads.

The speakers meticulously detail the architectural evolution, the custom tooling, and the significant challenges encountered when moving a critical, distributed database system without service interruption. They highlight how a dedicated migration controller, coupled with a robust orchestration system like Temporal, enabled this complex undertaking. The talk provides invaluable insights into overcoming both Kubernetes-specific limitations and inherent database elasticity challenges, offering a blueprint for organizations managing large-scale stateful applications in dynamic cloud environments.

The importance of this work cannot be overstated for the cloud-native community. As more critical applications move to Kubernetes, the ability to perform fundamental infrastructure changes, such as scaling and re-orchestration, without impacting user experience becomes paramount. This presentation serves as a testament to the innovation required to push the boundaries of what's possible with stateful workloads on Kubernetes, moving beyond the "break first" paradigm to achieve true operational agility and resilience.

Background

▶ Watch: Talk introduction and agenda overview (0:00)

The foundation of the problem addressed by Jayme Bird and Manish Gill lies in the inherent limitations of Kubernetes StatefulSets when it comes to dynamic resizing and advanced orchestration for stateful applications like databases. ClickHouse, an open-source, column-oriented OLAP database designed for petabytes of data and rapid analytical queries, is a prime example of such a workload. In its cloud version, ClickHouse leverages modern architectural patterns, storing actual data on object storage (like S3) while using Persistent Volume Claims (PVCs) attached to pods for metadata.

Traditionally, autoscaling in Kubernetes, especially vertical scaling (resizing CPU/memory requests and limits), necessitates pod evictions. This "break first" approach means that to resize a pod, it must be restarted. For a database fleet, this translates into a rolling restart, where pods are updated one by one, respecting a Pod Disruption Budget (PDB) (e.g., maxUnavailable=1). While preventing full downtime, this process is inherently slow and disruptive. During the restart of a replica, the remaining replicas bear increased load, which is precisely when the system needs more capacity. This often forces operators to provision extra overhead to absorb these load spikes. Manish Gill referenced his previous KubeCon Paris talk, "Fantastic Ordinals," which delved deeper into the specific limitations of StatefulSets that prevent more agile scaling patterns.

The desired solution was a "make before break" strategy. Instead of restarting existing pods, new, larger-capacity pods are brought online before the old ones are decommissioned. This approach offers immediate capacity increase, avoids putting pressure on existing replicas, and significantly speeds up scaling operations. However, a single StatefulSet, due to its deterministic naming and ordinal management, cannot natively support this. To achieve "make before break," ClickHouse's operator was refactored to manage a multi-StatefulSet architecture, where each pod is managed by its own dedicated StatefulSet. This provides the granular control necessary for adding and removing individual replicas independently.

The core challenge then became a massive migration: thousands of existing production ClickHouse clusters were still running on the old, single-StatefulSet orchestration model. The goal was to migrate all these customers to the new multi-StatefulSet approach without any downtime, while continuously serving queries, and doing so with full visibility and the ability to roll back if issues arose. This required a sophisticated, controlled, and resilient migration process that mirrored the "make before break" philosophy at an orchestration level.

Key Findings

▶ Watch: Kubernetes autoscaling: The pod eviction problem (1:40)

The talk presents several critical findings and contributions essential for managing stateful applications in cloud-native environments:

1. "Make Before Break" for Stateful Workloads: The primary finding is the successful implementation of "make before break" vertical scaling for a distributed database (ClickHouse) on Kubernetes. This was achieved by moving from a single StatefulSet managing multiple pods to a multi-StatefulSet architecture where each pod is managed by its own StatefulSet. This fundamental shift provides the necessary control to add new capacity before removing old, disruptive for traditional rolling updates.

1. Zero-Downtime Live Migration at Scale: The speakers demonstrated that a live migration of thousands of production database clusters from one Kubernetes orchestration model to another is feasible without downtime. This was accomplished using a "make before break" style migration strategy at the orchestration layer, ensuring continuous service availability throughout the transition.

1. Importance of Dedicated Orchestration and Controllers: For complex, large-scale, and critical operations like fleet-wide migrations, a dedicated external orchestration system (like Temporal) combined with a specialized migration controller is indispensable. This decouples migration logic from the main application operator, provides durable execution, robust error handling, state management, and the ability to manage migrations in batches.

1. Database-Specific Cloud-Native Elasticity Challenges: The migration exposed several intrinsic limitations within ClickHouse (and by extension, other databases) regarding cloud-native elasticity. Issues such as external table validation on sync, materialized view source table checks, inefficient system sync replica lightweight with continuous inserts, and the loss of local system tables highlighted areas where databases need to evolve to fully embrace dynamic Kubernetes environments.

1. Solutions for Database Elasticity: For each ClickHouse-specific challenge, the team developed targeted solutions. These included distinguishing between primary table creation and secondary replica sync for validation, adding a FROM modifier to the system sync replica lightweight command, and introducing a new S3 plain new writable disk type for system tables, enabling zero-copy attach to preserve observability data across replica changes.

1. Zone Balancing is Crucial: Maintaining zone balance is a non-trivial problem during dynamic scaling and migration events. The talk underscored that relying solely on Kubernetes' default scheduling might lead to undesirable replica distributions, necessitating explicit zone pinning and tracking within the operator to ensure high availability.

Technical Deep Dive

▶ Watch: Why 'break-first' vertical scaling is slow (3:00)

The technical deep dive begins by contrasting the standard Kubernetes autoscaling approach with the desired "make before break" model. In break-first vertical scaling, a Vertical Pod Autoscaler (VPA) might recommend new resource limits. An autoscaler then triggers a pod eviction. A controller (StatefulSet, Deployment, or custom) resubmits the pod. A mutating webhook intercepts the pod spec, applies the new resource limits, and the pod restarts with the new configuration. For stateful applications, this typically involves a rolling restart, where maxUnavailable=1 in a PDB ensures one replica is restarted at a time. This process is slow, disruptive, and places temporary additional load on remaining replicas.

The "make before break" paradigm, in contrast, involves adding new, larger capacity pods before removing the old ones. This is faster, non-disruptive, and temporarily provides even more capacity than requested, ensuring a smooth transition. However, vanilla Kubernetes StatefulSets are not designed for this. Their rigid ordinal system (pod-0, pod-1) makes it difficult to add new pods without affecting existing ones or to seamlessly replace them.

To overcome this, ClickHouse adopted a multi-StatefulSet architecture: instead of one StatefulSet managing N pods, N individual StatefulSets are created, each managing a single ClickHouse pod. This gives the operator fine-grained control over each replica, enabling true "make before break" operations.

The migration itself was designed as a "make before break" operation at an orchestration level. Starting with N replicas managed by a single StatefulSet, the process immediately adds N new replicas, each managed by its own dedicated StatefulSet. This results in 2N active replicas temporarily. A crucial synchronization step then copies catalog information and performs database-internal operations between the old and new replicas. Once synchronization is complete, the original single StatefulSet and its pods are safely "broken" (removed). The cluster is then left with N replicas, each managed by its new individual StatefulSet, having achieved the transition without any downtime.

This complex migration flow required robust orchestration. The team utilized Temporal, a durable execution system, as a central orchestrator. Temporal workflows handled:

• Saving the cluster's initial state and resetting it post-migration.
• Detecting configuration drift and human intervention.
• Sending notifications (e.g., to Slack).
• Pausing and resuming migrations.
• Providing durable execution, fault tolerance, and retries out-of-the-box.

A parent Temporal workflow acts like a glorified cron job, identifying batches of eligible clusters for migration daily. For each cluster, it kicks off a child workflow that interacts with the ClickHouse cluster Custom Resource (CR).

A specialized custom migration controller was developed to execute the migration logic. This controller is distinct from the main ClickHouse operator to avoid polluting the operator's codebase with one-time migration logic. During migration, clusters are put into a partial maintenance mode, which disables autoscaling, idling, and backups (operations that could interfere with the migration) but crucially keeps inserts and selects active. The migration controller operates alongside the main operator, using a mutex to ensure only one controller reconciles the CR at a time. Interestingly, the migration controller embeds the reconciler of the main Kubernetes operator, calling its functions for common tasks like adding or removing replicas and syncing ClickHouse state, while handling the specific orchestration and migration-specific logic itself.

Several ClickHouse-specific challenges emerged during rollout:

• Zone Imbalance: During make before break, new replicas might schedule into specific availability zones (AZs). If old replicas from other AZs are then removed, the cluster could end up with all replicas in one or two AZs, violating the desired max skew of one for high availability. This also occurred during idling/unidling. The fix involved implementing zone pinning and explicit zone tracking within the operator.
• External Table Engine Validation: ClickHouse performs validation (e.g., checking if a PostgreSQL instance is reachable) when creating external table engines. During a replica sync, it would attempt this validation, failing if the external system was temporarily unavailable or no longer existed. The solution was to modify ClickHouse to treat these as secondary create queries during sync, relaxing validation.
• Materialized View Validation: Similar to external tables, materialized views validate the existence of their source table during creation. If a source table was deleted but the materialized view remained valid, syncing it to a new replica would fail. ClickHouse was made more cloud-native by adjusting this validation during sync.
• system sync replica lightweight with Continuous Inserts: The command system sync replica lightweight syncs part metadata. If new replicas (M1, M3) were actively receiving continuous inserts while M2 was syncing, the command could time out or never complete as it tried to sync from all active replicas. The fix was a new FROM modifier for the command, allowing it to sync specifically from the old replicas (S1-S3) that were being decommissioned.
• Distributed DDL Queue and DNS Issues: ClickHouse's distributed DDL queue records hostnames (e.g., S1) for ON CLUSTER queries. If a replica (S1) was dropped before a DDL query completed on it, other replicas later attempting to process that query from the queue would fail to resolve S1's hostname, leading to errors. The talk highlighted this as a significant challenge related to DNS resolution post-replica removal.
• Loss of System Tables: Replica-local system tables (e.g., query logs, metric logs) provide crucial observability. With "make before break," dropping old replicas meant losing this historical data. A stop-gap was manual SELECT INSERT operations. The permanent solution involved introducing a new S3 plain new writable disk type, which moves system table data and metadata to S3, allowing zero-copy attach from old to new replicas, preserving observability history.

Demo / Proof of Concept

▶ Watch: Implementing Multi-StatefulSet for flexibility (5:20)

While the talk did not feature a live demonstration or explicit proof-of-concept during the presentation, the entire discussion serves as a detailed account of a large-scale, real-world deployment and migration of a ClickHouse database fleet. The speakers presented architectural diagrams, workflow charts, and specific code-level challenges and solutions derived from successfully migrating thousands of production customer clusters. This comprehensive exposition implicitly acts as a proof of concept, illustrating the practical feasibility and the intricate details involved in achieving zero-downtime, make-before-break migrations for stateful workloads on Kubernetes.

Defensive Implications

▶ Watch: Make Before Break demonstrated with new pods (5:55)

For organizations operating stateful applications on Kubernetes, the insights from this talk offer several critical defensive implications:

1. Re-evaluate StatefulSet Usage for Dynamic Scaling: Defenders should acknowledge the limitations of vanilla Kubernetes StatefulSets for achieving true "make before break" elasticity. For applications requiring frequent, non-disruptive vertical scaling, exploring alternatives like the multi-StatefulSet pattern or custom operator logic is crucial to avoid performance degradation and downtime during scaling events.

1. Invest in Robust Orchestration for Critical Operations: Complex, fleet-wide changes like migrations or major upgrades should not be managed by ad-hoc scripts. Implementing a durable workflow engine like Temporal provides resilience, observability, and automated recovery for multi-step processes, significantly reducing the risk of failures and simplifying operational burden.

1. Decouple Migration Logic: For one-time or infrequent, complex migrations, creating a dedicated migration controller separate from the main application operator is a sound defensive strategy. This prevents polluting the core operator codebase with transient logic, maintains its stability, and allows for specialized testing and rollback mechanisms for the migration process itself.

1. Enhance Database Cloud-Native Readiness: Application and database administrators must collaborate to identify and address inherent limitations in their chosen database's elasticity. This includes how databases handle catalog synchronization, DDL queues, external dependencies, and local state during replica additions or removals. Proactive engagement with database vendors or open-source communities to introduce features like relaxed validation for secondary syncs or intelligent DDL queue handling is essential.

1. Prioritize Zone Awareness and Anti-Affinity: To maintain high availability and disaster recovery capabilities, operators must actively ensure replicas are evenly distributed across availability zones. Implement explicit zone pinning or advanced Kubernetes scheduling features (like topology spread constraints) within custom operators to prevent unintended zone imbalances, especially during scaling, migrations, and recovery from outages.

1. Ensure Observability Data Persistence: The loss of replica-local system tables during "make before break" operations can severely impact troubleshooting and auditing. Defenders should implement mechanisms, such as offloading log and metric data to centralized storage (e.g., S3) with zero-copy attach capabilities, to ensure continuous observability history across pod lifecycles.

1. Plan for Phased Rollouts and Rollbacks: Even with extensive testing, real-world customer environments will reveal edge cases. Defensive strategies must include phased rollouts (batch migrations), comprehensive monitoring, and well-defined rollback procedures. Implementing "partial maintenance modes" that allow critical operations (like inserts/selects) to continue while limiting conflicting administrative actions is a valuable tactic.

Key Takeaways

• StatefulSet limitations for elasticity: Vanilla Kubernetes StatefulSets hinder "make before break" scaling for databases. A multi-StatefulSet approach (one StatefulSet per pod) provides the necessary granular control for non-disruptive scaling.
• Orchestration for complex migrations: Zero-downtime migration of a large stateful fleet requires sophisticated orchestration, often involving dedicated migration controllers and durable workflow systems like Temporal for robust execution, error handling, and state management.
• Database elasticity challenges: Databases have inherent limitations to cloud-native elasticity; specific features (DDL queues, catalog sync, materialized views, local system tables) require modifications or careful handling during dynamic replica changes.
• Zone balance is critical: Maintaining even replica distribution across availability zones is crucial for high availability during scaling and migration events, necessitating explicit zone pinning or advanced Kubernetes scheduling configurations.
• Persistent observability data: Moving replica-local operational data (e.g., query logs, metric logs) to object storage with zero-copy attach is essential for maintaining observability and historical context during "make before break" operations.
• Migrations are complex engineering: Even "live" migrations are non-trivial, requiring meticulous planning, custom tooling, rigorous testing, phased rollouts, and robust rollback strategies to ensure product reputation and customer satisfaction.

About the Speaker(s)

Jayme Bird and Manish Gill are Software Engineers at ClickHouse, deeply involved in the development and management of the ClickHouse cloud platform. Manish Gill, who initiated the talk by providing crucial background context on autoscaling and the limitations of StatefulSets, previously presented on the topic of "Fantastic Ordinals" at KubeCon Paris, indicating his expertise in Kubernetes orchestration for stateful workloads. Jayme Bird then delved into the specifics of the migration controller's implementation and the various ClickHouse-specific challenges encountered and solved during the migration, showcasing his hands-on experience in making ClickHouse more cloud-native and elastic. Both speakers contribute to ensuring ClickHouse can operate efficiently and reliably in dynamic cloud environments.