Flink on Karmada: Building Resilient Data Pipelines on Multi-Cluster K8s - Michas Szacillo & Wang Li
Michas Szacillo, Wang Li
KubeCon + CloudNativeCon Europe 2025 · Session
This talk, presented by Michas Szacillo and Wang Li from Bloomberg, delves into the critical challenges of operating large-scale, stateful streaming applications like Apache Flink in a multi-cluster Kubernetes environment. It highlights the journey undertaken by Bloomberg's streaming platform team, in close collaboration with the Karmada community, to achieve automated, stateful failover for Flink jobs across multiple Kubernetes clusters. The core of their solution involves leveraging **Karmada**, an open-source Kubernetes management system, to orchestrate applications and implementing custom enhancements for Flink-specific state preservation during failover events.
AI review
This talk presents an exceptional deep dive into building truly resilient, stateful data pipelines using Apache Flink on multi-cluster Kubernetes, leveraging Karmada and Kverno. The Bloomberg team's collaboration with the Karmada community to engineer custom state preservation and injection mechanisms for Flink's job ID represents a significant, production-grade defensive innovation, solving a critical operational challenge of automated cross-cluster failover for stateful applications. This is not theoretical fluff; it's a meticulously detailed, hard-won solution that will directly inform…