Stateful Connections in Kubernetes: The Scaling Secrets Nobody... André Mocke & Rodrigo Fior Kuntzer

Stateful Connections in Kubernetes: The Scaling Secrets Nobody Talks About

Speakers: André Mocke, SRE; Rodrigo Fior Kuntzer, Visionary, Cloud-Native Lead

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=h1AyaAdf3HA

Overview

In this insightful KubeCon EU talk, André Mocke and Rodrigo Fior Kuntzer from Miro unveil the intricate journey of migrating Miro's critical real-time collaboration backend, specifically its websocket manager, from a traditional stateful EC2 service to a modern, stateless architecture on Amazon EKS. Their presentation dives deep into the often-overlooked complexities of managing long-lived, stateful connections within a Kubernetes environment, offering practical solutions and hard-won lessons.

The speakers meticulously detail the technical challenges encountered, ranging from Linux kernel limitations like ephemeral ports and connection tracking, to the nuances of load balancing and autoscaling for services handling highly variable traffic patterns. This talk is essential for platform engineers, SREs, and developers grappling with real-time applications in cloud-native settings, providing a comprehensive blueprint for achieving both scalability and resilience while significantly reducing operational costs.

Background

▶ Watch: Talk introduction, speakers, and agenda overview (0:00)

Miro, a leading online collaborative whiteboard platform, operates as a sophisticated "gaming engine for enterprise collaboration." At the core of its real-time functionality lies the board server, a stateful service crucial for low-latency interactions like collision detection and object locking. Historically, every user on a board required a persistent websocket connection to the same board server. This stateful requirement presented a significant routing challenge: how do clients consistently connect to the correct server?

Miro's legacy architecture relied on Fabio LB, an open-source HTTP and TCP reverse proxy, integrated with HashiCorp Consul for service discovery. Board servers would register themselves with Consul, and Fabio would dynamically update its routing tables, often using path-based routing where the URL's last segment directly mapped to an EC2 instance name. While functional, this setup introduced complexities: clients had to renegotiate connections if a server went down, and potential split-brain issues could arise if the internal board registration logic diverged from Consul's service discovery. This architecture, rooted in traditional monolithic designs, was far from cloud-native.

Recognizing the need for a more agile and scalable foundation, Miro embarked on a transformative journey in 2021. They committed to Kubernetes and Amazon EKS as their next-generation compute platform, adopting a microservices architecture. This platform integrated best-in-class operators and controllers, including Kartha for cluster autoscaling and Kyverno as a dynamic admission controller. By 2024, with the upcoming "Canvas 2024" launch event and its new real-time collaboration features, the limitations of the legacy websocket infrastructure became critical. Browsers impose hard limits on simultaneous connections to a single domain. To support new features requiring multiple logical connections multiplexed over a single physical connection to the client, a smarter, more flexible websocket routing solution was imperative. This necessity drove the migration of the websocket manager to Kubernetes, a collaborative effort involving Miro's Cloud Networking, Compute, and Collaboration Runtime teams.

The fundamental challenges in handling stateful connections, particularly websockets, within Kubernetes are multifaceted:

• Expensive New Connections: Websockets, being an extension of HTTP 1.1, involve a two-phase handshake process: first, establishing a secure TLS connection, and then upgrading it to a websocket connection. This dual handshake increases resource consumption and tail latency, especially under load.
• Keep-Alives and Idle Timeouts: Proper configuration of keep-alive mechanisms and idle timeouts across all hops in the communication flow is critical to prevent premature connection closures and ensure connection reuse. Misconfigurations can lead to frequent, unnecessary reconnections.
• Ephemeral Port Limits: On Linux, client connections utilize ephemeral ports, temporary ports assigned by the operating system. A connection tuple (source IP, destination IP, source port, destination port) has a theoretical limit of roughly 65,000 unique connections. In a Kubernetes pod, the default range for available ephemeral ports is often around 28,000. Exhausting this range leads to connection failures, manifesting as dropped requests and poor user experience. Vertical scaling alone cannot bypass this hard limit; horizontal scaling is essential.
• Conntrack Table Limits: The conntrack (connection tracking) feature in the Linux kernel maintains a table of all active network connections, vital for stateful firewalls and Network Address Translation (NAT). If this table becomes full, new connection attempts are silently dropped, impacting application availability and making diagnosis difficult. These limits necessitate careful consideration of node sharing and load distribution.

Key Findings

▶ Watch: Miro's transformative journey to Kubernetes and EKS (4:10)

Miro's ambitious migration yielded several pivotal findings and significant improvements:

• Successful State to Stateless Transition: The team successfully re-architected Miro's stateful websocket manager into a stateless service running on Amazon EKS. This fundamental shift unlocked greater scalability, resilience, and alignment with cloud-native principles.
• Development of the RTC Gateway: A custom-built, high-performance Real-Time Collaboration Command Gateway (RTC Gateway) was developed in-house to replace the legacy Fabio LB and Consul setup. This bespoke solution demonstrated superior performance characteristics tailored to Miro's specific websocket traffic patterns.
• Dramatic Performance Enhancement: The migration resulted in a 10x reduction in initial connection latency, significantly improving the real-time responsiveness and overall user experience for Miro's customers.
• Substantial Cost Reduction: By moving from EC2-based stateful services to a stateless, containerized solution on EKS, Miro achieved operational cost savings of approximately $40,000 per year, highlighting the economic benefits of cloud-native adoption.
• Mastery of Horizontal Autoscaling: Miro effectively leveraged KEDA (Kubernetes Event-driven Autoscaling) to implement sophisticated horizontal autoscaling based on custom metrics like active connections, ensuring optimal resource utilization and preventing saturation under varying loads.
• Innovative Load Balancing and Readiness Probes: The team devised an ingenious solution involving split readiness endpoints to address the challenges of load balancing long-lived connections and mitigating "cold starts" on newly scaled-up pods, ensuring smooth traffic distribution without disrupting existing user sessions.
• Robust Graceful Shutdowns: A comprehensive graceful shutdown mechanism was implemented, combining application-level connection draining, thoughtful ALB deregistration delays, and Kubernetes pre-stop hooks, eliminating user-facing connection disruptions during deployments and scaling events.

Technical Deep Dive

▶ Watch: Why smarter multiplexed websocket routing became essential (6:00)

The migration hinged on several critical architectural components and clever configurations:

Real-Time Collaboration Command Gateway (RTC Gateway)

The heart of the new architecture is Miro's custom RTC Gateway, an in-house developed application designed to replace Fabio LB and Consul. This gateway was engineered for maximum performance:

• It utilizes threads equal to the number of vCPUs to minimize context switching overhead.
• All inbound and outbound connections are processed within the same thread, further boosting efficiency.
• The default memory allocator was swapped for Jemalloc, a production-proven general-purpose malloc implementation. Jemalloc helps prevent memory fragmentation, which is particularly beneficial when handling wildly varying packet sizes and frequencies inherent in websocket traffic.

Edge Routing with AWS Load Balancer Controller

To route traffic from user devices to the new RTC Gateway pods, Miro leveraged the AWS Load Balancer Controller. This controller natively integrates Kubernetes APIs (Ingress, Services) with AWS load balancers. Crucially, they utilized a Custom Resource Definition (CRD) called Target Group Binding. This CRD allowed fine-grained configuration of how the Application Load Balancer (ALB) sends traffic to Kubernetes pods, defining load balancing rules and traffic protocols specific to their needs.

Securing Connections with Cert-Manager

Ensuring encrypted communication in transit without burdening SREs with manual certificate rotation was paramount. Cert-Manager, a CNCF controller, proved invaluable. It automates the provisioning and management of PKI (Public Key Infrastructure) within the cluster. Miro configured Cert-Manager to automatically provision certificates valid for a year, with monthly rotations. Combined with a maximum node lifetime of 30 days, this ensured that every pod always had a fresh, valid certificate upon startup, minimizing operational overhead and enhancing security posture.

Graceful Shutdowns

Transitioning from a stateful EC2 service to stateless EKS pods required a robust graceful shutdown mechanism to prevent user disruption during scaling down or rolling deployments. The goal was to avoid "stuttering connections" where users experience frequent disconnections and reconnections.

• Application-Level Draining: The RTC Gateway implemented a protocol to send close events to all connected clients. Clients were designed to transparently acquire new connections to existing, healthy pods upon receiving these events.
• ALB Deregistration Delay: To allow sufficient time for the RTC Gateway to drain connections, the ALB's deregistration delay was set to 4 minutes, double the application's estimated 2-minute draining time.
• Pre-Stop Hook: A Kubernetes pre-stop hook was used to ensure that a terminating pod was removed from the load balancing pool before the application began its connection draining sequence. This prevents the ALB from sending new connections to a pod that is actively trying to shut down.

Horizontal Autoscaling with KEDA

Given the ephemeral port and conntrack limits, horizontal autoscaling was prioritized over vertical scaling. Traditional resource-based scaling (CPU, memory) proved insufficient for the RTC Gateway due to the highly variable nature of websocket traffic (e.g., many users on a single board generating high frequency, small packets vs. a single user generating large, infrequent packets).

• Connection-Based Scaling: Miro's team performed extensive performance testing to determine the optimal resource ratio profiles, concluding that each pod could efficiently handle approximately 8,000 active connections without degradation.
• KEDA for Custom Metrics: They adopted KEDA (Kubernetes Event-driven Autoscaling), an operator that extends Kubernetes' native Horizontal Pod Autoscaler (HPA) capabilities to scale deployments based on custom metrics. Miro configured a ScaledObject to scale based on the "active connections" metric.
• KEDA Configuration: The ScaledObject was configured for rapid scale-up (no limit on the upsert rate) and slow scale-down (max one pod every 5 minutes). A rolling window sample was used to prevent premature scaling down during temporary metric dips, and a 5-minute cooldown period prevented "flapping" (rapid scale-up/down cycles). This precise configuration ensured the deployment remained well under saturation levels.

Sophisticated Load Balancing and Readiness Probes

Miro initially experimented with least active connections as the ALB load balancing algorithm, hoping to prevent older, saturated pods from being overloaded. However, this led to severe "cold starts" on newly scaled-up pods, which were immediately flooded with a barrage of new websocket connections. The CPU-bound work of TLS handshakes on mass caused significant latency spikes and unacceptable customer experience.

The team revisited round robin and devised an innovative solution: split readiness endpoints:

• Kubernetes Readiness Probe: The standard readiness probe (used by Kubernetes to determine if a pod is ready to serve traffic and be included in service endpoint slices) was configured to always report ready. This ensures that Kubernetes never removes a pod with active connections from the service, preventing the termination of existing user sessions.
• Target Group Specific Readiness Probe: A separate, dedicated endpoint was introduced specifically for the ALB's target group health checks. The RTC Gateway was updated to report "not ready" to this target group probe when its active connections exceeded 10,000, and "ready" again when connections dropped below 9,000.
• Result: This dual-probe strategy meant the ALB would stop sending new connections to saturated pods (because the target group probe failed), but Kubernetes would keep the pod in the service's endpoint slice, preserving existing connections. This successfully mitigated cold starts on new pods and ensured balanced load distribution without disrupting user experience.

HPA Algorithm Nuances

Miro also encountered two important nuances related to the HPA algorithm:

• Built-in Tolerance: The Kubernetes HPA algorithm has a default 10% tolerance. This means it will only trigger scaling actions if the observed metric deviates by more than 10% from the target. In EKS, this tolerance is often not configurable. Miro's developers needed to account for this tolerance when defining their KEDA thresholds to ensure predictable scaling based on their desired saturation levels.
• Flapping Metrics and Scale-Up Policies: An initial issue involved severe metric flapping, leading to uncontrolled scaling up (doubling of pods in a single event). After extensive debugging, they discovered their monitoring system was occasionally providing misleading data points to KEDA. This highlighted how blindly the HPA system follows metrics. The solution was to implement explicit scale-up policies in the KEDA ScaledObject definition. By default, HPA might allow 100% scale-up in a given minute. By setting a specific pods or rate limit in the behavior.scaleUp.policies section, they could control the maximum number of pods created in a time frame, mitigating the impact of erroneous metric spikes and preventing over-provisioning.

Demo / Proof of Concept

▶ Watch: Key teams and tools driving the websocket migration (7:00)

While the talk did not feature a live, interactive demonstration of the implemented system, the speakers thoroughly detailed the architectural changes, configuration specifics, and presented concrete performance metrics and operational results. This comprehensive exposition effectively served as a proof of concept for their successful approach to managing stateful connections in Kubernetes.

Defensive Implications

▶ Watch: Unpacking fundamentals and challenges of stateful connections (8:00)

The insights shared by Miro offer critical defensive strategies for organizations operating real-time, stateful applications on Kubernetes:

• Deep Protocol Understanding: Defenders must understand the nuances of the underlying wire protocols, especially for websockets (TLS handshake followed by HTTP upgrade). This knowledge informs proper timeout configurations and resource planning.
• Precise Keep-Alive and Timeout Configuration: Carefully configure idle timeouts and keep-alive settings across all components in the communication path (client, load balancer, proxy, application) to prevent premature connection closures and reduce reconnection overhead.
• Account for Linux Kernel Limits: Be acutely aware of operating system limits such as ephemeral port exhaustion (net.ipv4.ip_local_port_range) and conntrack table saturation. Design architectures with horizontal scaling in mind to distribute load and avoid hitting these hard limits.
• Prioritize Horizontal Scaling: For applications with long-lived, stateful connections, horizontal scaling is generally superior to vertical scaling. It provides better resilience against single-node failures and circumvents kernel-level connection limits.
• Implement Robust Graceful Shutdowns: Ensure applications can gracefully drain existing connections during termination. Combine application-level draining logic with load balancer deregistration delays and Kubernetes pre-stop hooks to prevent user-facing disruptions during deployments or scaling events.
• Leverage Custom Metrics for Autoscaling: Relying solely on CPU or memory for autoscaling stateful services with variable connection patterns can be inefficient. Utilize operators like KEDA to scale based on application-specific custom metrics, such as active connections, to achieve optimal saturation and resource utilization.
• Sophisticated Load Balancing with Split Readiness Probes: Adopt a nuanced approach to readiness checks. Differentiate between readiness for new connections (controlled by load balancer health checks, signaling saturation) and readiness for existing connections (controlled by Kubernetes readiness probes, ensuring existing sessions are preserved). This strategy effectively mitigates cold starts and ensures stable traffic distribution.
• Careful HPA/KEDA Configuration: Pay close attention to the HPA algorithm's built-in tolerance and configure explicit scale-up and scale-down policies. This prevents "flapping," over-scaling due to metric spikes, and ensures predictable autoscaling behavior.
• Validate Observability Data: Recognize that autoscaling systems act blindly on reported metrics. Ensure the accuracy and reliability of your monitoring infrastructure to prevent incorrect scaling decisions that can lead to performance degradation or resource waste.

Key Takeaways

• Modern real-time applications require smarter, multiplexed websocket routing to overcome browser connection limits and support complex features in a microservices architecture.
• Managing long-lived stateful connections in Kubernetes demands a deep understanding of the entire network stack, including TLS handshakes, HTTP upgrade protocols, and critical Linux kernel limits like ephemeral ports and conntrack table size.
• Horizontal autoscaling driven by custom metrics (e.g., active connections) via tools like KEDA is paramount for achieving stable performance and efficient resource utilization for stateful workloads.
• Graceful shutdown mechanisms are non-negotiable. They must integrate application-level draining, load balancer deregistration delays, and Kubernetes lifecycle hooks to ensure seamless user experience during scaling and deployments.
• Implementing split readiness endpoints—one for Kubernetes (preserving existing connections) and another for the load balancer (controlling new connections)—is a powerful technique to mitigate cold starts and ensure effective load balancing for stateful services.
• Careful configuration of HPA/KEDA scale-up/down policies and understanding the HPA algorithm's built-in tolerance are crucial for preventing erratic scaling behavior and optimizing resource allocation.

About the Speaker(s)

Rodrigo Fior Kuntzer is presented as the visionary force behind Miro's evolution into cloud-native technologies. He played a pivotal role in the adoption of Kubernetes and CNCF best practices, driving the development of Miro's Kubernetes platform. His work has been recognized by AWS through published case studies, highlighting his expertise and impact in the cloud-native space.

André Mocke is described as a newer SRE who ventured into infrastructure and platforming after a decade of experience as a product engineer. His background as a product engineer brings a valuable user-centric perspective to infrastructure challenges, reflecting a deep understanding of how platform decisions impact the end-user experience.