Redesigning Ingress: Docker’s Transition To the Next-Gen Gate... Kateryna Nezdolii & Ryan Hristovski

Redesigning Ingress: Docker’s Transition To the Next-Gen Gateway

Speakers: Kateryna Nezdolii, SRE, Docker; Ryan Hristovski, Infrastructure Engineer, Docker

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=Ea5OuNjpi9M

Overview

In this KubeCon EU talk, Kateryna Nezdolii and Ryan Hristovski from Docker share their extensive journey in redesigning the company's critical ingress system. The presentation details Docker's strategic shift from a complex, legacy HAProxy and Nginx stack to a modern, Kubernetes-native solution built upon Envoy Proxy and the Gateway API. This migration was driven by significant operational pain points, including technological fragmentation, high latency, excessive resource consumption, and a lack of modern observability.

The speakers meticulously outline the architectural challenges of their previous setup and the rigorous evaluation process that led them to select Envoy and the Gateway API. They delve into the technical intricacies of their new architecture, the innovative migration strategies employed, and the unforeseen challenges encountered along the way. This talk is highly relevant for organizations grappling with legacy ingress systems, those considering adopting the Gateway API, or anyone interested in large-scale infrastructure modernization within a Kubernetes ecosystem. It offers a candid look at the complexities and benefits of such a transformative project, providing valuable insights for both infrastructure engineers and SREs.

Background

▶ Watch: Docker's ingress redesign journey begins (0:00)

Docker's legacy ingress system, which had served the company for nearly a decade, presented a growing number of operational and technical challenges. The architecture involved multiple layers of proxies and supporting services, leading to significant complexity and fragility. A typical Docker Hub request would traverse an external AWS Network Load Balancer (NLB), then hit an external HAProxy instance responsible for basic abuse rate limiting. If cleared, the request would be forwarded to Nginx, which handled JWT token validation using Lua scripts, checked Redis for session revocation, and then passed authenticated requests to an internal NLB and internal HAProxy for load balancing to the final backend service.

This multi-hop setup created several core pain points:

1. Console Reliance: The system heavily depended on Consul clients and servers for key-value storage, dynamic reconfiguration, and in some cases, service discovery. Running Consul introduced another critical component that needed to be highly available, making the entire ingress system more fragile and susceptible to downtime or networking issues within Consul itself. This increased the maintenance burden and risk profile.

1. Nginx Complexity and Overhead: Nginx was introduced approximately 10 years ago to address requirements that HAProxy couldn't fulfill at the time. Operated by a separate external team, it created a functional gap between the proxy layers. Nginx acted as an additional proxy hop for nearly all traffic, introducing unnecessary latency, complexity, and maintenance overhead. Its reliance on Lua scripts for JWT-based authentication, rate limiting, session management, and CORS configuration meant Docker was managing multiple, disparate rate limiting and authentication mechanisms across different layers, further increasing operational overhead and potential points of failure. The goal was to consolidate these functionalities into a single, unified proxy.

1. HAProxy Sidecar Blowout: Running HAProxy in their environment necessitated a complex ecosystem of supporting components. This included two init containers for healthy startup, the main HAProxy container, two sidecars for dynamic configuration generation, two for log management, two for additional monitoring, and finally, the HAProxy controller. This proliferation of components significantly increased the risk of cascading failures, required careful coordination across all parts, consumed excessive resources, and made debugging a nightmare.

1. Lack of OpenTelemetry: A critical missing piece in the legacy stack was native support for OpenTelemetry. As all other services at Docker had adopted OpenTelemetry for tracing, the ingress layer represented a massive hole in their observability story, hindering effective troubleshooting and performance analysis.

These cumulative issues underscored the urgent need for a more modern, efficient, and maintainable ingress solution that could consolidate functionalities, improve observability, and scale with Docker's evolving needs.

Key Findings

▶ Watch: Identifying critical pain points: Console, Nginx, sidecars (1:35)

The migration to a new ingress system yielded several transformative findings and improvements for Docker:

1. Strategic Proxy Selection: Envoy: After evaluating various proxies, including Traefik, Docker ultimately selected Envoy Proxy as the foundation for its next-generation ingress. While Traefik offered simpler runtime updates and Go-based extensibility, Envoy was favored for its superior high-load performance, advanced native features, robust community, and the potential for Go-based Wasm extensions. Envoy’s dynamic XDS reconfiguration capabilities were a crucial differentiator, enabling real-time updates without service restarts. The team's confidence was further bolstered by Kateryna Nezdolii's role as an Envoy maintainer, providing direct insight and influence over its development. This choice aligned best with Docker's long-term strategy for scale, performance, and efficiency.

1. Kubernetes-Native Control Plane: Gateway API and Envoy Gateway: To manage the new Envoy instances, Docker opted for the Kubernetes Gateway API implemented by Envoy Gateway. This decision was made after rejecting a fully custom XDS control plane (due to significant engineering effort) and the standard Kubernetes Ingress object (which lacked necessary complexity and customization). The Gateway API provided a richer routing model, operational simplicity, and seamless integration with existing Kubernetes workflows, enabling self-service routing for application teams. Envoy Gateway, having recently graduated to General Availability, provided advanced routing capabilities without requiring Docker to build a control plane from scratch.

1. Significant Performance and Efficiency Gains: The new architecture delivered substantial improvements in performance and resource utilization. Docker observed a four-fold increase in system throughput and a 50% reduction in the number of CPU cores used for ingress, with potential for further optimization. Request latency was also notably improved by eliminating multiple unnecessary proxy hops in the request path.

1. Enhanced Reliability and Observability: The new system drastically improved overall reliability and availability through the use of ALB switch canary support and a clear separation of control and data plane layers. The adoption of Envoy natively supported OpenTelemetry, providing Docker's customers with more detailed observability and richer tracing capabilities, a critical missing feature in the legacy stack.

1. Streamlined Developer Experience and Reduced Fragmentation: By merging multiple previously used proxy technologies (HAProxy, Nginx) into a single Envoy proxy stack, Docker significantly reduced technology fragmentation. The Kubernetes-native, unified routing API provided a consistent and centralized way for users to interact with the ingress system, simplifying configuration and empowering internal developers.

1. Innovative Migration Strategy with AWS ALB: A key finding was the successful implementation of a granular, real-time migration strategy utilizing AWS Application Load Balancer (ALB) rule-based traffic shifting. This allowed Docker to migrate individual services incrementally, routing specific percentages of traffic to Envoy while maintaining the rest on the legacy HAProxy stack. This approach offered instantaneous rollbacks, providing immense confidence during the transition and overcoming the limitations of DNS-based weight shifting.

1. Challenges and Workarounds Highlight API Maturity: The migration also highlighted the evolving maturity of the Gateway API. Docker encountered challenges with documentation lags and missing features (e.g., global rate limiting, IP tagging, native canary support) at the time of migration. The team had to resort to using Envoy patch policy, an unstable API, to modify generated XDS configurations directly. This revealed complexities in merging Gateway API configurations with raw XDS patches and the lack of a definite merge strategy, sometimes leading to unpredictable results like filters being "nuked." These experiences underscore the need for early adopters to be prepared for such workarounds and to actively contribute to upstream projects.

Technical Deep Dive

▶ Watch: The 'sidecar blowup' and resource consumption issues (3:00)

The technical redesign of Docker's ingress system represents a comprehensive overhaul, moving from a multi-component, custom-scripted environment to a highly integrated, Kubernetes-native solution.

New Ingress Architecture

The new architecture introduces a streamlined and robust request flow:

1. Entry Point: Dual-Stack AWS Application Load Balancer (ALB): The request flow begins with the client connecting to a dual-stack AWS ALB. This Layer 7 load balancer serves as the primary entry point to the Docker network. Traffic is logically split into external (HTTPS or gRPC) and internal (VPN + HTTPS or gRPC) streams. The ALB performs crucial functions:

• TLS termination: Decrypting incoming HTTPS traffic.
• Host-based routing: Directing traffic based on the requested hostname.
• Load balancing: Distributing decrypted requests across target proxy groups based on health and capacity statuses.

1. Data Plane: Envoy Proxies: After decryption and initial routing by the ALB, requests are sent over HTTP/1 or HTTP/2 connections to selected Envoy Proxy instances. These proxies, which form the data plane, are grouped by Gateway resources, each serving a specific traffic type (e.g., external, internal, stable, canary). Each gateway aggregates specific business logic for its traffic type, such as rate limiting, geolocation decoration, and authentication. Envoy proxies process requests through their filter chains, applying the configured business logic. If a request does not terminate at the proxy (e.g., direct response), it is then load balanced and proxied to an upstream or backend service using HTTP/1 or gRPC.

1. Global Rate Limit Service: Docker heavily utilizes rate limiting. Most incoming traffic is first routed over gRPC to a global rate limit service prior to being sent upstream. This service is an open-source implementation called Envoy Proxy Rate Limiter, a Go service that is part of the Envoy ecosystem. It operates with managed Redis instances for state. The rate limit service is dynamically configured via XDS by the control plane.

1. Control Plane: Envoy Gateway: The Envoy Gateway acts as the foundational component of Docker's ingress ecosystem. It implements the Kubernetes Gateway API, provisioning and managing infrastructure components like the data plane (Envoy proxies) and the rate limit service. Its core responsibilities include:

• Dynamic configuration: Providing real-time updates to the data plane and rate limit service via Delta XDS.
• Kubernetes object consumption: Consuming, validating, and transforming user-submitted Kubernetes objects (e.g., HTTPRoute, BackendTrafficPolicy) into native Envoy configurations.

Internal Abstraction Layer

To simplify the developer experience and enforce best practices, the Ingress team introduced an internal abstraction layer on top of the Envoy Gateway API. Docker engineers rely on this custom internal API, which acts as a wrapper Helm chart for the Gateway API chart, to configure routing for their services in a decentralized manner.

This abstraction layer offers several benefits:

• Sensible defaults: It prepopulates configurations with defaults that cover most use cases for the Docker ecosystem.
• System integrity: It ensures the ingress system is used in a sensible way, preventing configurations that could exhaust proxy memory (e.g., buffering requests for an hour).
• Complexity abstraction: It hides the intricacies of the Gateway API configuration surface, allowing engineers to focus on feature delivery.

When a user wants to expose a service, they add the internal Gateway Routing API Helm chart as a dependency. This downloads templates for HTTPRoute and BackendTrafficPolicy with pre-configured settings for health checks and timeouts.

User Flow for Service Exposure

The process for a Docker engineer to expose a service via the new ingress system is:

1. Define HTTP Routes: Engineers define HTTPRoute objects using the custom Gateway Routing API Helm chart within their service repository.
2. Submit Configuration: The configuration is submitted to the Kubernetes cluster via a deployment pipeline.
3. OPA Validation: An Open Policy Agent (OPA) agent running within the cluster validates the routing configuration against admission rules (e.g., checking that the domain and path are not already in use).
4. Kubernetes Admission: If OPA validation passes, the configuration is admitted and deployed to the Kubernetes cluster.
5. Envoy Gateway Processing: The Envoy Gateway monitors Kubernetes objects like HTTPRoute, validates them against the Gateway API schema.
6. XDS Transformation and Delivery: If validation is successful, the new routing configuration is transformed into a raw XDS representation and sent to the proxy layer (Envoy instances).
7. Proxy Consumption: The Envoy proxies validate the supplied XDS configuration and consume it if validation passes.

Migration Strategy: Leveraging AWS ALB

The migration from the legacy NLB/HAProxy stack to the ALB/Envoy stack was meticulously planned to minimize disruption.

1. NLB to ALB Transition: The first step involved replacing the Layer 4 NLB with a Layer 7 ALB. This was achieved using Route 53 DNS weight shifting, gradually moving a small percentage of traffic from the NLB to the ALB while monitoring metrics. A key challenge encountered was the ALB's behavior of appending an additional internal IP address of the load balancer itself to the X-Forwarded-For header chain, requiring application modifications.

1. ALB as the Migration Enabler: While DNS weight shifting was useful for the initial cutover, its limitations (caching delays, unpredictable distribution, lack of service-specific path-based routing) made it unsuitable for granular service migrations. The ALB, once in place, provided powerful routing rules that could be applied to domain names and paths. This was the critical enabler for migrating individual services one at a time. Unlike DNS, ALB offered instantaneous and accurate traffic shifting mechanisms, allowing for rapid rollbacks if issues arose. For instance, Docker could route 10% of traffic for "Service A" to Envoy and leave 90% on HAProxy, gradually increasing the Envoy percentage. The ALB had a hard limit of 200 rules, which was sufficient for Docker's needs.

1. Post-Migration Simplification: Once all services were transitioned, the environment was simplified, with the ALB defaulting its listener to Envoy. The legacy HAProxy remained as an emergency fallback. The ALB also facilitated canary deployments for new Envoy versions or features, allowing precise traffic percentages to be sent to a canary Envoy service, independent of pod counts.

1. AWS Load Balancer Controller: All ALB components are built and managed using the AWS Load Balancer Controller. This Kubernetes controller allows for on-the-fly management of ALB configurations without relying on slower infrastructure-as-code tools like Terraform for every change. Traffic weight shifting and unique forwarding rules (domains, paths, references) are managed through annotations and Helm chart configurations.

Challenges and Workarounds

The migration, while successful, presented several challenges:

• Learning Curve and Documentation: Adopting new, complex technologies like Kubernetes Gateway API and the Envoy ecosystem required a significant learning investment. Documentation for Envoy Gateway sometimes lagged behind implementation, necessitating direct source code inspection.
• Missing Gateway API Features: At the time of migration, certain critical features like global rate limiting and IP tagging were not yet natively supported by the Gateway API. Docker addressed this using the Envoy patch policy mechanism, an unstable API that allows users to modify the generated Envoy XDS configuration before it's sent to the proxy. This avoided developing and maintaining a custom control plane extension but came with its own complexities:
• Verbosity and Envoy Knowledge: Patch policies are verbose and demand deep knowledge of Envoy's filter chain configuration.
• Unpredictable Merge Strategy: A major issue was the lack of a definite merge strategy between features defined in the Gateway API and those defined via XDS patching. Partial enabling sometimes led to unpredictable results, where essential business logic filters were "nuked" when attempting to patch in new features like connection limits, though it surprisingly worked for rate limiting.
• Lack of Native Canary Support (Gateway API): The Gateway API lacked native canary support for both data and control planes. Experiments with Argo CD Rollouts were unsuccessful due to the dynamic configuration management by Envoy Gateway, which required manual duplication of numerous config objects for canary deployments. Docker ended up with a custom, manual canary setup for the data plane and no canary for the control plane, highlighting a critical area for upstream contribution.
• Secure Envoy Admin Endpoints: There was no easy solution or documented example for configuring role-based access to write admin endpoints in the Envoy admin console. This meant anyone with cluster access could potentially invoke critical operations like draining traffic fleet-wide.
• Remote Access to Envoy Admin Console: Secure remote access to the Envoy admin console is typically disabled. During incidents, checking server status or specific stats across multiple Envoy instances required slow, manual port-forwarding loops, hindering rapid troubleshooting.

Demo / Proof of Concept

▶ Watch: Gateway API chosen for rich routing and Kubernetes native (6:00)

While the talk did not feature a live demonstration of a proof-of-concept, the speakers provided detailed configuration examples and architectural diagrams that illustrated how the new ingress system was implemented and managed. Key examples included:

• Helm chart configurations for the internal Gateway Routing API, showing how developers define HTTP routes and backend traffic policies with sensible defaults.
• OPA agent validation flow, detailing the steps from user configuration submission to Kubernetes admission and Envoy Gateway processing.
• AWS Load Balancer Controller configurations, demonstrating how traffic weight shifting for different backend services (legacy HAProxy vs. new Envoy) is managed through annotations on the ALB. This included concrete examples of routing 90% of traffic to Envoy and 10% to Canary Envoy for a fully migrated service, versus 90% to legacy HAProxy and 10% to new Envoy for an ongoing migration service.
• XDS patch policy example, showcasing the verbose format required to configure an entire filter chain when Gateway API features were not yet available, and highlighting the challenges with unpredictable merging behavior.

These detailed examples served as a practical walkthrough of the system's capabilities and how Docker engineers interact with it, effectively illustrating the "how it worked" aspect without a live demo.

Defensive Implications

▶ Watch: High-level overview of the new Envoy Gateway ingress (6:30)

Docker's ingress redesign offers several critical defensive implications for organizations looking to enhance their security posture and operational resilience:

1. Consolidated Security Posture: By merging multiple proxy technologies (HAProxy, Nginx) into a single Envoy Proxy stack, Docker significantly reduced technological fragmentation. This consolidation simplifies security management, reduces the attack surface by eliminating unnecessary components, and allows for consistent application of security policies (e.g., authentication, rate limiting) at a single, well-understood layer rather than across disparate systems with varying capabilities and configurations.

1. Leveraging L7 Security Features: The transition to AWS Application Load Balancer (ALB) as the primary entry point, coupled with Envoy, enables advanced Layer 7 security features. ALB provides native TLS termination, host-based routing, and robust health checks, offloading these responsibilities from the backend services. Envoy, with its rich filter chain capabilities, can then enforce sophisticated security policies such as advanced rate limiting, authentication/authorization filters, and IP tagging, all configured through a standardized API.

1. Granular and Secure Deployments: The ability to perform granular traffic shifting with ALB rules is a powerful defensive tool. It allows for canary deployments with precise traffic percentages, enabling security teams to monitor new code or configuration changes in a controlled environment before a full rollout. In case of security regressions or vulnerabilities introduced in a new deployment, the instantaneous rollback capability minimizes exposure and blast radius, allowing for rapid incident response.

1. Kubernetes-Native Policy Enforcement: Adopting the Kubernetes Gateway API and integrating it with an Open Policy Agent (OPA) for admission control provides a robust framework for enforcing security policies at the configuration layer. OPA validates routing configurations against predefined rules (e.g., preventing unauthorized domain use, enforcing secure path patterns) before they are admitted to the cluster, acting as an early warning system against misconfigurations that could lead to security vulnerabilities.

1. Enhanced Observability for Threat Detection: The native OpenTelemetry support in Envoy provides comprehensive tracing and metrics. This enhanced observability is crucial for detecting anomalous behavior, identifying potential attacks (e.g., volumetric attacks triggering rate limits, unusual request patterns), and accelerating forensic analysis during a security incident. Detailed traces across the ingress path allow for quicker pinpointing of compromised components or malicious activity.

1. Addressing Admin Interface Security: The identified challenge regarding the lack of native role-based access control (RBAC) and secure remote access for Envoy's admin endpoints is a critical defensive concern. Unrestricted access to these endpoints could allow an attacker with cluster access to drain traffic, disable security filters, or exfiltrate sensitive configuration details. Organizations must implement custom solutions, such as network policies, dedicated jump hosts, or secure proxy layers, to restrict and audit access to these powerful interfaces. This could involve leveraging existing identity and access management (IAM) solutions to gate access to the Envoy admin console securely.

1. Community Collaboration for Security Improvements: Docker's active contributions to the Envoy and Envoy Gateway projects (bug fixes, feature requests for canary support, secure admin access, global overload settings) highlight the importance of community involvement. This collaboration directly contributes to improving the security features and stability of these critical open-source components, benefiting the entire ecosystem. Defenders should encourage and participate in such upstream efforts to drive security enhancements.

Key Takeaways

• Consolidate for Simplicity and Performance: Docker's migration demonstrates that consolidating disparate legacy proxy technologies (HAProxy, Nginx, Consul) into a single, modern Envoy Proxy stack significantly reduces operational complexity, improves request latency, and boosts system throughput by four times while reducing core usage by 50%.
• Embrace Kubernetes Gateway API for Modern Ingress: The Kubernetes Gateway API, implemented by Envoy Gateway, provides a powerful, Kubernetes-native control plane for advanced routing, enabling self-service for developers and a unified API experience, despite an initial learning curve and evolving feature set.
• L7 Load Balancers are Key for Granular Migration: Leveraging AWS Application Load Balancer (ALB) with rule-based traffic shifting is a critical enabler for safe, granular, and instantaneous migrations and canary deployments. This approach offers superior control and rollback capabilities compared to traditional DNS-based methods.
• Be Prepared for API Gaps and Custom Workarounds: Early adopters of rapidly evolving APIs like the Gateway API should anticipate potential documentation lags and missing features. Solutions like Envoy patch policy can fill these gaps, but come with complexities like verbosity and unpredictable merging behaviors that require deep technical understanding and careful implementation.
• Prioritize Observability and Secure Admin Access: Integrating OpenTelemetry for rich tracing is essential for understanding system behavior and rapid incident response. Simultaneously, addressing the lack of native RBAC and secure remote access for Envoy's admin endpoints is crucial for preventing unauthorized operations and facilitating secure troubleshooting.
• Active Upstream Contribution is Vital: Docker's experience underscores the importance of actively contributing to open-source projects like Envoy and Envoy Gateway. Reporting bugs, requesting features, and collaborating with the community helps mature the technology and address shared challenges, benefiting all users.

About the Speaker(s)

Kateryna Nezdolii is an SRE at Docker and a maintainer of the Envoy Proxy project. Her deep involvement with Envoy significantly contributed to Docker's confidence and success in adopting the technology for their ingress redesign. She brings extensive expertise in site reliability engineering and proxy technologies.

Ryan Hristovski is an Infrastructure Engineer at Docker. He played a key role in the ingress redesign project, focusing on the practical implementation and migration strategies. His insights into the challenges and solutions of transitioning a large-scale system are invaluable.