How To Gateway With Ingress - 140 Days InGate - Marco Ebert & James Strong

How To Gateway With Ingress - 140 Days InGate

Speakers: Marco Ebert, Site Reliability Engineer, Giant Swarm; James Strong, Solutions Architect, Isurvalent (now Cisco)

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=zTLbnstVjHc

Overview

This talk, "How To Gateway With Ingress - 140 Days InGate," delivered by Marco Ebert and James Strong, delves into the critical juncture facing the widely adopted Kubernetes Ingress NGINX controller and introduces its strategic successor, Ingate. The speakers, both long-standing maintainers of the Ingress NGINX project, candidly address the significant security vulnerabilities that recently plagued the controller, including a critical unauthenticated remote code execution (RCE) flaw. Beyond immediate security patches, the presentation outlines a profound shift for the Ingress NGINX project, signaling its transition into a maintenance-only mode and eventual archiving.

The core of the discussion revolves around the new Ingate project, an ambitious effort to build a next-generation Kubernetes ingress solution based on the Gateway API. This initiative is driven by a desire to overcome the technical debt, architectural limitations, and security challenges inherent in the legacy Ingress NGINX controller. By embracing the Gateway API, Ingate aims to provide a more secure, maintainable, and extensible foundation for ingress in Kubernetes, fundamentally altering how advanced traffic management features are implemented and managed.

The talk is crucial for anyone relying on Ingress NGINX in production, offering not only urgent security advisories but also a clear roadmap for the future of Kubernetes ingress. It highlights the community-driven effort required to build Ingate, the architectural principles guiding its development, and the significant feature parity challenges that must be addressed during the transition. For platform engineers, security professionals, and Kubernetes operators, understanding this strategic pivot is paramount for ensuring the security, stability, and future-proofing of their cluster networking.

Background

▶ Watch: Introduction and '40 days InGate' context (0:00)

The Kubernetes Ingress NGINX controller has been a cornerstone of many Kubernetes deployments, serving as the de facto standard for exposing HTTP and HTTPS routes to services within a cluster. Its widespread adoption, however, came with a growing burden of technical debt, primarily due to its highly extensible nature, which often involved exposing low-level NGINX configuration directives and a vast array of annotations. While offering immense flexibility, this approach inadvertently created complex attack surfaces and made maintainability increasingly challenging for the project's volunteer maintainers.

The immediate catalyst for this talk and the subsequent strategic shift was a series of critical security vulnerabilities, dubbed the "Ingress Nightmare" by media, that were published shortly before the conference. These CVEs exposed severe weaknesses in the controller's configuration validation and processing, leading to unauthenticated remote code execution and secret exfiltration. These incidents underscored the inherent risks of tightly coupled, highly configurable systems and amplified the urgency for a more secure and robust alternative.

In response to these challenges and in recognition of the evolving landscape of Kubernetes networking, the community has been actively developing the Gateway API. This API aims to provide a more expressive, role-oriented, and extensible approach to traffic routing, moving beyond the limitations of the original Ingress resource. The Ingate project emerges as the Ingress NGINX community's commitment to this new paradigm, representing a fresh start designed to leverage the security and architectural advantages of the Gateway API while shedding the legacy burden of its predecessor. The original timeline for Ingate envisioned significant progress within 140 days, but as the speakers candidly admit, the complexity of the CVEs and the foundational work required for Ingate pushed this timeline back, hence the revised "40 days InGate" in the title, reflecting the recent commencement of active community meetings for the new project.

Key Findings

▶ Watch: Introduction to 'Ingress Nightmare' critical CVEs (2:30)

The talk presented several critical findings and strategic decisions regarding the future of Kubernetes ingress:

• Critical CVEs in Ingress NGINX: The most significant finding was the disclosure of a critical unauthenticated remote code execution (RCE) vulnerability in the Ingress NGINX controller, alongside other related CVEs. This RCE, with a CVSS score of 9.8, stemmed from a flaw in how the controller validated NGINX configurations, allowing malicious input to inject configuration and load arbitrary code or modules. Other vulnerabilities enabled secret exfiltration. Immediate patching to versions 1.12.1 or 1.11.5 was strongly recommended.
• Ingress NGINX Entering Maintenance Mode: As a direct consequence of the technical debt and security challenges, the Ingress NGINX project is transitioning into a maintenance-only mode. This means future efforts will focus on security patches (CVEs), bug fixes, and dependency updates (Alpine, Golang, Kubernetes versions). Version 1.13 is projected to be the last minor release to include new features. The project is slated for archiving by 2027, contingent on the stability and migration path provided by Ingate.
• Introduction of Ingate as the Successor: Ingate was formally introduced as the spiritual successor to Ingress NGINX, a new project built from the ground up to implement the Gateway API. This project aims to provide a cleaner, more secure, and maintainable ingress solution for Kubernetes, addressing the architectural shortcomings of the legacy controller.
• Significant Feature Gap with Gateway API: A detailed analysis revealed a substantial gap between the extensive feature set of Ingress NGINX (especially its 118 annotations and numerous ConfigMap options) and the current capabilities of the Gateway API. Only 35% of Ingress NGINX features are currently supported in Gateway API, with another 55% expected to be supported soon (potentially by 2026). A significant 45% of features, including complex functionalities like mod_security and specific cookie settings, currently have "no plans" for implementation within the Gateway API, highlighting a critical area for community engagement.
• Architectural Overhaul for Security: Ingate's design prioritizes security by separating the controller from the NGINX proxy into distinct components. This addresses a major vulnerability source in Ingress NGINX, where the tightly coupled architecture allowed an exploit in the proxy to gain elevated privileges through the controller's permissions.

Technical Deep Dive

▶ Watch: Unauthenticated remote code execution CVE (9.8 critical) (3:00)

The technical core of the talk centered on the critical vulnerabilities discovered in the Ingress NGINX controller and the architectural shift planned for Ingate.

The "Ingress Nightmare" CVEs

The most severe vulnerability discussed was an unauthenticated remote code execution (RCE) with a CVSS score of 9.8, affecting the Ingress NGINX controller. This critical flaw was traced to specific lines of code responsible for validating NGINX configurations during the ingress resource validation phase. The admission controller, which handles this validation, was accessible without authentication. Attackers could craft malicious admission requests to inject arbitrary NGINX configuration directives.

The exploit leveraged the nginx -t flag, which is intended to test configuration syntax but, under certain conditions, could be coerced to load external modules or binaries. This allowed attackers to upload and execute code on the Ingress NGINX controller pod, leading to full RCE. Compounding this, other related CVEs enabled configuration injection and secret exfiltration. By manipulating annotations to reference secrets, attackers could cause the controller to write sensitive data (like API server tokens) to publicly accessible locations within the web server's file system, thereby gaining access to the entire Kubernetes API server.

To mitigate these risks, users were strongly urged to upgrade their Ingress NGINX deployments to versions 1.12.1 or 1.11.5. A significant change introduced in these patched versions is the disabling of the NGINX configuration test during ingress validation. While this prioritizes security, it means that invalid configurations might not be caught until NGINX attempts to reload, potentially causing disruptions if new pods are added or restarted. The speakers also emphatically stated that chroot environments, sometimes considered a security measure, are explicitly not a security boundary for NGINX and offer no protection against these types of vulnerabilities.

Ingress NGINX Maintenance Mode and Archiving

The Ingress NGINX project is entering a maintenance mode to manage its technical debt and facilitate a transition. This mode entails:

• Continued monthly patch releases for Alpine, Golang, Kubernetes, and other third-party dependencies.
• Ongoing bug fixes and, crucially, immediate CVE remediation.
• Version 1.13 will likely be the final minor release to accept new features. After this, new feature proposals will be evaluated on a case-by-case basis, with no general policy for merging.
• Support for new Kubernetes releases will continue, as this typically involves running existing end-to-end (E2E) tests against new kind clusters.
• The ultimate goal is to archive the Ingress NGINX project by 2027, once Ingate provides a stable and viable migration path.

The Ingate Project: A Gateway API Future

Ingate is designed as a fresh implementation of a Kubernetes ingress controller, built atop the Gateway API. Key architectural and philosophical decisions for Ingate include:

• Strict Gateway API Adherence: Ingate will provide a "very clean, very exact implementation" of the Gateway API. This means it will not expose low-level NGINX configuration directives, Lua scripting, or the extensive annotation system that characterized Ingress NGINX. The goal is to avoid the "sins of the past" that led to technical debt and security vulnerabilities.
• Decoupled Architecture: A cornerstone of Ingate's design is the separation of the controller and the NGINX proxy into distinct pods. Ingress NGINX ran both components in the same pod, granting the NGINX proxy elevated permissions it didn't strictly need, which was exploited in recent CVEs. Ingate will tackle the challenge of securely pushing configurations from the controller to a fleet of NGINX proxy pods.
• Leveraging Controller-Runtime: The project plans to use controller-runtime to accelerate the bootstrapping process for managing Gateway API objects like GatewayClass, HTTPRoute, and gRPCRoute.
• Conformance Testing: Ingate will prioritize passing Gateway API conformance tests to ensure interoperability and adherence to standards. This requires significant effort in writing bootstrap code and setting up testing infrastructure, likely using kind clusters initially, with community contributions needed for broader cloud testing (e.g., AWS, GCP, Azure, OpenShift).

Feature Gap Analysis and Community Call to Action

A critical aspect of Ingate's development is bridging the feature gap between Ingress NGINX and Gateway API. A detailed analysis revealed:

• 118 annotations in Ingress NGINX, grouped into approximately 40 categories.
• Only 35% of these features are currently supported in the Gateway API.
• Another 55% are slated for future support (e.g., nginx.ingress.kubernetes.io/auth-url is "in progress").
• A concerning 45% of features, including functionalities like nginx.ingress.kubernetes.io/set-cookie-path and more complex integrations such as mod_security (Web Application Firewall), currently have "no plans" for implementation in Gateway API.
• The analysis only covered annotations, not the extensive ConfigMap options for features like TCP/UDP routing, meaning the full gap is even larger.

This significant disparity represents a crucial call to action for the community. Users are urged to identify which Ingress NGINX features are critical for their deployments and engage with the Gateway API community to advocate for their inclusion. The Ingate team will prioritize features based on this feedback, ensuring that future implementations within the Gateway API framework are secure and well-designed.

Demo / Proof of Concept

▶ Watch: Chroot deployments are not a security measure (6:00)

The talk focused on the strategic transition, the critical security vulnerabilities, and the technical roadmap for Ingate, rather than a live demonstration of Ingate's current capabilities. Given that Ingate was only 40 days into its community meetings and early development stages, the speakers concentrated on outlining the vision, challenges, and architectural decisions for the new project. They provided insights into the ongoing discussions, the initial setup of the controller manager, and the work required for bootstrapping and conformance testing, emphasizing that much of the foundational code was still being written. Therefore, a functional proof of concept of Ingate itself was not presented during this session.

Defensive Implications

▶ Watch: Other project status: build, OpenResty, registry issues (6:30)

For organizations utilizing Kubernetes and particularly the Ingress NGINX controller, the defensive implications of this talk are immediate and far-reaching:

1. Immediate Patching for Ingress NGINX: The most urgent action is to upgrade all Ingress NGINX deployments to versions 1.12.1 or 1.11.5 without delay. This addresses the critical unauthenticated remote code execution (RCE) vulnerability (CVSS 9.8) and other related security flaws. Failure to patch leaves clusters highly susceptible to compromise.
2. Understand Security Trade-offs of Patches: Be aware that the patches disable the NGINX configuration test during ingress validation. This means invalid NGINX configurations might not be detected until runtime, potentially causing reload failures. Implement robust CI/CD pipelines and pre-deployment validation steps to ensure configuration correctness before applying changes to the cluster.
3. Do Not Rely on chroot for Security: The speakers unequivocally stated that chroot is not a security boundary for NGINX. Defenders should not consider it a primary isolation mechanism for their ingress controllers and should instead focus on other layers of defense, such as strict RBAC, network policies, and container runtime security.
4. Begin Planning for Gateway API Migration: With Ingress NGINX entering maintenance mode and slated for archiving, organizations should start evaluating their current Ingress NGINX usage and planning a migration strategy to Gateway API-based solutions like Ingate. This involves understanding the differences in API models and identifying which current features are critical.
5. Assess Feature Parity and Engage with the Community: Conduct an audit of all Ingress NGINX annotations and ConfigMap options currently in use. Compare these against the capabilities of the Gateway API. For features that fall into the "no plans" category (currently 45% of Ingress NGINX features), actively engage with the Gateway API community and the Ingate project (via Slack channels like #ingate-dev and #ingate-users, GitHub discussions, and community meetings) to advocate for their inclusion or find alternative approaches. This community feedback is vital for shaping Ingate's development priorities.
6. Embrace a More Secure Architecture: The decoupled controller and proxy architecture planned for Ingate represents a significant security enhancement. Defenders should prepare for this shift, understanding that it will lead to a more secure ingress layer by limiting the blast radius of potential exploits and adhering to the principle of least privilege.
7. Contribute to Ingate's Development: For organizations with the resources, contributing to Ingate's development – through coding, documentation, testing (especially on various cloud providers or OpenShift), or participating in discussions – will directly accelerate the availability of a stable, secure, and feature-rich Gateway API implementation. This is a chance to influence the future of Kubernetes networking.

Key Takeaways

• The Ingress NGINX controller has critical unauthenticated remote code execution (RCE) vulnerabilities (CVSS 9.8). Immediate upgrade to versions 1.12.1 or 1.11.5 is mandatory.
• The Ingress NGINX project is transitioning into maintenance mode, with v1.13 expected to be its last feature release, and is planned for archiving by 2027.
• Ingate is the new, community-driven project built on the Gateway API to provide a more secure, maintainable, and modern ingress solution for Kubernetes.
• Ingate will strictly adhere to the Gateway API, intentionally avoiding direct exposure of NGINX configuration or Lua scripting to prevent past security and technical debt issues.
• A significant feature gap exists between Ingress NGINX's extensive annotations/ConfigMap options and the current Gateway API capabilities, requiring active community involvement to prioritize and implement missing critical features.
• Ingate will feature a decoupled controller and NGINX proxy architecture, a key security enhancement to limit permissions and reduce the attack surface.
• The community is invited to contribute to Ingate's development, testing, and feature discussions to ensure a successful migration path for all users.

About the Speaker(s)

Marco Ebert is a Site Reliability Engineer at Giant Swarm, bringing over 10 years of experience in open-source projects and working with Kubernetes since 2016. He proudly became a maintainer of Ingress EngineX in November 2023. Outside of his technical pursuits, Marco is an avid climber and mountain biking enthusiast.

James Strong serves as a Solutions Architect at Isurvalent, which is now part of Cisco. He has been a maintainer of the Kubernetes Ingress NGINX project for many years. James is also the author of the O'Reilly book "Networking Kubernetes" and an instructor for Cloudera, focusing on topics derived from his book. In his personal life, he is a Gimli cosplay enthusiast.