Emissary-ingress: Version 4 and the Road Ahead - Flynn, Buoyant

Emissary-ingress: Version 4 and the Road Ahead - Flynn, Buoyant

Speakers: Flynn; Buoyant

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=7U6nAxUxG6c

Overview

This talk, presented by Flynn, a maintainer of Emissary-ingress at Buoyant, provides a comprehensive update on the project's current status, its journey to becoming a fully community-driven initiative, and the ambitious roadmap leading to Emissary 4.0.0. Emissary-ingress functions as a Kubernetes API Gateway and Ingress controller, designed to solve the fundamental "Ingress problem": safely exposing services running inside a Kubernetes cluster to external users. The presentation delves into the technical challenges of evolving a long-standing open-source project within the dynamic Kubernetes ecosystem, particularly concerning Custom Resource Definitions (CRDs) and API versioning.

The talk is crucial for anyone currently using Emissary-ingress, considering it for future deployments, or interested in the complexities of maintaining core infrastructure components in a rapidly changing cloud-native landscape. Flynn emphasizes the project's transition away from its commercial origins, highlighting the significant need for community contributions to realize its ambitious goals. The discussion not only covers immediate release plans, such as the final Emissary 3.x release (3.10.0) and the upcoming Emissary 4.0.0, but also explores strategic long-term visions, including potential integration with the Kubernetes Gateway API to streamline its architecture and reduce maintenance overhead.

Background

▶ Watch: Introduction: What is Emissary Ingress? (0:00)

Emissary-ingress has a rich history, dating back to 2017, predating the widespread adoption and standardization of Custom Resource Definitions (CRDs) in Kubernetes. This early genesis led to several design decisions in its initial API, particularly in versions V1 and V2, that do not align with modern structural CRD definitions. A canonical example cited by Flynn is the ambassador ID field, which was originally designed to accept either a string or a list of strings—a flexibility not supported by structural CRDs. This incompatibility necessitated the use of conversion webhooks to manage breaking changes when transitioning to V3 alpha 1, a process described as "remarkably unpleasant" for both developers and users.

Beyond the structural CRD challenges, Emissary-ingress faced an additional hurdle related to its API group. Historically, the project used the getambassador.io API group, inherited from its original affiliation with Ambassador Labs (formerly Datawire, now Ambassador). As Emissary-ingress has fully transitioned into a community-owned, open-source project, completely divorced from Ambassador Labs, the reliance on a domain (getambassador.io) that the project no longer controls became untenable. Changing the API group presents a unique challenge, as the Kubernetes API server lacks direct support for conversion webhooks across different API groups, making a clean transition exceptionally difficult.

At its core, Emissary-ingress is an API Gateway and Ingress controller built on Envoy Proxy. It addresses the "Ingress problem" by acting as the edge component of a Kubernetes cluster, mediating external requests to internal services. Its feature set extends far beyond basic routing, encompassing advanced capabilities such as authentication and authorization, sophisticated traffic splitting for canary deployments and A/B testing, retries, circuit breaking, and rate limiting. Flynn characterizes Emissary as a developer-centric, self-service, role-based, opinionated CNCF incubating open-source API server. The "developer-centric" aspect is particularly emphasized, aiming to minimize the friction Kubernetes often presents to application developers by providing an intuitive configuration language—the mapping resource—that allows them to deploy and expose their applications independently. The "role-based" nature allows infrastructure teams to manage the underlying gateway, while application developers focus on their specific service configurations.

The transition to a fully community-driven project from Ambassador Labs was amicable but introduced new operational challenges. The project now relies entirely on volunteers, leading to a critical need for increased community involvement to achieve its ambitious development goals. This shift underscores a broader trend in the cloud-native ecosystem where projects evolve from commercial backing to independent community stewardship, bringing both freedom and the responsibility of self-sustained development.

Key Findings

▶ Watch: Emissary's developer-centric, self-service design (3:00)

The talk reveals several critical developments and strategic directions for the Emissary-ingress project:

1. Final Emissary 3.x Release: Emissary 3.10.0 has been tagged and released just prior to the talk, marking it as the definitive final release in the Emissary 3 series. This release incorporates support for Endpoint Slices, addressing modern Kubernetes resource management. However, no further Emissary 3 builds are planned, underscoring the project's pivot towards version 4.
2. Community-Driven Project in Critical Need of Help: The project has successfully completed its divorce from Ambassador Labs and is now a fully community-maintained open-source initiative. This transition, while positive for independence, has highlighted a severe shortage of volunteer contributors, significantly impacting the project's ability to achieve its development roadmap. Flynn issues a strong call to action for community members to get involved.
3. Emissary 4.0.0 as the Primary Future Focus: The immediate and overarching goal is the development and release of Emissary 4.0.0. A development build for Emissary 4 is already in progress, running successfully on ARM64 architectures. This new major version represents a significant refactoring effort, including the "cathartic" removal of obsolete code and CRD conversion logic.
4. Mandatory API Group Change: Emissary 4 will introduce a new API group, moving away from the getambassador.io domain to emissary.dev. This change is non-negotiable due to the project's independence from Ambassador Labs. While Kubernetes API server lacks native support for converting CRDs across API groups, Emissary 4 plans a unique migration strategy.
5. Clean Migration Path for Emissary 4: To address the breaking changes and API group shift, Emissary 4 is designed for a mechanical, tool-assisted migration. Users will be able to run Emissary 3 and Emissary 4 side-by-side in the same cluster, each with its own services and configurations, minimizing disruption during the upgrade process. This approach is intended to be much cleaner than previous major version transitions due to the distinct API groups.
6. API Clean-up and Standardization: Emissary 4 will seize the opportunity presented by the breaking changes to clean up its API. This includes removing underscores from field names and explicitly adding units to duration fields, enhancing consistency and usability.
7. Potential Integration with Kubernetes Gateway API: A significant strategic finding is the potential for Emissary to evolve into a "thin translation layer" over the Kubernetes Gateway API. This opportunity is driven by the impending winding down of Ingress NGINX and its replacement by Ingate, which will be built on Gateway API. The necessity for Ingate to support standard API gateway features might force the Gateway API to mature more rapidly, allowing Emissary to leverage it and potentially offload the maintenance of its underlying Envoy Proxy configuration, which is currently a substantial effort.

Technical Deep Dive

▶ Watch: Past challenges: CRDs and API versioning (5:00)

The technical challenges facing Emissary-ingress, particularly with the transition to version 4, are deeply rooted in Kubernetes API evolution and project governance. The core issue stems from Emissary's pre-CRD origins, where custom resources were defined in ways that later became incompatible with the strict schema enforcement of structural CRDs. The ambassador ID field, allowing either a string or a list of strings, is a prime example. Modern CRDs require a fixed type, forcing a breaking change in Emissary 3 alpha 1, which was mitigated using conversion webhooks. These webhooks dynamically translate between different versions of a CRD schema during API requests, ensuring compatibility for older configurations while allowing new ones. However, they add complexity and maintenance burden.

The more significant technical hurdle for Emissary 4 is the API group change. Emissary's historical API group, getambassador.io, is tied to a domain no longer owned or managed by the Emissary-ingress project. A fundamental requirement for Emissary's independence is to transition to a new API group, emissary.dev. The Kubernetes API server, however, does not natively support direct conversion webhooks across different API groups. This means a direct, in-place upgrade from getambassador.io CRDs to emissary.dev CRDs via conversion webhooks is not feasible.

To circumvent this limitation, Emissary 4 adopts a strategy of a mechanical translation and side-by-side migration. This involves:

1. New API Group and CRDs: Emissary 4 will introduce entirely new CRD definitions under the emissary.dev API group, designed to be structurally compliant and improved (e.g., consistent field naming, explicit units for durations).
2. Migration Tooling: The project plans to develop tooling that will mechanically translate existing Emissary 3 configurations (using getambassador.io CRDs) into the new Emissary 4 format (emissary.dev CRDs). This tooling will ensure a predictable and automated upgrade path for users.
3. Side-by-Side Deployment: Crucially, Emissary 3 and Emissary 4 instances can run concurrently within the same Kubernetes cluster. This is facilitated by their distinct API groups and the requirement for separate Kubernetes Services (e.g., different port numbers). This allows users to test Emissary 4 with a subset of traffic or configurations before fully committing to the migration, providing a robust rollback strategy. This approach is a significant improvement over previous major version transitions, which often involved more complex, in-place upgrades.

Another critical technical consideration is Emissary's underlying proxy, Envoy. Emissary currently manages and configures Envoy directly. The speaker highlights a strategic opportunity to potentially reduce this maintenance burden by integrating with the Kubernetes Gateway API. The Gateway API is a newer, more expressive, and extensible API for managing ingress and traffic routing in Kubernetes, intended to eventually supersede the older Ingress resource. The context for this opportunity is the announced winding down of Ingress NGINX and its replacement by Ingate, which is being built on Gateway API. If the pressure to support common API gateway features for Ingate leads to a rapid maturation of the Gateway API, Emissary could potentially become a "thin translation layer" that maps its developer-centric configuration language onto the standardized Gateway API. This would allow Emissary to leverage the community's efforts in Gateway API development, potentially offloading the direct management and configuration of Envoy, which is a complex and resource-intensive task for the Emissary maintainers. This strategic shift could significantly improve the project's long-term sustainability.

Finally, the discussion touches upon Endpoint Slices. Emissary 3.10.0 includes support for Endpoint Slices, a more scalable and efficient way to track network endpoints for services in Kubernetes. The question arose whether Emissary 3.10.0 would gracefully handle the complete absence of the older Endpoints resource (as it is slated for deprecation). Flynn clarifies that 3.10.0 is designed to use Endpoint Slices if present, falling back to Endpoints otherwise. While he believes it would handle the absence of Endpoints gracefully, he also notes that any such issues would become a strong motivator for users to upgrade to Emissary 4.

Demo / Proof of Concept

▶ Watch: Emissary as a community project: Call for help (7:00)

While the talk did not feature a live, interactive demo, Flynn did mention a significant proof of concept: a development build of Emissary 4.0.0 is already running successfully on ARM64. This internal test build serves as a critical validation of the refactoring efforts and the removal of deprecated code, demonstrating the project's tangible progress towards the next major release. The speaker’s enthusiasm for the "cathartic" process of cutting unnecessary code underscores the practical, hands-on development work being done.

Defensive Implications

▶ Watch: Emissary 3.10.0 released: The final Emissary 3 (8:00)

For organizations relying on Emissary-ingress, the insights from this talk carry several crucial defensive implications:

1. Plan for Emissary 4 Migration: With Emissary 3.10.0 being the final 3.x release, users must begin planning their migration to Emissary 4. This isn't just a minor update; it involves significant breaking changes, particularly the shift to a new API group (emissary.dev). Organizations should anticipate a period of transition and allocate resources for testing the new version and utilizing the forthcoming migration tooling.
2. Understand the API Group Change: The move from getambassador.io to emissary.dev for CRDs means that existing configurations will not be directly compatible. Defenders need to be aware that this change will require a mechanical translation of their existing Emissary 3 configurations. The planned side-by-side deployment capability of Emissary 3 and 4 will be crucial for a controlled, risk-averse migration, allowing for thorough testing before cutting over traffic.
3. Leverage Emissary's Advanced Gateway Features: Emissary-ingress offers a robust suite of API Gateway features beyond simple routing, including authentication, authorization, rate limiting, and circuit breaking. Defenders should fully utilize these capabilities to enhance the security posture of services exposed via Emissary. Implementing strong authentication at the edge, rate limiting to prevent abuse, and circuit breaking to enhance resilience against backend failures are critical security practices.
4. Stay Informed on Gateway API Evolution: The potential for Emissary to become a "thin translation layer" over the Kubernetes Gateway API is a strategic long-term development. Defenders should monitor the progress of the Gateway API and its adoption, as this could simplify Emissary's architecture, potentially leading to a more secure and stable foundation by leveraging broader community efforts.
5. Community Engagement for Sustained Security: The project's full transition to a community-driven model means its continued development, maintenance, and security patches are reliant on volunteer contributions. Organizations that benefit from Emissary-ingress should consider contributing back to the project, whether through code, documentation, bug reports, or financial sponsorship, to ensure its long-term viability and security. Neglecting community involvement could lead to slower bug fixes and security updates.
6. Clarify Licensing and Support: It's important to differentiate between the open-source Emissary-ingress project and Ambassador Edge Stack. Emissary is purely open source with no commercial license key requirements. Ambassador Edge Stack, a commercial product, has its own licensing model. Organizations using Emissary-ingress should be aware that commercial support is not currently available for the open-source project, reinforcing the need for internal expertise or community engagement.
7. Beware of Artifact Confusion: The speaker highlighted past confusion regarding unofficial 3.12.2 artifacts mistakenly dumped into Emissary's repository by Ambassador Labs' build machinery. Defenders should ensure they are pulling official Emissary releases from designated sources (e.g., GHCR as planned for future artifacts) to avoid deploying unofficial or potentially broken versions.

Key Takeaways

• Emissary 4.0.0 is the Project's Future: The Emissary-ingress project is fully focused on the upcoming Emissary 4.0.0, which will introduce significant breaking changes, including a new API group (emissary.dev), and substantial internal clean-up.
• Final Emissary 3.x Release: Emissary 3.10.0 is the last planned release in the 3.x series, meaning users must plan for migration to Emissary 4 for future updates and features.
• Community Contributions are Critical: Now a fully community-driven project, Emissary-ingress desperately needs more volunteer contributors to achieve its ambitious roadmap, including documentation migration and Emissary 4 development.
• Smooth Migration Path Planned: Emissary 4 will support side-by-side deployment with Emissary 3 and will provide tooling for mechanical translation of configurations, ensuring a cleaner and less disruptive upgrade process despite the breaking changes.
• Strategic Gateway API Integration: The project is exploring the possibility of becoming a thin translation layer over the Kubernetes Gateway API, potentially offloading Envoy Proxy maintenance and leveraging broader community efforts in the cloud-native ecosystem.
• Developer-Centric API Gateway: Emissary-ingress remains a powerful, developer-centric API Gateway offering robust features like traffic splitting, authentication, authorization, rate limiting, and circuit breaking, designed to simplify exposing services in Kubernetes.

About the Speaker(s)

Flynn is a dedicated maintainer of the Emissary-ingress project, currently affiliated with Buoyant. His deep involvement with Emissary-ingress (and its predecessor, Ambassador) spans many years, providing him with an intimate understanding of its architecture, history, and the challenges of evolving such a critical piece of Kubernetes infrastructure. Flynn is a passionate advocate for open-source community involvement, frequently emphasizing the need for contributors to sustain and advance projects like Emissary-ingress. He is known for his direct and transparent communication style, particularly when discussing technical hurdles and the strategic direction of the project. Flynn can be reached on the CNCF Slack for further discussions and community engagement.