How We Progressively Deliver Changes To Kubernetes Using Canary Deployments and Featur... Bob Walker

How We Progressively Deliver Changes To Kubernetes Using Canary Deployments and Feature Flags

Speakers: Bob Walker, Field CTO, Octopus Deploy

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=zZ7bDPZMCqY

Overview

In this insightful KubeCon EU talk, Bob Walker, Field CTO at Octopus Deploy, delves into the intricate challenges and innovative solutions his company adopted for progressively delivering changes to their Kubernetes-hosted SaaS platform. The presentation focuses on the evolution of their deployment strategies, highlighting the transition from traditional, time-consuming methods to a sophisticated system integrating Canary deployments with feature flags. Walker articulates how this hybrid approach addresses the complexities of managing over 1,600 customer-dedicated application instances while maintaining high availability and rapid iteration cycles.

The core of Walker's discussion revolves around the critical need to decouple the deployment of new code from the release of new functionality. He debunks common misconceptions about Canary deployments, illustrating how Octopus Deploy leverages them not just for traffic routing but for strategic risk mitigation across a vast, multi-tenant environment. The talk is particularly relevant for organizations struggling with frequent releases, managing large-scale Kubernetes deployments, or seeking to enhance their progressive delivery capabilities, offering actionable solutions derived from real-world experience.

Walker’s presentation is a candid look at the self-inflicted challenges and subsequent triumphs of a company that builds deployment software, meaning their customers have exceptionally high expectations for uptime and reliability. By sharing Octopus Deploy's journey, he provides a blueprint for other engineering teams aiming to achieve faster lead times, improved service level objectives (SLOs), and a more resilient software delivery pipeline in a dynamic cloud-native landscape.

Background

▶ Watch: Speaker introduction and progressive delivery talk overview (0:00)

Octopus Deploy, initially launched in June 2012 as a self-hosted software solution, underwent a significant architectural and operational transformation with the introduction of its Software-as-a-Service (SaaS) offering in July 2018. This shift from a traditional on-premises model to a cloud-native SaaS platform presented a unique set of challenges, particularly given the company's commitment to providing a deployment tool that customers rely on for their own production environments.

The architecture adopted for the SaaS offering is a cell-based architecture, designed for scalability and customer isolation. Each customer receives a dedicated instance of the Octopus Deploy application, comprising a dedicated Kubernetes pod, a unique namespace, a dedicated database, and a dedicated file share. This granular isolation allows for dynamic scaling of compute resources per customer, accommodating diverse workloads ranging from tens to thousands of deployments per day. These customer cells are hosted within larger Kubernetes clusters, referred to as reefs, which also include Azure SQL Server and Azure storage. Currently, Octopus Deploy manages approximately 1,600 customer instances across three Azure regions, distributed over eight Kubernetes clusters (with a ninth soon to come online). Each reef can host between 300 to 500 instances and is backed by robust Virtual Machine Scale Sets with three to seven large nodes.

This infrastructure, while powerful, introduced several self-inflicted challenges. A primary concern was the expectation that the Octopus Deploy service would be operational 24/7 for customer deployments, regardless of maintenance activities. To manage this, customers could select a specific maintenance window, typically during off-peak hours, allowing Octopus Deploy to perform necessary upgrades and backups. However, a significant portion of customers defaulted to a midnight maintenance window, leading to a concentrated load and potential disruption. Compounding this, Octopus Deploy itself practices extremely frequent deployments, sometimes multiple times a day, meaning the 24-hour upgrade cycle for customer instances would frequently reset. Furthermore, new features and functionality are released continuously throughout the quarter, rather than in large, infrequent "big bang" deployments.

The biggest mental hurdle stemmed from the company's long history as a self-hosted product. In the self-hosted era, upgrades occurred at the customer's pace, often spanning months. With the SaaS offering, this multi-month release window compressed dramatically to a mere 24 hours. The immediate consequence was that if a critical code bug was discovered during an upgrade, it would propagate to all remaining instances, impacting a significant portion of their customer base before it could be halted. This highlighted an urgent need for a more robust, controlled, and de-risked deployment strategy for their cloud-native environment.

Key Findings

▶ Watch: Session goals: solutions, debunking myths, separating deployment (2:00)

Octopus Deploy's journey through progressive delivery yielded several pivotal findings that reshaped their approach to software releases. Initially, they implemented a Canary deployment strategy, segmenting customers into staff, insiders (MVPs), early adopters (a static list), stable, and laggards. While this introduced a phased rollout, it extended the upgrade cycle to a lengthy 15 days. Crucially, this initial approach proved insufficient, as critical bugs still managed to escape into the stable ring even after a fortnight of staged releases. This demonstrated that static Canary rings, while helpful for general code stability, could not account for every possible code path, configuration, or unique customer use case, thus failing to adequately de-risk new features and functionality.

A significant operational innovation was the adoption of the Andon cord concept, pioneered by Toyota. Treating their deployment pipeline as an assembly line, Octopus Deploy empowered anyone within the team to "pull the cord," immediately halting the deployment process if an issue was detected. This fostered a culture of psychological safety, encouraging prompt reporting of problems without fear of blame. While effective in preventing critical bugs from propagating to all instances once identified, the underlying issue of detecting those bugs early remained.

The most profound realization was that the overwhelming majority of issues stemmed from the introduction of new features and functionality, rather than existing code spontaneously breaking. This shifted their focus from merely de-risking code deployments to strategically managing the rollout of new capabilities. The solution involved a paradigm shift: separating the deployment of new code versions from the release of new functionality.

This led to their next-generation solution: a refined Canary deployment strategy for code stability, dramatically shortening the upgrade time for all customers from 15 days to a maximum of 5 days. This new model involved immediate dogfooding on their main deploy instance, followed by a randomized, rotating selection of 5% of customer instances for the Canary ring. Concurrently, they embraced feature flags, powered by OpenFeature, as the primary mechanism for controlling the release of functionality. This allowed for trunk-based development, enabling small, frequent merges to main while keeping new features hidden until ready. The combination of these strategies resulted in significant improvements: a reduced lead time to changes (from well over 15 days to an average of 4 days and 3 hours, with a goal of one day), and a notable increase in their per-customer service level objectives (SLOs) to 99.9%, even when accounting for scheduled maintenance.

Technical Deep Dive

▶ Watch: Octopus Deploy's cell-based SaaS architecture explained (3:00)

Octopus Deploy's journey involved a significant evolution in its technical approach to progressive delivery, moving from a basic staged rollout to a sophisticated combination of Canary deployments and feature flagging.

Initial Canary Deployment Strategy (Pre-2024)

The first iteration of their Canary deployment strategy aimed to mitigate risk by progressively exposing new code versions to different groups of users. The rollout rings were structured as follows:

1. Staff Instances: First to receive updates, used for internal validation.
2. Insiders: A group of MVPs (Most Valuable Professionals) and trusted users, updated after 1 day if staff instances were stable.
3. Early Adopters: A static list of customers, updated 7 days after insiders, assuming positive feedback.
4. Stable: The majority of customers, updated 7 days after early adopters.
5. Laggards: A small group updated last.

This sequential process meant that a full upgrade cycle for all customers took approximately 15 days. While providing some level of risk mitigation, this approach had critical flaws. The static nature of the early adopter list meant it couldn't comprehensively cover all possible code paths or customer-specific configurations. As a result, critical bugs still frequently bypassed these rings, making it all the way to the stable ring, leading to prolonged exposure to defects and significant remediation efforts. This highlighted the limitation of using Canary deployments solely for de-risking new features and functionality.

The Andon Cord Implementation

To address the issue of critical bugs propagating, Octopus Deploy adopted the Andon cord concept, inspired by Toyota's manufacturing processes. The deployment pipeline was conceptualized as an assembly line, where any team member could "pull the cord" to immediately halt all ongoing deployments. This mechanism was crucial for preventing widespread impact when a severe bug was detected post-deployment. The implementation fostered a culture of psychological safety, ensuring that individuals felt empowered to stop the process without fear of reprisal, as demonstrated by Bob Walker himself pulling the cord two weeks prior to the talk. While not a preventative measure, the Andon cord proved invaluable for damage control, preventing identified critical bugs from reaching hundreds or thousands of additional customer instances.

Next-Generation Progressive Delivery (Current Approach)

Recognizing the shortcomings of their previous methods, Octopus Deploy overhauled its strategy, focusing on significantly reducing upgrade times and intelligently decoupling code deployment from feature release.

1. Accelerated Code Canary Deployments:

• The total time to upgrade all customer instances was drastically reduced from 15 days to at most 5 days.
• Dogfooding: The Octopus Deploy internal instance, which is used to deploy Octopus Deploy itself, is always the very first instance to be upgraded. This ensures immediate internal validation of the core deployment mechanism.
• Dynamic Canary Ring: Instead of a static list, 5% of customer instances are randomly selected and rotated for the Canary ring. These instances receive updates within 1 to 3 days (at most 3 days). This dynamic selection ensures broader coverage of configurations and use cases, increasing the likelihood of early bug detection.
• Stable Ring: The vast majority, 93-94%, of customer instances reside in the stable ring, receiving updates after the Canary phase.

1. Feature Flag-Driven Functionality Release (OpenFeature):

• The most significant shift was the adoption of feature flags as the primary mechanism for releasing new functionality, using OpenFeature. This enabled a move towards trunk-based development, where feature branches are kept small and short-lived (2-3 days, maximum a week) and merged into the main branch frequently. The main branch is always expected to be deployable.
• Phased Feature Rollout:
• New features are initially merged into the main codebase with their corresponding feature flags turned off in production environments, but turned on in lower environments for testing.
• Over several months, additional functionality for that feature is developed and merged, remaining hidden behind the flag.
• Once confidence is built, the feature flag is incrementally enabled:
• First, for the majority of staff instances.
• Then, for multiple early adopter programs concurrently. Unlike the old static Canary rings, feature flags allow for different groups of early adopters to test different features simultaneously. Octopus Deploy currently runs four separate early adopter programs.
• Finally, the feature is gradually enabled for the rest of the customer base. This ramp-up can be highly controlled (e.g., 5%, then 10%, 15%) for high-risk features like a new user interface, or a quicker 5% to 100% for lower-risk items.
• Dynamic Control and Exclusion: A key advantage of feature flags is the ability to pause a rollout if issues are reported. Furthermore, if a bug is specific to a unique customer configuration, that particular customer can be excluded from receiving the new functionality while the rollout continues for others. This provides granular control and prevents widespread impact from isolated issues.
• Internal Interface: The talk showcased a screenshot of Octopus Deploy's internal feature flagging interface, which visualizes the status of flags across customer instances using "chicklets" to represent individual customer environments, allowing specific toggles for specific customers.

By combining rapid, dynamic Canary deployments for code stability with granular, OpenFeature-driven feature flags for functionality release, Octopus Deploy achieved a highly agile and resilient progressive delivery model. This technical architecture underpins their ability to deliver changes frequently while significantly reducing the risk of customer-facing issues.

Demo / Proof of Concept

▶ Watch: Challenge: managing customer maintenance windows for upgrades (5:00)

While the talk did not feature a live, interactive demonstration of the progressive delivery system in action, Bob Walker did provide crucial visual aids in the form of screenshots from Octopus Deploy's internal systems. These served as a "proof of concept" by illustrating the practical application of their feature flagging capabilities and the impact of their revised deployment strategies.

One key visual was a comparison of the Octopus Deploy user interface at the start of 2024 versus the end of 2024, showcasing a significant UI overhaul. This example effectively highlighted the challenge of introducing such a drastic change to users without a robust feature flagging system, emphasizing the potential for user confusion if a traditional Canary deployment (where the entire UI might flip back and forth) were the sole method. The animation of the UI transition underscored the need for controlled, gradual feature exposure.

Another critical screenshot displayed Octopus Deploy's internal feature flagging interface. This visual depicted a matrix-like view where individual "chicklets" represented specific customer instances. Walker explained how this interface allows their teams to precisely control which customers receive a particular feature, enabling granular rollout strategies. For instance, he demonstrated how they could toggle a feature on for specific staff instances or a select group of customers, providing concrete evidence of their ability to manage multiple concurrent early adopter programs and even exclude individual customers from a feature if a unique configuration issue arises. This visual evidence clearly demonstrated the fine-grained control over functionality release that feature flags, powered by OpenFeature, provide.

Defensive Implications

▶ Watch: Challenge: impact of very frequent production deployments (6:30)

The progressive delivery strategies implemented by Octopus Deploy offer a robust blueprint for other organizations seeking to enhance their security, stability, and agility in a Kubernetes environment. Defenders should consider the following implications:

1. Embrace Progressive Delivery: The talk strongly advocates for combining Canary deployments with feature flags as a core defensive strategy. Canary deployments provide initial risk mitigation for new code versions by exposing them to a small, controlled subset of the environment. Feature flags, powered by solutions like OpenFeature, then offer granular control over the release of functionality, allowing features to be toggled on/off independently of code deployment. This separation is crucial for de-risking new capabilities, which are identified as the primary source of production issues.

1. Adopt Trunk-Based Development: Moving from long-lived feature branches to trunk-based development with small, short-lived branches (2-3 days) is a critical enabler. This reduces merge conflicts, keeps the main branch perpetually deployable, and ensures that new code is integrated and tested continuously. From a defensive perspective, smaller changes are easier to review, test, and revert if necessary, reducing the blast radius of potential vulnerabilities.

1. Implement an "Andon Cord" Mechanism: Empowering any team member to halt a deployment process provides an essential safety net. This Andon cord fosters a culture of psychological safety, encouraging prompt reporting of issues without fear of blame. Defenders should integrate such a mechanism into their CI/CD pipelines, ensuring it can be activated quickly to prevent widespread propagation of critical bugs or security vulnerabilities.

1. Invest in Advanced Observability and Monitoring: While progressive delivery reduces risk, proactive detection of issues remains a challenge. Octopus Deploy's experience shows it takes an average of 5 days and 20 hours to detect issues. Defenders need sophisticated monitoring tools that can differentiate between various types of errors (e.g., user misconfiguration vs. widespread critical bug), track new error messages, and alert appropriate teams rapidly. Faster detection directly translates to reduced mean time to recovery (MTTR) and minimized customer impact.

1. Establish Robust Feature Flag Management and Hygiene: The power of feature flags comes with the responsibility of managing them effectively. Defenders must ensure:

• Clear Policies: Define when a feature toggle is appropriate (e.g., high-risk features are a "no-brainer," low-impact new pages are debatable).
• Expand and Contract Pattern: Developers must be disciplined in implementing the feature toggle (expansion phase with if/then statements and tests) and, crucially, in cleaning up the toggle and associated code once the feature is fully rolled out and stable (contraction phase). The warning about DynaTrace having 1,200 unused feature toggles highlights the technical debt that can accumulate without this discipline.
• Auditability and Access Control: Feature flag systems should have robust auditing capabilities to track who enabled/disabled what, and granular access controls to prevent unauthorized changes.

1. Learn from "Defect Escapes" with Blameless Retrospectives: Octopus Deploy tracks high severity defect escape rates (issues causing Sev1/Sev2 incidents). Even with advanced strategies, some issues will make it to production (e.g., two in December for Octopus Deploy). Conducting blameless retrospectives is vital for understanding root causes, improving processes, and preventing recurrence. This continuous learning loop strengthens the defensive posture over time.

1. Consider Self-Hosted Implications: For organizations with both SaaS and self-hosted offerings, feature flags often apply primarily to SaaS. The self-hosted release cadence can serve as a forcing mechanism for cleaning up feature toggles, as those features must be fully baked and flags removed before shipping to self-hosted customers who may not have telemetry or cloud connectivity. This highlights the need for a bifurcated strategy or careful management of feature parity.

By integrating these defensive implications into their software development and operational practices, organizations can build more resilient, secure, and adaptable systems capable of handling the demands of continuous delivery in a cloud-native landscape.

Key Takeaways

• Decouple Deployment from Release: The most critical takeaway is to separate the act of deploying new code versions from the act of releasing new functionality. Use Canary deployments for validating code stability and feature flags (like OpenFeature) for controlled, progressive rollout of features.
• Embrace Trunk-Based Development: Foster a culture of small, short-lived feature branches and frequent merges to the main branch. This approach, coupled with feature flags, keeps the main branch always deployable and reduces integration headaches.
• Canary Deployments are More Than Traffic Routing: Canary deployments can be leveraged at the instance or customer level, not just for routing network traffic. This allows for de-risking code changes across a multi-tenant environment by exposing them to a rotating subset of users.
• Empower Teams with an "Andon Cord": Implement a mechanism that allows anyone to stop the deployment pipeline if an issue is detected. This fosters psychological safety, encourages early problem reporting, and prevents widespread impact.
• Feature Flags Offer Granular Control: Feature flags provide unprecedented control over functionality release, enabling multiple concurrent early adopter programs, controlled ramp-ups (e.g., 5%, then 10%, 15%), and the ability to pause or even exclude specific customers from a feature if unique issues arise.
• Discipline is Key for Feature Flag Hygiene: Implement processes for adding feature flags (expand) and, critically, for removing them (contract) once a feature is stable. Without discipline, feature toggles can lead to significant technical debt and code complexity.

About the Speaker(s)

Bob Walker is the Field CTO at Octopus Deploy. In this role, he functions as an individual contributor, focusing on external engagement rather than setting company direction or managing teams. His responsibilities include speaking at conferences, developing thought leadership pieces, and interacting directly with customers to understand their challenges and provide solutions. Bob is based in Omaha, Nebraska, where he lives with his wife, two cats, and a dog, and enjoys biking in his free time. He began his career as a developer in the early days of .NET, bringing a wealth of practical experience to his current role.