Are You Covered? Falling in Love With E2E Testing - Scott McAllister, ngrok

Are You Covered? Falling in Love With E2E Testing

Speakers: Scott McAllister, Developer Advocate, ngrok (at time of talk); now Loft Labs

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=emjrmJZR-ZI

Overview

In this KubeCon EU talk, "Are You Covered? Falling in Love With E2E Testing," Scott McAllister, then a Developer Advocate at ngrok and now with Loft Labs, presents a comprehensive approach to building robust and reliable cloud-native applications through end-to-end (E2E) testing integrated with GitOps principles. The presentation delves into how a well-structured Continuous Integration/Continuous Deployment (CI/CD) pipeline, leveraging tools like Cypress, K3S, ngrok, and Argo CD, can ensure application functionality, accelerate development cycles, and maintain environmental consistency from development to production.

McAllister emphasizes that thorough testing, particularly E2E testing that simulates the user experience, is paramount for verifying code behavior and preventing costly bugs. Beyond just testing, the talk highlights the critical role of GitOps in managing infrastructure and application configurations as code, ensuring traceability, auditability, and automated deployments. This combination empowers development teams to push changes rapidly and confidently, knowing their applications will perform as expected in production environments. The methodologies presented are crucial for any organization striving for high-quality, resilient, and efficiently managed Kubernetes deployments.

Background

▶ Watch: Why end-to-end testing is crucial for software. (0:50)

The journey of software development is fraught with potential pitfalls, from subtle code errors to complex integration failures. To mitigate these risks, various forms of testing are employed. Unit testing focuses on the smallest possible code units or functions, ensuring individual components work correctly in isolation. Integration testing then verifies that these individual components interact correctly when combined. However, the ultimate validation comes from end-to-end (E2E) testing, which simulates the complete user flow through an application, interacting with the user interface, backend services, and data stores. This type of testing is critical because it mirrors the real-world experience, uncovering issues that might only manifest when the entire system operates together.

A recurring challenge in software deployment is the discrepancy between testing and production environments. Often, an application functions perfectly in a controlled test environment but encounters unexpected issues in production due to differences in configuration, dependencies, or underlying infrastructure. McAllister stresses the importance of making test environments "as close to production as possible." While containers promise consistency by bundling application code and dependencies, the configuration around those containers – how they are deployed, networked, and managed – can still vary significantly. This disparity can lead to the infamous "it works on my machine" or "it worked in staging" problems, undermining confidence in deployments.

To address these challenges, the talk introduces GitOps, a modern operational framework that extends DevOps principles to infrastructure management. GitOps mandates that the desired state of an application and its infrastructure be declared as code in a Git repository, which then serves as the single source of truth. All changes to the system, whether application updates or infrastructure modifications, are initiated through Git commits. This approach offers several benefits: a clear audit trail of all changes, easy rollback capabilities, and the ability to leverage Git's powerful collaboration and versioning features. A key GitOps pattern highlighted is the rendered manifest pattern, where fully templated and resolved Kubernetes manifests (containing all final values like image tags and URLs) are committed to Git, rather than just templates that are rendered at deployment time. This ensures that the exact configuration deployed to production is version-controlled and auditable.

Key Findings

▶ Watch: Test environments must mirror production closely. (4:00)

The core message of Scott McAllister's presentation is that a holistic and automated approach to testing and deployment is indispensable for modern cloud-native applications. Several key findings and contributions emerge:

1. Comprehensive Testing Builds Stronger Applications: The talk reinforces that while unit and integration tests are essential, end-to-end testing is crucial for validating the complete user experience across the entire application stack. A balanced testing strategy covering all layers—from individual functions to full user flows—significantly strengthens application reliability and resilience.
2. Environmental Consistency is Non-Negotiable: A central tenet is the necessity of making test environments as identical to production as possible. This includes not only containerized application code but also the surrounding Kubernetes configurations, networking, and external dependencies. Inconsistent environments are a primary source of post-deployment issues.
3. GitOps Enhances Confidence and Auditability: By adopting GitOps principles, where Git serves as the single source of truth for both application code and infrastructure configuration, teams gain unparalleled traceability, version control, and automation. The rendered manifest pattern specifically ensures that the exact configuration deployed to any environment (test or production) is explicitly committed and auditable, removing ambiguity.
4. Integrated Tooling Enables Robust CI/CD: The demonstration effectively showcases how a combination of open-source tools—Cypress for E2E UI testing, K3S for ephemeral Kubernetes clusters in CI, ngrok for secure ingress in test environments, GitHub Actions for CI orchestration, and Argo CD for continuous deployment—can form a powerful, automated pipeline. This integration allows for rapid iteration, automated validation, and reliable deployments.
5. Ephemeral, Production-Like Test Environments are Achievable: For complex applications, especially those heavily reliant on Kubernetes Custom Resource Definitions (CRDs), spinning up full, isolated test clusters can be resource-intensive. The introduction of vCluster as a solution for creating "clusters inside a cluster" highlights a powerful method for generating ephemeral, production-grade test environments that share the host cluster's CRDs, offering a scalable and efficient testing paradigm.

These findings collectively advocate for a development and operations philosophy that prioritizes automated validation across the entire software delivery lifecycle, underpinned by transparent and auditable infrastructure management.

Technical Deep Dive

▶ Watch: Demo application: "Hamburger Tracker" overview. (4:50)

The technical core of McAllister's talk revolves around a practical demonstration of integrating E2E testing with a GitOps-driven CI/CD pipeline for a Kubernetes application. He uses a simple yet illustrative "hamburger joints" application, comprising a frontend and a backend, to showcase the workflow.

The application's architecture is straightforward: two distinct containers for the frontend and backend, each with its own Docker image. These images are deployed to a Kubernetes cluster using standard Kubernetes manifests (Deployments, Services, and Ingress). A key detail is the use of image labels (e.g., burger-back-image:TAG) that will be dynamically updated during the build process to reflect the exact commit SHA.

For E2E testing, McAllister employs Cypress, a popular JavaScript-based testing framework. Cypress allows developers to write tests that interact with the application's UI, simulating user actions like clicking buttons, filling forms, and asserting expected outcomes. The demo Cypress test sequence includes:

1. Visiting the application's URL.
2. Finding and asserting the presence of an "Add" button.
3. Clicking the "Add" button.
4. Filling out a form to add a new hamburger joint.
5. Saving the new entry.
6. Verifying the new entry appears in the list.

Crucially, Cypress can capture screenshots upon test failure, providing valuable debugging information.

The entire CI/CD process is orchestrated using GitHub Actions. The workflow is triggered on pull requests (both initial opening and subsequent synchronizations/commits) targeting the main branch. The GitHub Action performs the following steps:

1. Environment Setup: Runs on an Ubuntu runner, installing K3S (a lightweight Kubernetes distribution) to provision an ephemeral Kubernetes cluster directly within the CI environment. This ensures a consistent, isolated testing ground.
2. Image Building and Tagging: Logs into Docker, then builds the frontend and backend Docker images. Each image is tagged with the SHA of the Git commit that triggered the workflow. This practice is vital for traceability, linking a specific deployed container directly back to its source code revision. The images are then pushed to a remote Docker registry (e.g., DockerHub).
3. Configuration Management (GitOps in Action):

• The GitHub Action checks out a separate configuration repository (config repo). This adheres to GitOps principles, separating application code from infrastructure definitions, acknowledging their different lifecycles and team responsibilities.
• Within the config repo, Kubernetes manifests contain placeholder values (e.g., BURGER_FRONT_IMAGE_TAG, FRONTEND_URL).
• The sed command (or templating tools like Kustomize or Jinja) is used to perform a find-and-replace operation on these manifests, injecting the dynamically generated image SHAs and the test URL.
• This step exemplifies the rendered manifest pattern: the CI pipeline generates fully rendered Kubernetes manifests with all specific values resolved.

1. Ingress Provisioning: The ngrok operator is installed via Helm. ngrok is chosen as the ingress provider because it simplifies exposing services running in the ephemeral K3S cluster to the internet (where Cypress can access them) without requiring complex IP address management or DNS configuration. ngrok provides a stable public URL that tunnels directly to the service.
2. Application Deployment to Test Cluster: The now fully rendered Kubernetes manifests (containing the SHA-tagged images and ngrok URL) are applied to the ephemeral K3S cluster.
3. E2E Test Execution: A dedicated test container, pre-configured with Cypress, is spun up. This container executes the Cypress tests against the deployed application, using the ngrok-provided frontend URL.
4. GitOps - Rendered Manifest Commit: If all tests pass, the GitHub Action takes the rendered Kubernetes manifests (the ones with the actual SHAs and ngrok URL) and commits them to dedicated, non-merging branches within the config repo (e.g., render-test, render-prod). This is a crucial aspect of the rendered manifest pattern: the exact configuration that passed tests is now version-controlled in Git, ready for deployment. The main branch of the config repo retains the templated manifests.

Finally, Argo CD is introduced as the continuous deployment tool. Argo CD is configured to watch the render-prod branch of the config repo. When changes are committed to this branch (which only happens after successful CI/E2E tests), Argo CD automatically detects the drift from the desired state and synchronizes the production Kubernetes cluster to match the committed, fully rendered manifests. This ensures that only validated configurations are deployed to production, with a complete audit trail in Git.

McAllister also briefly introduces vCluster as an advanced solution for scaling this testing approach to larger, more complex applications. vCluster allows running "virtual Kubernetes clusters" inside a host Kubernetes cluster. This enables teams to spin up lightweight, ephemeral vClusters for each pull request, inheriting CRDs and other configurations from the host cluster, providing a near-production testing environment without the overhead of full, dedicated clusters.

Demo / Proof of Concept

▶ Watch: Introducing Cypress for automated end-to-end testing. (6:00)

The practical demonstration was a cornerstone of the talk, illustrating the entire CI/CD and E2E testing workflow using a simple "hamburger joints" web application. This application, designed to track burger places and allow friends to vote on them, featured a basic UI with a list of entries and an "Add" button.

The demonstration unfolded through a recorded video walkthrough of a GitHub Actions pipeline:

1. Application Change: McAllister initiated the demo by making a minor functional change to the application's frontend. He modified the title from "Burger Joints List" to "Burger Places of Seattle," reflecting a more specific scope for his personal tracker.
2. Pull Request and CI Trigger: This code change was committed and a pull request opened, automatically triggering the GitHub Actions workflow.
3. CI Pipeline Execution: The video showcased the GitHub Action spinning up a K3S cluster, building Docker images for the frontend and backend (tagged with the commit SHA), and pushing them to DockerHub.
4. Configuration Rendering: The action then checked out the separate config repo, used the sed command to inject the new image SHAs and an ngrok-generated URL into the Kubernetes manifests, effectively creating a fully rendered manifest.
5. E2E Test Run: The core of the demo involved the execution of the Cypress E2E tests against the application deployed in the ephemeral K3S cluster via the ngrok URL. The tests successfully validated the application's functionality, including the ability to load the page, locate the "Add" button, click it, fill out a form, save, and verify the new entry. McAllister also highlighted Cypress's ability to provide screenshots upon test failure, aiding in quick debugging.
6. Rendered Manifest Commit: Upon successful completion of all tests, the GitHub Action committed the fully rendered Kubernetes manifests (now containing the updated image SHAs and configuration) to a dedicated render-prod branch in the config repo.
7. Production Deployment with Argo CD: Finally, the demonstration shifted to Argo CD, which was configured to watch the render-prod branch. Argo CD detected the new commit, automatically synced the changes, and deployed the updated configuration to the production cluster. The visual representation of Argo CD's sync status and the subsequent refresh of the live production website clearly showed the title change ("Burger Places of Seattle") taking effect.

A minor caveat was noted regarding the demo application's use of local storage, meaning data would be reset with new deployments. In a real-world production scenario, persistent storage would be crucial to avoid data loss. Despite this, the demo effectively illustrated a complete, automated, and auditable workflow for deploying changes with high confidence.

Defensive Implications

▶ Watch: Diving into the E2E test code structure. (7:00)

The methodologies presented in Scott McAllister's talk, while primarily focused on development efficiency and reliability, carry significant defensive implications for security and operational resilience in cloud-native environments.

1. Enhanced Reliability and Stability: By implementing robust end-to-end testing in a production-like environment, organizations significantly reduce the likelihood of deploying bugs or regressions to production. This "shift-left" approach catches issues early, preventing outages, data corruption, or unexpected behavior that could be exploited by attackers or lead to operational instability. The talk demonstrates how Cypress can validate user flows, ensuring critical application paths remain functional and secure.

1. Consistent and Secure Environments: The emphasis on making test environments "as close to production as possible" directly contributes to security. Inconsistent environments often hide configuration drift or vulnerabilities that are only exposed in a live setting. By using tools like K3S for ephemeral clusters and ngrok for secure, temporary ingress, the testing environment more accurately reflects the production attack surface. This helps identify misconfigurations or security flaws (e.g., exposed endpoints, incorrect access controls) before they reach production.

1. Auditability and Traceability with GitOps: The adoption of GitOps with the rendered manifest pattern is a powerful defensive mechanism.

• Immutability: Every change to infrastructure or application configuration is a Git commit, creating an immutable, version-controlled history. This is invaluable for forensic analysis after a security incident, allowing teams to pinpoint exactly what changed, when, and by whom.
• Rollback Capabilities: In the event of a security vulnerability or critical bug discovered post-deployment, the ability to quickly and reliably roll back to a known good state via Git is a significant defensive advantage, minimizing the window of exposure.
• Compliance: For regulated industries, the comprehensive audit trail provided by GitOps is essential for demonstrating compliance with security policies and controls.

1. Reduced Manual Error and Human Factor Risk: Automating the build, test, and deployment processes through GitHub Actions and Argo CD drastically reduces the potential for human error. Manual deployments are prone to mistakes, such as deploying the wrong version, misconfiguring services, or overlooking security settings. Automation ensures that configurations are applied consistently and precisely as defined in Git, mitigating a common source of security incidents.

1. Shift-Left Security Validation: The pipeline enables security checks to be integrated earlier in the development lifecycle. While not explicitly covered in the talk, the framework allows for:

• Static Application Security Testing (SAST) and Dynamic Application Security Testing (DAST) tools to be run against the application in the ephemeral test environment.
• Container image scanning to be integrated during the Docker build phase, catching vulnerabilities in dependencies before deployment.
• Configuration validation for Kubernetes manifests to ensure they adhere to security best practices (e.g., no privileged containers, appropriate network policies).

1. Proactive Monitoring with Smoke Screening: The concept of "smoke screening" (running additional tests on the deployed application, even in a test environment) after deployment, is an excellent practice for proactive defense. It ensures that the application remains functional and secure even after external changes or dependencies are updated.

By embracing these practices, organizations can build a more resilient and secure software delivery pipeline, reducing their attack surface, improving their response capabilities, and ensuring higher confidence in their deployments.

Key Takeaways

• End-to-End Testing is Crucial for User Confidence: E2E testing, simulating full user flows, is indispensable for verifying application functionality across the entire stack and ensuring a positive user experience.
• Production-Like Test Environments are Essential: To prevent "it works on my machine" issues, test environments must closely mirror production, encompassing application code, dependencies, and surrounding Kubernetes configurations.
• GitOps Provides Unparalleled Auditability and Control: Leveraging Git as the single source of truth for both application and infrastructure code, especially with the rendered manifest pattern, ensures every deployed configuration is version-controlled, auditable, and easily reversible.
• Automated CI/CD Pipelines Drive Speed and Reliability: Tools like GitHub Actions, K3S, Cypress, ngrok, and Argo CD can be integrated to create a powerful, automated pipeline for building, testing, and deploying cloud-native applications rapidly and with high confidence.
• Ephemeral Clusters Solve Complex Testing Challenges: For intricate, CRD-heavy applications, solutions like vCluster enable the creation of lightweight, ephemeral Kubernetes clusters within a host cluster, providing a scalable and efficient way to achieve production-grade test environments.
• Proactive Testing and GitOps Enhance Security: These practices contribute significantly to a strong defensive posture by reducing human error, providing clear audit trails, enabling quick rollbacks, and catching potential issues earlier in the development lifecycle.

About the Speaker(s)

Scott McAllister is a Developer Advocate passionate about helping developers build and deploy robust applications. At the time of this KubeCon EU talk, he was serving as a Developer Advocate at ngrok, a company known for its secure ingress solutions. Shortly after the talk, Scott joined Loft Labs, the creators of vCluster, further demonstrating his commitment to advancing cloud-native development and testing paradigms. Throughout his presentation, McAllister showcased his visual learning style, often relying on diagrams and code examples to convey complex ideas. His personal anecdotes, such as his love for hamburgers, added a relatable touch to the technical discussion. Scott's expertise lies in guiding teams through the intricacies of CI/CD, testing, and GitOps, making him a valuable voice in the cloud-native community.