Surviving Day2 : Picking the Right Tool To Secure Your... Bruno Gabriel da Silva & Henrique Santana

Surviving Day2: Picking the Right Tool To Secure Your Kubernetes Habitat

Speakers: Bruno Gabriel da Silva, Senior Solutions Engineer at CYIC; Henrique Santana, Principal Cloud Support Engineer at AWS

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=FqUPqroF-Rw

Overview

In the rapidly evolving landscape of cloud-native security, the sheer volume of available tools can be overwhelming. Henrique Santana and Bruno Gabriel da Silva addressed this challenge head-on in their KubeCon EU talk, "Surviving Day2: Picking the Right Tool To Secure Your Kubernetes Habitat." The speakers highlighted that the Cloud Native Computing Foundation (CNCF) ecosystem alone boasts over 80 security projects, a number that continues to grow, making it incredibly difficult for practitioners to identify and implement the most appropriate solutions for their specific needs.

This presentation offered a pragmatic, lifecycle-oriented approach to securing Kubernetes environments. Santana and Silva meticulously broke down the Kubernetes application lifecycle into four distinct stages – Build, Deploy, Start, and Run – and then mapped prominent open-source security tools to each phase. Their core message was about demystifying the complex array of tools and providing a clear framework for integrating security throughout the entire development and operational pipeline, emphasizing that different tools are designed for different types of threats and stages.

The talk is crucial for anyone navigating Kubernetes security, from platform engineers and security architects to developers. It provides a foundational understanding of where and why specific security measures are essential, moving beyond the superficial "scan my containers" mentality to a deep dive into image integrity, policy enforcement, secure secret management, and critical runtime threat detection. By correlating technology with real-world analogies and practical examples, the speakers delivered an analytical yet accessible guide to building a robust Kubernetes security posture.

Background

▶ Watch: Introduction: The overwhelming number of CNCF security projects (0:00)

The proliferation of open-source projects within the CNCF ecosystem, while a testament to community-driven innovation, has inadvertently created a significant challenge for organizations: choice paralysis. With over 80 security projects available, identifying the right tool for a specific security problem in Kubernetes can feel like an insurmountable task. Many practitioners struggle with understanding the nuanced differences between tools, their optimal application points, and how they integrate into a cohesive security strategy. This often leads to either under-securing environments, implementing redundant solutions, or misapplying tools to problems they were not designed to solve.

A common misconception, as highlighted by the speakers, revolves around the terminology of "scanning containers." Many users express a desire to "scan their containers for vulnerabilities," when in reality, they intend to scan their container images. This distinction is critical because a container image is a static, immutable artifact composed of read-only layers, akin to a recipe for a cake. It's built once and contains the base operating system, configurations, dependencies, and the application itself. In contrast, a running container introduces a runtime layer, which is read-write and exists only for the duration the container is active. Understanding this fundamental difference dictates which security tools are appropriate for each phase of the application lifecycle.

To address this complexity and guide security practitioners, the talk structured Kubernetes security around four key stages:

1. Build: Focusing on the creation of secure container images.
2. Deploy: Ensuring that only compliant configurations and applications are allowed into the cluster.
3. Start: Managing sensitive information securely as applications prepare to run.
4. Run: Detecting and responding to threats within live, operational workloads.

This phased approach provides a much-needed framework for evaluating and integrating security tools, moving beyond ad-hoc solutions to a comprehensive, lifecycle-driven security strategy.

Key Findings

▶ Watch: Four key security stages: Build, Deploy, Start, Run (3:40)

The central finding of this talk is the imperative for a structured, multi-layered approach to Kubernetes security, meticulously aligned with the application lifecycle. The speakers effectively demonstrated that no single tool can provide comprehensive security; instead, a combination of specialized open-source projects, each excelling in a particular phase, is required.

Their key contributions include:

• Demystifying the CNCF Security Landscape: By categorizing and explaining the purpose of specific tools within distinct lifecycle stages, the talk provides a much-needed clarity amidst the overwhelming number of available projects.
• Highlighting Critical Distinctions: The emphasis on differentiating between container images (static, read-only) and running containers (dynamic, read-write runtime layer) is fundamental to correctly applying security measures.
• A Practical Framework for Tool Selection: The four-stage model (Build, Deploy, Start, Run) offers a clear mental model for practitioners to assess their current security posture and identify gaps, enabling informed decisions about tool adoption.
• Showcasing Essential Open-Source Tools: The selection of Trivy, Kyverno, External Secrets Operator, and Falco as exemplary tools for each stage provides actionable recommendations, backed by their prevalence and effectiveness in the community.
• Underscoring the Necessity of Runtime Security: Through the real-world example of the NGINX Ingress Controller vulnerability, the speakers powerfully illustrated that even with robust pre-runtime security, threats can emerge during execution, making runtime detection indispensable.

Ultimately, the talk's key finding is that effective Kubernetes security requires strategic thinking, a deep understanding of the platform's lifecycle, and the deliberate integration of specialized tools at each critical juncture to create a resilient defense-in-depth strategy.

Technical Deep Dive

▶ Watch: Build phase: Clarifying container vs. container image misconception (4:00)

The core of the presentation involved a detailed technical examination of specific open-source tools tailored for each stage of the Kubernetes application lifecycle.

Build Phase: Securing Container Images with Trivy

The first stage, Build, focuses on ensuring the integrity and security of container images before they are deployed. The speakers clarified a crucial distinction: "scanning containers" actually refers to scanning container images. A container image is a static, immutable artifact, built in layers. Each layer represents a change, such as adding a base OS (e.g., Alpine), copying configurations, running installation commands (like apk add), or adding the application code. These layers are read-only. A running container, however, adds a new, read-write runtime layer on top of the image, which exists only while the container is active.

To inspect these static images for vulnerabilities, the recommended tool is Trivy. Trivy, a widely recognized open-source scanner, simplifies the process into three main steps:

1. Extract Layers: Trivy first extracts all the individual layers that compose the container image.
2. Build File System: It then reconstructs the file system based on these layers, mirroring how the image would appear.
3. Identify OS and Non-OS Packages: Trivy identifies all installed operating system packages (e.g., apk packages in Alpine) and language-specific packages (e.g., Python pip packages, Node.js npm packages).
4. Cross-Check with Vulnerability DB (VDB): Finally, Trivy takes this inventory of packages and cross-references it against various Vulnerability Databases (VDBs) to identify known CVEs (Common Vulnerabilities and Exposures).

The result is a clear, actionable report. For instance, an example output showed a Python 3.9 base image with zero OS vulnerabilities, but identified three specific vulnerabilities within Python packages, detailing the affected package. This allows developers and security teams to address issues at the earliest possible stage, shifting security left.

Deploy Phase: Enforcing Policies with Kyverno

The Deploy phase is where requests to modify the Kubernetes cluster (e.g., create a pod, deploy an application) are validated and potentially altered. This process is managed by Admission Controllers, which act as checkpoints in the Kubernetes API workflow. When a request is made, it passes through the admission controller, which inspects the request and enforces predefined policies before the object is stored in etcd.

For robust policy enforcement, the talk highlighted Kyverno. Kyverno functions as a Kubernetes admission controller and operates directly with YAML policies, which is a significant advantage as it means users don't need to learn a new policy language (like Rego for OPA Gatekeeper). Kyverno supports several types of policies, with two being most commonly used:

• Mutating Webhooks: These policies allow for on-the-fly modification of requests before they are persisted. For example, if a company policy dictates that all pods must have memory limits but a developer forgets to include them in a deployment definition, a Kyverno mutating policy can automatically inject a default memory limit. This prevents non-compliant deployments without blocking the request entirely.
• Validating Webhooks: These policies strictly enforce compliance by blocking any request that does not meet the defined criteria. An example given was a production cluster requiring a specific label on all new namespaces. If a user attempts to create a namespace without this label, Kyverno's validating policy will block the request, preventing non-compliant resources from being created.

The talk also briefly mentioned Generate and Clean Up policies, acknowledging their more advanced use cases. Kyverno's ability to define, apply, and manage policies using native Kubernetes YAML makes it an accessible and powerful tool for maintaining cluster hygiene and security best practices.

Start Phase: Secure Secret Management with External Secrets Operator

The Start phase addresses the secure handling of sensitive information just before an application begins running. While Kubernetes offers native Secrets objects to store credentials, tokens, keys, and certificates, these come with significant limitations. Default Kubernetes Secrets lack:

• Built-in Rotation Mechanisms: Changing passwords or keys stored in native secrets can be challenging and manual.
• Versioning: There's no easy way to track changes or roll back to previous versions of a secret out-of-the-box.
• Integration with External Vaults: Native Kubernetes secrets do not inherently integrate with dedicated external secret management systems like AWS Secrets Manager, HashiCorp Vault, or Azure Key Vault. This makes it difficult to centralize secret management, especially in hybrid environments or when integrating with GitOps workflows.

To overcome these limitations, the External Secrets Operator (ESO) is introduced. ESO acts as a bridge, fetching secret values from external secret management systems and injecting them into the Kubernetes cluster as native Kubernetes Secrets. It supports various providers, including AWS Secrets Manager, AWS Systems Manager Parameter Store, HashiCorp Vault, and others.

The process typically involves:

1. IAM Role Configuration: The External Secrets Operator is configured with an IAM role (or equivalent credentials for other cloud providers/vaults) that has the necessary permissions to retrieve secrets from the external vault.
2. Periodic Synchronization: ESO periodically queries the external secret store for the latest secret values.
3. Kubernetes Secret Creation/Update: It then creates or updates corresponding native Kubernetes Secrets within the cluster, making them available for pods to consume as environment variables or mounted volumes.

This approach brings the benefits of automatic secret rotation, versioning (managed by the external vault), and seamless integration with GitOps pipelines, as the Kubernetes Secret definition itself can be managed in Git, while the sensitive value resides securely in the external vault.

Run Phase: Real-time Threat Detection with Falco

Finally, the Run phase focuses on monitoring and detecting threats within live, running workloads. The speakers stressed that despite robust security in previous stages, runtime security is non-negotiable, citing the critical NGINX Ingress Controller vulnerability (dubbed the "ingress nightmare") as a prime example of a runtime exploit that pre-runtime scanning or policy enforcement could not catch.

Observing running containers traditionally involves instrumentation or SDKs, which can introduce overhead and require developer involvement. A more efficient and secure method involves monitoring system calls (syscalls) – the interface between user-space applications and the kernel. Every action an application takes, from reading a file to opening a network connection, involves syscalls.

Falco is the open-source solution championed for runtime security. Falco leverages eBPF (extended Berkeley Packet Filter), a powerful, lightweight, and safe technology that allows programs to run in the Linux kernel without modifying kernel source code or loading kernel modules (which can be risky and unstable). Falco uses eBPF to hook into the kernel and observe syscalls in real-time, effectively mirroring what applications inside pods are doing.

The challenge with syscall data is its sheer volume; a few seconds of activity can generate millions of syscalls. Falco addresses this with Falco rules. These rules are a logic layer that filters and interprets the massive stream of syscall data, transforming it into actionable security events. Examples of Falco rules include:

• Detecting when a process attempts to write to sensitive or unexpected paths, such as /tmp.
• Identifying if a shell is opened inside a container, which often indicates a compromise, whether through an application vulnerability or leaked kubectl credentials.

The NGINX Ingress Controller vulnerability perfectly illustrated Falco's value. This exploit was not a package vulnerability (so Trivy wouldn't find it) nor a misconfiguration of a Kubernetes object (so Kyverno wouldn't block it). Instead, it was a runtime behavior within the NGINX configuration that allowed access to all Kubernetes secrets. Only a runtime security tool like Falco could detect such an anomalous behavior, alerting defenders to a critical breach. Falco, an Italian word for falcon, aptly represents its ability to precisely identify and react to specific threats within the vast operational landscape.

Demo / Proof of Concept

▶ Watch: Trivy results and raccoon analogy for image inspection (7:30)

While the presentation did not feature a live, interactive demonstration in the traditional sense, the speakers effectively illustrated the capabilities of each tool through compelling examples and real-world scenarios.

For Trivy, a clear screenshot of its output was provided, showcasing how it identifies vulnerabilities within Python packages, even when the base OS image is clean. This visual evidence underscored the tool's ability to pinpoint specific vulnerable components within a container image, complete with package names and vulnerability counts.

The power of Kyverno was demonstrated through hypothetical yet common use cases: a developer forgetting to set memory limits (handled by a mutating policy) and a production cluster requiring specific namespace labels (enforced by a validating policy). These scenarios vividly explained how Kyverno prevents misconfigurations and enforces compliance without requiring a complex new policy language.

The functionality of the External Secrets Operator was explained using the practical example of AWS Secrets Manager. The speakers detailed how ESO, configured with an IAM role, fetches and synchronizes secrets into Kubernetes, addressing the shortcomings of native Kubernetes Secrets. This explanation served as a conceptual proof of concept for integrating external secret management.

The most impactful "proof of concept" for Falco was the detailed discussion of the NGINX Ingress Controller vulnerability. This real-world "ingress nightmare" served as a powerful case study, demonstrating precisely why runtime security is indispensable. The speakers meticulously walked through why pre-runtime tools like Trivy and Kyverno would have failed to detect this specific exploit, which involved malicious configuration leading to runtime access to all Kubernetes secrets. Falco's ability to detect anomalous syscalls, such as unauthorized shell access or writing to /tmp, was presented as the definitive countermeasure for such runtime-specific threats. This example was a compelling argument for the necessity of a runtime security layer, highlighting its role in catching what other security controls miss.

Defensive Implications

▶ Watch: Transitioning to the Deploy phase: Hitting Kubernetes API (8:00)

The comprehensive, lifecycle-based approach presented by Santana and Silva offers several critical implications for defenders seeking to secure their Kubernetes environments:

1. Shift Left with Image Scanning: Defenders must prioritize scanning container images for vulnerabilities early in the CI/CD pipeline, ideally at the build stage. Tools like Trivy are essential for identifying known CVEs in OS packages and application dependencies before deployment, significantly reducing the attack surface. It is crucial to remember that scanning a running container is fundamentally different from scanning its static image.

1. Enforce Policies at Admission: Implement strong policy enforcement at the Kubernetes admission control layer. Tools like Kyverno enable organizations to define granular policies using native YAML, preventing non-compliant or insecure configurations from ever entering the cluster. This includes validating resource definitions, mutating them to inject security best practices (e.g., resource limits, security contexts), and ensuring proper labeling. This acts as a critical gatekeeper for cluster integrity.

1. Centralize and Automate Secret Management: Abandon the use of native Kubernetes Secrets for sensitive, long-lived credentials. Instead, leverage external secret management solutions (e.g., AWS Secrets Manager, HashiCorp Vault) integrated with the cluster via tools like the External Secrets Operator. This ensures automated secret rotation, versioning, centralized control, and secure sharing across diverse cloud and on-premises environments, reducing the risk of hardcoded or exposed secrets.

1. Prioritize Runtime Threat Detection: Recognize that even with robust pre-deployment security, runtime threats can emerge from zero-day exploits, misconfigurations, or supply chain attacks. Implement a powerful runtime security solution like Falco, utilizing eBPF to monitor system calls and detect anomalous behavior in real-time. This is the last line of defense against active attacks, providing visibility into critical events like unauthorized shell access, suspicious file modifications, or unexpected network connections within containers.

1. Adopt a Layered Security Strategy: Understand that security is not a single tool but a continuous process involving multiple layers of defense. Each stage (Build, Deploy, Start, Run) requires specific security controls that complement each other. Defenders should avoid relying on a single project for all their security needs and instead build a comprehensive security architecture that integrates different tools across the entire application lifecycle.

1. Educate and Empower Developers: The complexity of Kubernetes security necessitates collaboration. Defenders should work to educate developers on secure coding practices, the importance of image hygiene, and how policies are enforced. Providing accessible tools and clear guidelines can empower developers to build security in from the start, rather than treating it as an afterthought.

By adopting these defensive implications, organizations can move beyond reactive security measures to proactive, integrated security practices that significantly enhance the resilience of their Kubernetes deployments.

Key Takeaways

• Image Scanning is Distinct from Container Scanning: Always remember to scan container images (static, read-only layers) for vulnerabilities during the build phase, as this is fundamentally different from observing a running container.
• Enforce Policies with Admission Controllers: Utilize tools like Kyverno to enforce security policies and best practices at the Kubernetes API admission level, preventing non-compliant configurations from being deployed to your cluster.
• Leverage External Secret Vaults: For robust secret management, integrate External Secrets Operator with dedicated external vaults (e.g., AWS Secrets Manager, HashiCorp Vault) to ensure automatic rotation, versioning, and secure distribution of sensitive data, rather than relying solely on native Kubernetes Secrets.
• Runtime Security is Non-Negotiable: Implement a runtime security solution like Falco to detect active threats and anomalous behavior within running containers using eBPF and system call monitoring, as pre-runtime controls cannot catch all exploits.
• Adopt a Lifecycle Approach: Kubernetes security requires a multi-layered strategy across the entire application lifecycle – Build, Deploy, Start, and Run – with specific tools designed for each stage.
• Simplify Complex Security Concepts: The talk highlighted the need for beginner-friendly content to bridge the gap between extensive tool documentation and practical, staged implementation.

About the Speaker(s)

Henrique Santana is a Principal Cloud Support Engineer at AWS. Based in Dublin, Ireland, Henrique brings extensive experience in cloud-native technologies, having worked with customers on Amazon ECS and Amazon EKS for nearly six years. His deep understanding of Kubernetes operations and challenges provides valuable insights into real-world security needs. Henrique is originally from Brazil.

Bruno Gabriel da Silva is a Senior Solutions Engineer at CYIC, also based in Dublin, Ireland, and originally from Brazil. Prior to his work with Kubernetes and open-source technologies, Bruno gained significant experience with traditional enterprise technologies. His journey from legacy systems to the cloud-native world gives him a unique perspective on the challenges and benefits of modern security approaches.