Kubernetes Security Scanner

Kubernetes Security Scanner

Speakers: Krishna Priya

Conference: Cloud Village

YouTube: https://www.youtube.com/watch?v=A9b0ZwBklx8

Overview

In this Cloud Village talk, Krishna Priya introduces and demonstrates a custom-built, open-source Kubernetes security scanner. The presentation provides a step-by-step guide on deploying this tool to identify and remediate common vulnerabilities within Kubernetes clusters. This scanner is designed to help organizations secure their containerized environments by proactively detecting misconfigurations, insecure practices, and potential attack vectors.

The core of the talk revolves around a practical demonstration. Priya first deploys an intentionally vulnerable Kubernetes cluster, leveraging a pre-configured setup from Mua. She then uses her scanner to uncover a wide array of security flaws, ranging from privileged container deployments to unhardened network policies and secret management issues. The session culminates in showing how identified critical vulnerabilities can be fixed, followed by a re-scan to confirm their elimination, illustrating the scanner's effectiveness in a remediation workflow.

This talk is crucial for anyone involved in Kubernetes operations, security engineering, or cloud-native development. It highlights the persistent challenges in securing complex Kubernetes environments and offers a tangible, open-source solution to address these. By integrating multiple specialized security tools under a single orchestration layer, the scanner provides a comprehensive assessment capability, empowering defenders to identify weaknesses before they can be exploited by malicious actors.

Background

▶ Watch: Introduction to Kubernetes security scanner and talk plan (0:00)

The rapid adoption of Kubernetes as the de facto standard for container orchestration has simultaneously introduced a new frontier of security challenges. While Kubernetes offers powerful features for deploying and managing applications at scale, its complexity also presents numerous opportunities for misconfiguration and oversight, leading to significant security risks. Default settings are often not secure by design, and developers or operators may inadvertently introduce vulnerabilities through permissive configurations, unpatched images, or inadequate network segmentation.

Prior to comprehensive scanning tools, security assessments in Kubernetes often relied on a fragmented approach, involving manual checks or disparate single-purpose tools. This could lead to incomplete coverage, increased manual effort, and a slower remediation cycle. The need for a consolidated, automated solution that can perform a holistic security assessment across various layers of a Kubernetes cluster—from individual pods and containers to node-level configurations and network policies—became evident.

Existing solutions like kube-bench address CIS Kubernetes Benchmark compliance, and container image scanners like Trivy or Grype focus on software bill of materials (SBoM) and known vulnerabilities. However, orchestrating these tools and correlating their findings into a single, actionable report for a live Kubernetes cluster often requires significant custom scripting and integration work. The problem that Krishna Priya's Kubernetes Security Scanner addresses is precisely this: providing an integrated, open-source framework that combines the capabilities of various specialized tools to offer a comprehensive security posture assessment, simplifying the detection and remediation of vulnerabilities in Kubernetes environments.

Key Findings

▶ Watch: Initial scan report highlights critical Kubernetes vulnerabilities (10:00)

The Kubernetes Security Scanner developed by Krishna Priya is designed to uncover a broad spectrum of vulnerabilities and misconfigurations across a cluster. During the demonstration on an intentionally vulnerable cluster, the scanner highlighted several critical and high-severity issues, underscoring common pitfalls in Kubernetes deployments.

The primary categories of vulnerabilities identified include:

• Privileged Container Deployments: A significant finding was the presence of deployments running in privileged mode and containers executing as the root user. This is a critical security risk, as privileged containers have extensive capabilities on the host system, making them prime targets for attackers to achieve host compromise.
• Overly Permissive Permissions: The scanner detected instances of overly permissive role-based access control (RBAC) or general access configurations, which could allow unauthorized users or processes to perform actions beyond their intended scope.
• Network Access Issues: Lack of proper network segmentation was a common theme, with no network access restrictions defined and NodePort services exposed without adequate controls. This allows for broad network reachability, increasing the attack surface.
• Resource Limit Neglect: Many deployments lacked defined resource limits (CPU/memory), making them susceptible to resource exhaustion attacks that could degrade cluster performance or lead to denial of service.
• Insecure Secret Management: The scanner identified issues with sensitive information, such as secrets being defined in plain text or exposed as environment variables, which can be easily accessed and exploited.
• Missing Security Contexts: Several pods or containers were found to have missing or inadequately configured security contexts, leading to default, less secure settings, such as running as root.
• Supply Chain Vulnerabilities: Through SBoM analysis, the tool identified vulnerable packages within container images, highlighting risks introduced through the software supply chain.
• CIS Benchmark Failures: The scanner revealed non-compliance with the CIS Kubernetes Benchmark, indicating deviations from industry best practices for securing Kubernetes components.

The scanner's ability to consolidate these diverse findings into a single report, categorized by severity, represents its key contribution. It translates complex security assessment data into actionable insights, enabling operators to prioritize and address the most critical risks effectively.

Technical Deep Dive

▶ Watch: Detailed explanation of security scanning tools used (12:00)

The Kubernetes Security Scanner is an orchestration layer built on top of several well-established open-source security tools, designed to provide a comprehensive security assessment of Kubernetes clusters. It automates the execution of these tools and aggregates their findings into a unified report. The architecture leverages a modular approach, allowing for different types of checks to be performed by specialized components.

At its core, the scanner is a Python-based application that manages the lifecycle of various security checks. It first ensures the presence of prerequisite tools, installing them if necessary, before initiating scans. The key components and checks performed include:

1. CIS Benchmarks Checker:

• Tool Used: kube-bench.
• Functionality: This module checks the cluster's adherence to the CIS Kubernetes Benchmark. It verifies configurations of Kubernetes components (like API server, controller manager, kubelet) against best practices defined by the Center for Internet Security.
• Process: The scanner first verifies kube-bench installation. If not found, it installs it. Then, it executes kube-bench to run the benchmark tests, parsing the output to identify failures. These failures are then prioritized (e.g., critical, high) based on the benchmark's severity ratings and incorporated into the final report.

1. Container Security Scans:

• Tools Used: kube-sec and Trivy.
• Functionality: This section focuses on individual container and pod security configurations.
• Process: The scanner checks for kube-sec and Trivy installations. It then uses these tools to identify issues such as:
• Containers running in privileged mode.
• Containers running as the root user.
• Missing or improperly defined security contexts.
• Other general container misconfigurations that could lead to privilege escalation or container escape.

1. Software Bill of Materials (SBoM) and Vulnerability Scanning:

• Tools Used: Sift and Grype.
• Functionality: This module is critical for supply chain security, analyzing container images for known vulnerabilities.
• Process: When enabled (e.g., via an enable_sbomb flag), the scanner iterates through every image deployed in the cluster. For each image, it uses sift to generate an SBoM, listing all packages and dependencies within the image. This SBoM is then fed into grype, which scans for known vulnerabilities (CVEs) associated with those packages. The findings, including vulnerable packages and their severities, are then compiled. This process can be time-consuming due to the depth of analysis required for each image.

1. Runtime Security Checks:

• Tools Used: Giser (referred to as giser in transcript for runtime checks) and Falco.
• Functionality: This component provides real-time or near real-time monitoring for suspicious activities within the cluster.
• Process: Giser is utilized to check for runtime anomalies, such as privileged containers taking more system capabilities than required, or unusual process behavior. Falco is also integrated to run against its defined rules, detecting activities like file access in sensitive locations, execution of untrusted binaries, or potential crypto-mining operations. These tools help identify active threats or deviations from expected behavior.

1. Pod Security Checks:

• Functionality: This module specifically focuses on common pod-level security misconfigurations.
• Checks Performed:
• Containers running as root or in privileged mode.
• Undefined resource limits (CPU/memory).
• Presence of sensitive mount paths (e.g., Docker socket mounts, which can lead to container escape).
• Other potential risks classified under "cluster hardening" and "microservices hardening."

1. Network Policy Checks:

• Functionality: This module assesses the network security posture of the cluster.
• Checks Performed:
• Identification of missing network policies, indicating a lack of segmentation.
• Detection of overly permissive ingress or egress rules, which could allow unrestricted traffic flow.
• Verification of proper pod isolation to prevent lateral movement.

1. Node Security Checks:

• Functionality: This section examines the security of the underlying worker nodes.
• Checks Performed:
• Review of kernel parameters for hardened configurations.
• Assessment of runtime checks to ensure no privileged containers are exploiting excessive system capabilities.

The scanner is designed to be run with read-only access to the cluster, requiring only the ability to list pods, services, and networking resources, making it a relatively safe tool to deploy in production environments for assessment purposes. The output consolidates all findings, categorizing them by severity, enabling users to quickly identify and prioritize the most critical security issues.

Demo / Proof of Concept

▶ Watch: Confirming vulnerability resolution in the updated scan report (19:10)

The core of Krishna Priya's presentation was a live demonstration showcasing the practical application and efficacy of her Kubernetes Security Scanner. The demo followed a clear, three-phase approach: deploying a vulnerable cluster, running the scan to identify issues, and then remediating critical findings.

Phase 1: Deploying the Vulnerable Cluster

The demonstration began by setting up an intentionally vulnerable Kubernetes cluster. Priya utilized a "kind" cluster (Kubernetes in Docker) for this purpose, specifically referencing Mua's vulnerable Kubernetes cluster environment. The setup involved:

1. Unzipping provided files containing the cluster configuration and images.
2. Executing a script (run_kubernetes_go_cluster.sh) to load pre-configured vulnerable images and start the kind cluster. This process included loading images with known security flaws, such as containers designed to run in privileged mode or expose sensitive data.

Phase 2: Initial Scan and Vulnerability Identification

Once the vulnerable cluster was operational, Priya proceeded to run her Kubernetes Security Scanner. The command for running the basic scan was provided, allowing the audience to potentially follow along. The scanner executed its various checks, leveraging the integrated tools like kube-bench, kube-sec, Trivy, Sift, Grype, Giser, and Falco.

The output of the initial scan was displayed, clearly listing numerous vulnerabilities categorized by severity (Critical, High, Medium, Low). Key issues highlighted during this phase included:

• Critical: Several deployments were identified as running in privileged mode. Specifically, a "health check deployment" was called out for having its container running with elevated privileges.
• High: Containers running as the root user, overly permissive permissions, lack of network access restrictions, exposed NodePorts, and undefined resource limits were among the high-severity findings.
• Secrets: Issues related to sensitive keys being defined without proper protection were also flagged.

The demo emphasized that the report provides a clear overview of the cluster's security posture, making it easy to pinpoint critical areas needing immediate attention.

Phase 3: Remediation and Re-Scan

To illustrate the scanner's utility in a remediation workflow, Priya then focused on fixing one of the identified critical issues: the "health check deployment" running in privileged mode. She manually modified the Kubernetes deployment configuration (e.g., via kubectl edit deployment health-check-deployment) to remove the privileged: true setting or to enforce a non-root user.

After applying the fix and allowing the pod to restart with the updated configuration, Priya re-ran the Kubernetes Security Scanner. The subsequent report showed that the previously identified critical issue related to the privileged health check deployment had been eliminated. She repeated this for a second critical item, demonstrating how targeted fixes can effectively improve the security posture, with the scanner validating the remediation. This iterative process of scan, fix, and re-scan highlighted the tool's practical value in a continuous security improvement cycle.

The demonstration successfully proved that the scanner can effectively detect a wide range of common Kubernetes vulnerabilities and that its output is actionable, enabling operators to verify the success of their security remediation efforts.

Defensive Implications

▶ Watch: Explaining CIS benchmarks and container image scanning (24:00)

The Kubernetes Security Scanner provides invaluable insights for defenders aiming to secure their cloud-native environments. The findings and capabilities demonstrated in this talk offer several key defensive implications:

1. Proactive Vulnerability Identification: Defenders should integrate such a scanner into their CI/CD pipelines or as a regular, scheduled scan against their running clusters. This proactive approach helps identify misconfigurations and vulnerabilities early in the development lifecycle or before they can be exploited in production.
2. Prioritized Remediation: The scanner's ability to categorize findings by severity (Critical, High, Medium) is crucial. Defenders should prioritize addressing critical and high-severity issues first, such as privileged containers, containers running as root, and insecure secret management, as these often represent the easiest paths for attackers to gain control.
3. Harden Pod Security: Pay close attention to Pod Security Contexts. Ensure that containers do not run as root (runAsUser: 1000 or higher), are not in privileged mode (privileged: false), and do not have unnecessary capabilities. Implement Pod Security Standards (PSS) or Pod Security Policies (PSP, deprecated but still used) to enforce these configurations across the cluster.
4. Implement Robust Network Policies: The scanner highlights the risks of overly permissive network access. Defenders must implement granular Network Policies to enforce least privilege networking, isolating pods and controlling ingress/egress traffic between namespaces and to external services. Avoid exposing NodePorts directly to the internet without proper firewalling.
5. Define Resource Limits: Mitigate Denial of Service (DoS) risks by consistently defining resource requests and limits (CPU and memory) for all containers. This prevents a single misbehaving application from consuming all cluster resources.
6. Secure Secret Management: The scanner's detection of plaintext secrets or secrets exposed via environment variables underscores a critical weakness. Defenders should leverage Kubernetes Secrets only for storing sensitive data securely, preferably with encryption at rest, and use tools like external-secrets or integrate with cloud KMS solutions to manage secrets outside of Kubernetes manifests. Avoid embedding secrets directly in configuration files or container images.
7. Software Supply Chain Security: The integration of SBoM and vulnerability scanning (Sift and Grype) is vital. Defenders must scan all container images used in their deployments for known vulnerabilities. This should be an automated gate in the image build process, blocking images with critical vulnerabilities from being deployed. Regularly update base images and application dependencies.
8. Continuous Runtime Monitoring: Tools like Falco and Giser (for runtime checks) are essential for detecting suspicious activities that bypass static analysis. Defenders should configure robust runtime monitoring to alert on anomalous process execution, unauthorized file access, or attempts at privilege escalation within the cluster.
9. CIS Benchmark Compliance: Regularly running kube-bench via the scanner helps ensure compliance with industry best practices. Defenders should aim for high compliance, understanding that deviations often indicate security misconfigurations.
10. Education and Awareness: Many vulnerabilities stem from a lack of understanding. Defenders should educate developers and operators on secure Kubernetes practices, the importance of security contexts, network policies, and responsible secret management.

By adopting the practices highlighted by this scanner, organizations can significantly strengthen their Kubernetes security posture, reduce their attack surface, and build more resilient cloud-native applications.

Key Takeaways

• Comprehensive Security Assessment: The Kubernetes Security Scanner integrates multiple open-source tools (kube-bench, Trivy, Sift, Grype, Falco, kube-sec, Giser) to provide a holistic view of Kubernetes cluster security, covering configurations, images, runtime, and network.
• Identifies Critical Misconfigurations: The scanner effectively detects common yet critical vulnerabilities such as privileged containers, root user execution, overly permissive network policies, and insecure secret management, which are major attack vectors.
• Enables Prioritized Remediation: Findings are categorized by severity, allowing defenders to focus on the most impactful vulnerabilities first and track their remediation progress with subsequent scans.
• Supports Supply Chain Security: With SBoM generation (Sift) and vulnerability scanning (Grype), the tool helps identify vulnerable packages within container images, addressing critical supply chain risks.
• Actionable Insights for Defenders: The tool provides practical, actionable insights for hardening Kubernetes environments, emphasizing the need for robust Pod Security Contexts, granular Network Policies, defined Resource Limits, and secure Secret Management practices.
• Open-Source and Extensible: Being an open-source tool, it offers transparency and the potential for community contributions, allowing for future enhancements and adaptation to evolving threat landscapes.

About the Speaker(s)

Krishna Priya is the developer and speaker behind the open-source Kubernetes Security Scanner presented at Cloud Village. Her work focuses on practical security solutions for cloud-native environments, addressing common vulnerabilities and providing tools for enhanced defensive capabilities in Kubernetes.