Spotter - Universal Kubernetes Security Engine

Spotter - Universal Kubernetes Security Engine

Speakers: Madhu Akula

Conference: Cloud Village

YouTube: https://www.youtube.com/watch?v=A1JZSSVYJIg

Overview

Madhu Akula's talk at Cloud Village introduced Spotter, an innovative open-source tool designed to serve as a universal security engine for Kubernetes environments. Spotter aims to bridge the significant gap between security engineering teams and day-to-day platform or development engineers by offering a unified approach to defining and enforcing security policies across the entire Kubernetes lifecycle. This tool addresses the inherent complexities and rapid evolution of the Kubernetes ecosystem, providing a consistent method for identifying and mitigating misconfigurations, vulnerabilities, and compliance issues.

The core premise of Spotter is to empower organizations to write security policies once and apply them everywhere – from CLI manifest scanning and Helm charts to CI/CD pipelines and live admission controllers. By leveraging the Common Expression Language (CEL) and a highly extensible rule engine, Spotter seeks to simplify security enforcement, making it more accessible and manageable for teams struggling to keep pace with Kubernetes' rapid development cycles and vast landscape of cloud-native projects. The release of Spotter at major security conferences like Defcon and Black Hat underscores its potential impact on improving Kubernetes security posture through practical, policy-driven automation.

Background

▶ Watch: Introduction to Spotter: Universal Kubernetes Security Engine (0:00)

The landscape of Kubernetes security presents unique and substantial challenges that traditional security models often fail to address effectively. As Madhu Akula highlighted, Kubernetes itself evolves at an astonishing pace, releasing three to four new versions annually. This rapid development makes it incredibly difficult for both security professionals and even core engineers to keep up with changes, let alone implement robust security measures that remain current. Furthermore, Kubernetes is merely a small component within the vast and ever-expanding Cloud Native Computing Foundation (CNCF) ecosystem (as seen on landscape.cncf.io), where organizations often leverage dozens of interconnected projects, each introducing its own security considerations.

A fundamental issue is the shift in how security incidents are handled. In a Kubernetes environment, simply deleting a compromised pod or node doesn't resolve the underlying issue; the system's inherent self-healing capabilities will automatically provision new, potentially vulnerable instances. This necessitates a proactive, policy-driven approach rather than reactive incident response. Security vulnerabilities can arise from multiple layers, not just the Kubernetes infrastructure itself. Supply chain attacks, for instance, can introduce backdoor images that grant internal cluster access, leading to privilege escalation and full cluster takeovers. Fixing security at one layer does not guarantee overall security, demanding a holistic, multi-layered strategy.

Recognizing the difficulty in learning and applying Kubernetes security, Akula previously developed Kubernetes Goat, an open-source project that serves as a vulnerable-by-design playground. Kubernetes Goat offers over 22 well-documented scenarios, ranging from basic attacks to cluster takeovers, and integrates with detection engineering tools like Falco and Tetragon, as well as policy engines like Kyverno. Widely adopted by security researchers and product vendors (including Microsoft Defender and Palo Alto), Kubernetes Goat helps teams understand and test their security controls against real-world attack vectors. Spotter emerges as the complementary tool designed to fix these identified security issues programmatically, bridging the gap between learning about vulnerabilities and implementing effective defenses.

Key Findings

▶ Watch: Navigating the Vast CNCF Cloud Native Landscape (2:48)

Spotter distinguishes itself as a universal Kubernetes security engine by addressing several critical pain points in cloud-native security. Its primary contribution is the introduction of a unified policy definition mechanism that allows security rules to be written once and applied consistently across the entire Kubernetes development and deployment lifecycle. This eliminates the need for disparate tools and policy languages for different stages, streamlining security operations.

The engine's core capabilities revolve around its ability to detect a broad spectrum of security issues. It effectively identifies misconfigurations within Kubernetes manifests and live clusters, pinpoints vulnerabilities that could be exploited, and ensures compliance with industry standards such as CIS Benchmarks, NIST, and NSA hardening guidelines. By mapping findings back to these recognized standards, Spotter provides actionable intelligence for organizations striving to meet regulatory and best-practice requirements.

A significant finding and architectural choice is Spotter's adoption of the Common Expression Language (CEL) for defining its security policies. This is a strategic move, as CEL is natively integrated into Kubernetes validation admission controllers, allowing organizations to reuse their Spotter-defined policies directly within the cluster without requiring Spotter itself to be the enforcement mechanism in every scenario. This future-proofs policy investments and simplifies migration paths.

Spotter consolidates and provides access to an extensive library of over 140 built-in security rules, covering various categories from workload security and access control to network traffic, configuration, resource sizing, and platform security. These rules are baked directly into the Spotter binary, making it a powerful out-of-the-box solution. Furthermore, Spotter's extensible rule engine allows for significant customization, enabling users to define custom API groups, include/exclude specific namespaces or resources, and even integrate their own custom rules tailored to unique organizational requirements. This flexibility ensures that Spotter can adapt to diverse and evolving Kubernetes environments, providing a comprehensive and adaptable security scanning solution.

Technical Deep Dive

▶ Watch: Learning Kubernetes Security: Introducing Kubernetes Goat (4:10)

Spotter's architecture is built around a philosophy of simplicity, extensibility, and reusability, primarily focusing on the configuration layer of Kubernetes. At its heart, Spotter defines security rules using a YAML-based specification, familiar to anyone working with Kubernetes or Infrastructure as Code (IaC). This YAML structure encapsulates the metadata and logic for each security check.

The true power of Spotter's policy definition lies in its integration of the Common Expression Language (CEL). CEL is a lightweight, open-source expression language designed for evaluating policies and conditions. Within Spotter, CEL expressions are embedded directly into the YAML rules to define the specific conditions that constitute a security violation. For example, a CEL expression might check if a container is running as root, if a security context is missing, or if a specific image tag is being used. This choice is particularly strategic because CEL is now natively integrated into Kubernetes validation admission controllers (which is Generally Available). This means that policies written for Spotter can be directly applied as validation webhooks within a Kubernetes cluster, ensuring consistency and avoiding the need to rewrite rules in different languages (like OPA/Rego, which can become complex) when shifting between pre-deployment scanning and runtime enforcement.

The rule engine within Spotter provides significant extensibility. Users can define custom API groups, allowing Spotter to scan not only native Kubernetes resources but also custom resources (CRDs) and operators specific to an organization's deployment (e.g., Cilium, Istio, or proprietary CRDs). The engine also supports granular control over scanning scope through include/exclude mechanisms for namespaces and resource types. This is crucial for managing scan noise and focusing on high-priority areas. For instance, an organization can choose to ignore findings in specific "sandbox" namespaces or temporarily disable certain checks for resources added to a "risk registry," with rules being version-controlled to re-enable checks as issues are addressed. Custom labels can also be used to tag and manage specific resources for exclusion or inclusion.

Spotter offers a comprehensive set of scanning capabilities across the Kubernetes lifecycle:

• Manifest Scanning: Directly analyzes YAML manifests or Helm charts before deployment, enabling shift-left security.
• Live Cluster Scanning: Connects to a running Kubernetes cluster and evaluates its current state against defined policies, identifying misconfigurations and vulnerabilities in deployed resources.
• Admission Controller: Can be deployed as a validation webhook within the Kubernetes API server request flow. This allows Spotter to intercept API requests for resource creation, update, or deletion, and prevent insecure configurations from being deployed in real-time.

Each rule in Spotter's YAML definition includes metadata such as:

• Severity: Mapping to CVSS scoring (Critical, High, Medium, Low).
• Categories: Grouping rules by type of issue (e.g., Workload Security, Access Control).
• Industry Standard Mappings: Linking rules to compliance frameworks like CIS, NIST, or NSA hardening guidelines, providing context and aiding compliance reporting.

Spotter provides various output formats including human-readable tables, JSON for machine processing, and SARIF (Static Analysis Results Interchange Format) for integration with CI/CD tools, vulnerability management platforms, and SIEM systems. The tool also consolidates rules from other industry tools like KICS and Kubescape, offering over 140 built-in rules directly accessible upon downloading the binary. This consolidation simplifies the security toolkit for Kubernetes users, providing a single, powerful engine for diverse scanning needs. Future plans include integration with KubeGPT for AI-driven insights and a completely open rule registry.

Demo / Proof of Concept

▶ Watch: Spotter's CEL Integration and OPA/Rego Comparison (5:40)

Madhu Akula provided a comprehensive demonstration of Spotter's capabilities, showcasing its versatility across different stages of the Kubernetes lifecycle.

The initial demonstration focused on manifest scanning. Akula used a simple scenario from Kubernetes Goat, a project specifically designed with intentional vulnerabilities. By running spotter scan manifest <path_to_kubernetes_goat_manifests>, Spotter executed its 142 built-in rules against the YAML files. The output clearly categorized findings by severity (Critical, High, Medium) and resource type (e.g., Pods, Deployments, Secrets), indicating potential security gaps. The demo highlighted the ability to use a --verbose flag for detailed explanations of detected issues and suggested fixes, as well as outputting results in JSON format for integration with other security tools like vulnerability management systems or SIEMs.

Next, the demonstration moved to live cluster scanning. Akula had pre-deployed the vulnerable Kubernetes Goat applications into a live Kubernetes cluster. Executing spotter scan cluster allowed Spotter to dynamically inspect all running resources within the cluster. This showcased Spotter's ability to scan not just standard Kubernetes objects like Pods and Deployments, but also custom resources (CRDs) and network policies (e.g., Cilium configurations), underscoring its extensibility beyond native Kubernetes APIs. The scan revealed numerous vulnerabilities, as expected from the intentionally insecure Kubernetes Goat deployment. The speaker emphasized the flexibility to include/exclude namespaces, scan specific rules, and define thresholds for breaking CI/CD pipelines based on the number of violations.

A critical part of the demo illustrated Spotter's deployment as an admission controller. Akula explained the Kubernetes API server request flow: requests go through authentication, authorization (RBAC), mutation webhooks (which can modify requests, e.g., replacing Docker Hub images with private registries), schema validation, and finally, validation webhooks. Spotter functions as a validation webhook, intercepting requests before workloads are scheduled. The demo showed Spotter deployed as a deployment in the cluster, acting as a custom validation webhook. When an attempt was made to deploy a completely secure pod manifest (with proper security contexts, dropped capabilities, and non-root user), Spotter allowed it. However, when an intentionally insecure pod manifest was deployed, Spotter denied the creation, explicitly stating that "spotter has denied this because so and so rule has prevented this to deploy into the kubernetes." This demonstrated Spotter's ability to enforce policies in real-time, preventing insecure configurations from ever reaching the cluster.

Further CLI features were briefly shown, including spotter rules list to display all 142 built-in rules and spotter rules info <rule_id> to view the detailed YAML structure of a specific rule, including its CEL query. Finally, Akula introduced the WebAssembly (WASM) playground available at play.spotter.run. This browser-based environment allows users to test manifests and policies directly in their web browser, utilizing the same Spotter binary compiled to WASM, further highlighting the tool's versatility and accessibility.

Defensive Implications

▶ Watch: Extensible Rule Engine and Integration Capabilities (6:10)

Spotter offers significant defensive implications for organizations operating Kubernetes environments, primarily by enabling a proactive, policy-driven security posture.

1. Shift-Left Security Enforcement: By integrating Spotter into CI/CD pipelines for manifest and Helm chart scanning, defenders can catch misconfigurations and vulnerabilities early in the development lifecycle. This "shift-left" approach significantly reduces the cost and effort of remediation, as issues are identified before deployment to production. Organizations can configure their pipelines to fail builds or deployments if critical security policies are violated.

1. Real-time Prevention with Admission Control: Deploying Spotter as a validation admission controller provides a crucial layer of defense at runtime. This prevents insecure or non-compliant workloads from ever being deployed to the Kubernetes cluster, acting as a gatekeeper for all API requests. This capability is vital for maintaining a strong security baseline, especially in dynamic environments where developers might inadvertently introduce vulnerabilities.

1. Unified Policy Management and Consistency: Spotter's single policy definition language (YAML + CEL) ensures consistency across all stages – from development to production. This eliminates the complexity and potential for discrepancies that arise from using different tools and policy engines for static analysis, pre-deployment checks, and runtime enforcement. Security teams can define policies once and be confident they are applied uniformly, reducing policy drift and improving auditability.

1. Customization for Organizational Needs: The extensible rule engine allows organizations to define custom rules tailored to their specific compliance requirements, internal hardening guidelines, and proprietary Custom Resources (CRDs). This ensures that Spotter can adapt to unique enterprise environments, rather than being limited to generic checks. Defenders can also leverage the include/exclude mechanisms for namespaces and labels to manage exceptions and prioritize remediation efforts effectively.

1. Leveraging Native Kubernetes Capabilities: The adoption of CEL is a forward-thinking defensive strategy. Since CEL is natively supported by Kubernetes validation webhooks, organizations are not locked into a proprietary policy engine. This means that even if they choose not to use Spotter as their admission controller in the future, their investment in CEL-based policies remains reusable within the Kubernetes ecosystem, future-proofing their policy definitions.

1. Enhanced Visibility and Reporting: Spotter's support for various output formats, including SARIF and JSON, facilitates integration with existing security tooling. This allows defenders to feed scan results into vulnerability management systems, SIEMs, or dashboards, providing centralized visibility into their Kubernetes security posture and streamlining reporting and compliance efforts.

1. Bridging the Gap Between Teams: By providing a practical and understandable way to apply security principles, Spotter helps bridge the communication and workflow gap between security engineers and development/platform teams. Engineers can use Spotter to self-check their manifests and understand security requirements, fostering a culture of shared responsibility for security.

Key Takeaways

• Unified Policy Engine: Spotter provides a single, consistent way to define and enforce Kubernetes security policies across the entire development and deployment lifecycle, reducing complexity and tool sprawl.
• CEL-Powered Policies: Leveraging the Common Expression Language (CEL) ensures policy reusability and future-proofing, as CEL is natively integrated into Kubernetes validation admission controllers.
• Comprehensive Scanning: Spotter can scan Kubernetes manifests, Helm charts, and live clusters, offering both static analysis and dynamic assessment capabilities.
• Real-time Prevention: When deployed as an admission controller, Spotter can prevent insecure or non-compliant Kubernetes resources from being deployed, enforcing security policies proactively.
• Extensible and Customizable: With over 140 built-in rules and the ability to define custom rules, API groups, and include/exclude specific resources/namespaces, Spotter adapts to diverse organizational requirements and compliance standards.
• Open Source and Community-Driven: As an open-source project, Spotter encourages community contributions and aims to integrate with other CNCF projects, fostering collaborative security improvements.

About the Speaker(s)

Madhu Akula is a prominent security researcher and expert in Kubernetes security. He has been actively involved in the Kubernetes security space since 2017, consistently presenting his research at leading security conferences such as Black Hat, Defcon, and Usenix. Akula is the creator and maintainer of Kubernetes Goat, a widely recognized open-source project that serves as an intentionally vulnerable playground for learning and hacking Kubernetes clusters. This project is extensively used by security researchers, detection engineering teams, and even product vendors (including Microsoft Defender and Palo Alto) to test and validate their security solutions. With Spotter, Akula continues his mission to bridge the gap between security and engineering, providing practical open-source tools to address the complex challenges of cloud-native security.