Kubernetes Security Detection Engineering

Kubernetes Security Detection Engineering

Speakers: Madhu Akula, Creator of Kubernetes Goat

Conference: HITB Amsterdam

YouTube: https://www.youtube.com/watch?v=5mgCssxpk84

Overview

In this insightful talk from HITB Amsterdam, Madhu Akula, the renowned creator of Kubernetes Goat, delves into the intricate world of Kubernetes security detection engineering. The presentation provides a comprehensive journey through the evolving landscape of container and Kubernetes security, highlighting the critical differences from traditional security paradigms. Akula meticulously outlines various attack vectors, from initial discovery and remote code execution to sophisticated cluster takeovers and persistent backdoors, mapping them against the MITRE ATT&CK framework.

The core of Akula's discussion revolves around understanding the unique architectural components of Kubernetes and how attackers exploit these native features. He emphasizes that effective defense requires a deep comprehension of these systems, rather than simply applying traditional security concepts. The talk culminates in a demonstration of modern detection techniques, particularly leveraging eBPF-based runtime security tools, and introduces Kubernetes Goat as an invaluable resource for hands-on learning and practice in this complex domain.

This article aims to dissect Madhu Akula's presentation, providing a detailed technical analysis of the attack paths, the underlying Kubernetes concepts exploited, and the crucial defensive strategies required to protect these dynamic environments. It underscores the urgent need for security professionals to bridge the knowledge gap and adopt advanced detection and prevention mechanisms to counter the rapidly growing threat landscape in cloud-native ecosystems.

Background

▶ Watch: Introduction to Kubernetes security and core concepts (0:00)

The journey into modern application deployment began with Docker and the concept of containers. Madhu Akula traces this evolution, explaining that Docker emerged to solve the pervasive "it works on my machine" problem. By packaging an application with all its dependencies and libraries, containers ensure consistent execution across any environment, from a developer's laptop to a cloud server. This abstraction layer, built upon a container runtime, still leverages the underlying operating system's kernel resources, offering efficiency compared to traditional virtual machines which encapsulate an entire OS.

While Docker streamlined individual application deployment, managing thousands of containers presented a new challenge: orchestration. This is where Kubernetes stepped in. As a powerful orchestrator, Kubernetes automates the deployment, scaling, and management of containerized applications. Its popularity stems from its portability and extensive features for scaling, load balancing, and traffic management, significantly reducing engineering overhead.

Akula provides a high-level overview of the Kubernetes architecture, crucial for understanding its security landscape. Key components include:

• API Server: The central brain of the cluster, handling all communication and managing the state of the cluster. Every interaction, from users to internal components, goes through the API server.
• Control Manager: Maintains the desired state of the cluster, ensuring that components like replica sets are running as specified.
• Scheduler: Assigns workloads (pods) to available worker nodes based on resource requirements and constraints.
• etcd: A distributed key-value store that serves as Kubernetes' backing store for all cluster data, configurations, and states.
• Worker Nodes: Where the actual containerized applications (pods) run.

The security of Kubernetes presents unique challenges distinct from traditional infrastructure:

• Knowledge Gap: Security teams often lack a deep understanding of Kubernetes concepts, leading to misconfigurations and ineffective defense strategies. For instance, simply deleting a compromised pod might be futile if Kubernetes immediately spins up another.
• Rapid Growth: The Cloud Native Landscape is constantly evolving, with new tools and projects emerging daily. This makes it difficult for security teams to keep pace.
• Misconceptions: Many traditional security concepts don't directly translate. For example, namespaces are often mistakenly perceived as providing network isolation, but by default, they operate within a flat networking scheme.
• Increased Attack Surface: The distributed nature and numerous interacting components create a larger attack surface. Past incidents like Shopify's cluster hack via an SSRF vulnerability and early CTF (Capture The Flag) events exploiting default cluster admin privileges demonstrate the severity of these risks. The prevalence of crypto miners also highlights how easily compromised clusters can be leveraged for illicit activities.

Akula stresses that while terminology changes (e.g., VLANs become VPC peering, security groups become network security policies), the underlying security fundamentals remain. The critical step is to understand the new technology to effectively secure it.

Key Findings

▶ Watch: Demo: Initial access and container escape to host (10:40)

Madhu Akula's talk highlights several crucial findings regarding Kubernetes security:

• Attackers Exploit Native Kubernetes Features: The most effective and stealthy attacks don't rely on zero-days but rather on misconfigurations and a deep understanding of Kubernetes' built-in functionalities. Features like node ports, kubelet configurations, service accounts, node selectors, static pods, and cron jobs can be weaponized for initial access, privilege escalation, lateral movement, and persistence.
• The Kubernetes Kill Chain is Multi-Layered: A successful compromise typically involves a sequence of steps, moving from external reconnaissance to initial access, container escape, privilege escalation within the cluster, lateral movement across nodes, and ultimately achieving persistent cluster control and potential cloud provider compromise. This mirrors the traditional MITRE ATT&CK framework, adapted for the cloud-native context.
• Traditional Security Tools are Insufficient: Standard host-based or network-based security tools often fail to provide adequate visibility or real-time detection within the dynamic and ephemeral nature of containers and Kubernetes. The speed at which new pods can spin up means that slow, static monitoring systems are easily bypassed.
• Runtime Security is Paramount: Real-time detection and prevention at the syscall detection level, particularly leveraging eBPF-based technologies, are essential. Such tools can identify and even block malicious activities like container escapes or unauthorized process executions as they happen, significantly reducing the window of opportunity for attackers.
• Defense-in-Depth is Non-Negotiable: No single security control is a silver bullet. A layered approach encompassing secure configurations, strong RBAC (Role-Based Access Control), network policies, image scanning, and robust runtime security is necessary to create a resilient Kubernetes environment.
• Knowledge and Practice are Critical: The significant knowledge gap within security teams regarding Kubernetes architecture and operations is a major vulnerability. Hands-on learning environments, like Akula's Kubernetes Goat, are vital for practitioners to understand attack techniques and develop effective defensive strategies in a safe, practical setting. This includes understanding the nuances of how components like etcd store critical data, influencing persistence strategies.

Technical Deep Dive

▶ Watch: Lateral movement and full Kubernetes cluster takeover (19:00)

Madhu Akula provided a compelling demonstration of a full Kubernetes cluster takeover, illustrating how attackers chain together various techniques. The attack path unfolds as follows:

1. Initial Reconnaissance and Access (00:11:00):

• The attack begins with external reconnaissance, using tools like Nmap to scan for exposed services. A common misconfiguration in Kubernetes is the exposure of node ports. If an application is exposed via a node port, it bypasses traditional API gateways, firewalls, and WAFs, directly exposing the cluster.
• Assuming an initial Remote Code Execution (RCE) vulnerability in an exposed web application (a common starting point), the attacker gains a foothold.

1. Container Compromise and Reconnaissance (00:13:00):

• Once RCE is achieved, the attacker establishes a reverse shell into a compromised pod (container).
• Inside the container, reconnaissance begins:
• Identifying the operating system, kernel version (important for potential kernel exploits, as containers share the host kernel).
• Examining network interfaces and Cgroups to understand the container runtime environment (e.g., Docker, containerd).
• Crucially, observing processes. Akula highlights a critical finding: host system processes might be mounted into the container, giving the attacker visibility and potential interaction points with the host.
• Enumerating mount points to discover volumes mounted from the host system.
• Checking container capabilities. By default, containers have limited capabilities, but certain applications or monitoring tools require elevated capabilities like cap_sys_ptrace or cap_sys_admin. Attackers can leverage these.

1. Privilege Escalation: Container Escape to Host Node (00:16:00):

• Leveraging mounted volumes (e.g., /etc/host) and elevated capabilities like cap_sys_admin, the attacker performs a container escape using a classic chroot technique. By chrooting into a mounted host directory, the attacker effectively breaks out of the container's isolation and gains a shell on the underlying worker node. This is a significant privilege escalation.

1. Lateral Movement: Accessing the Kubernetes API Server (00:17:00):

• From the compromised node, the attacker targets the Kubernetes API Server. Kubernetes components, like kubelet (the agent that runs on each node and manages pods), need credentials to communicate with the API server.
• The attacker locates the kubelet configuration file, which contains credentials (often a kubeconfig file) allowing the kubelet to authenticate to the API server.
• By using this kubeconfig file, the attacker can now interact with the API server as the kubelet, gaining elevated privileges. Akula demonstrates this by running kubectl get pods using the kubelet's config, successfully listing pods.
• The kubectl auth can-i command is then used to enumerate the exact permissions granted by these stolen credentials, revealing the full extent of the attacker's capabilities.

1. Defense Evasion (00:18:00):

• To avoid detection, the attacker can delete pod logs (which are often forwarded to SIEMs), modify configurations, or employ other techniques to obscure their activities.

1. Lateral Movement: Node Hopping and Backdoor Deployment (00:19:00):

• The goal is to move beyond a single node to compromise other nodes, potentially containing more sensitive workloads.
• While older vulnerabilities like the "node admission plug-in" bug allowed easy node hopping, modern Kubernetes requires more sophisticated methods.
• The attacker retrieves node labels from other nodes.
• They then leverage a service account token (a JWT token typically mounted in /var/run/secrets/kubernetes.io/serviceaccount/token) that has permissions to create pods.
• Using the extracted token and the target node's label, the attacker crafts a pod definition that includes a node selector. This Kubernetes native feature allows administrators to specify which node a pod should be deployed on.
• The attacker uses curl to deploy a malicious "backdoor.json" pod onto the target node, effectively gaining a foothold on a new machine. This demonstrates how native features designed for legitimate purposes can be repurposed for malicious lateral movement.

1. Persistence: Static Pods, Cloud Credential Theft, and Cron Jobs (00:23:00):

• Static Pods: Akula shows how to deploy a static pod. These pods are managed directly by the kubelet on a specific node, not by the API server. This means even if the Kubernetes control plane is compromised or an administrator tries to delete the pod via kubectl, the kubelet will keep spinning it up, ensuring persistence at the node level.
• Cloud Provider Compromise (00:24:00): Many Kubernetes clusters run on cloud providers. Attackers can leverage access to metadata APIs (e.g., 169.254.169.254 for AWS) to steal cloud provider credentials. Using STS get caller identity (for AWS), the attacker can assume the role associated with the Kubernetes cluster, potentially gaining access to the underlying cloud infrastructure itself.
• Resource Consumption / DDoS (00:25:00): To impact services or for financial gain, an attacker might deploy a crypto miner or use tools like stress-ng to consume excessive CPU and memory, effectively performing a Distributed Denial of Service (DDoS) attack or resource exfiltration.
• Persistent Cron Jobs (00:26:00): The most robust persistence mechanism demonstrated is the use of Kubernetes native cron jobs. Unlike traditional Linux cron jobs, Kubernetes cron jobs are stored in etcd. This means that even if all nodes are deleted, the cluster is upgraded, or individual pods are removed, the cron job definition (and thus the backdoor) persists within the cluster's state, ensuring a continuous reverse shell to the attacker.

This comprehensive attack chain underscores the complexity and multi-faceted nature of Kubernetes security, where each layer of compromise opens doors to the next, ultimately leading to full cluster and potentially cloud environment control.

Demo / Proof of Concept

▶ Watch: Demo: Real-time container attack detection with eBPF (28:00)

Madhu Akula transitioned from demonstrating the offensive kill chain to showcasing a live proof-of-concept for real-time detection, emphasizing the need for modern defensive tooling.

The detection demo utilized a simple Kubernetes cluster and a scenario from Kubernetes Goat, Akula's vulnerable playground. The setup involved:

1. A running pod within the Kubernetes cluster.
2. Cilium Tetragon, an eBPF-based syscall detection tool, deployed on the host system to monitor activities.

Akula then initiated a simulated container escape attack, similar to the chroot technique demonstrated in the technical deep dive. As the attacker executed the escape command from within the container, Cilium Tetragon immediately detected the malicious syscalls associated with the container escape.

The key takeaway from this demo was the speed and granularity of eBPF-based detection. Unlike traditional logging or endpoint detection systems that might have delays or lack the necessary context, Tetragon provided real-time alerts on the suspicious system calls. Akula highlighted that not only can such tools detect these attacks, but they can also be configured with "tracing policies" to prevent specific malicious system calls from executing, effectively stopping the attack before it can establish itself or cause damage. This proactive prevention capability is crucial in highly dynamic environments where reaction time is measured in milliseconds.

The demo effectively illustrated that effective Kubernetes security requires moving beyond post-incident analysis to real-time, runtime-level threat detection and, ideally, prevention.

Defensive Implications

▶ Watch: Introducing Kubernetes Goat: a security learning playground (30:40)

The detailed attack demonstration by Madhu Akula underscores the critical need for a robust and multi-layered defense-in-depth strategy in Kubernetes environments. Relying on a single security control is insufficient, as attackers will always find ways to bypass isolated defenses. Defenders must adopt a comprehensive approach that addresses every stage of the attack kill chain.

Here are the key defensive implications:

1. Eliminate Initial Access Vectors:

• Secure Network Exposure: Strictly control external access to Kubernetes services. Avoid exposing applications directly via node ports unless absolutely necessary and with robust security controls. Utilize ingress controllers, API gateways, and proper firewall rules.
• Web Application Security: Implement rigorous application security testing to prevent common vulnerabilities like RCE and SSRF that serve as initial entry points.
• Regular Audits: Continuously audit cluster configurations for misconfigurations that can lead to initial access.

1. Harden Containers and Nodes:

• Principle of Least Privilege: Run containers with the absolute minimum necessary capabilities. Avoid granting cap_sys_admin or cap_sys_ptrace unless strictly required and justified.
• Secure Base Images: Use minimal, hardened base images and regularly scan them for vulnerabilities.
• Restrict Mount Points: Limit the mounting of host paths into containers, especially sensitive directories like /etc or /var/run/docker.sock.
• Process Monitoring: Monitor processes within containers for unusual activity, especially the presence of host processes or unexpected binaries.
• Kernel Security: Keep the host kernel patched and up-to-date to mitigate vulnerabilities that could lead to container escape.

1. Implement Robust Runtime Security:

• eBPF-based Tools: Deploy eBPF-based runtime security solutions like Cilium Tetragon or Falco. These tools provide deep visibility into system calls, allowing for real-time syscall detection and even prevention of malicious activities like container escape, unauthorized file access, or process execution.
• Real-time Monitoring: Integrate runtime security alerts into a centralized SIEM for rapid response. Static or slow monitoring systems are inadequate for the ephemeral nature of containers.

1. Secure Kubernetes API Server and Control Plane:

• Strong RBAC (Role-Based Access Control): Implement granular RBAC policies. Grant users, service accounts, and components only the permissions they absolutely need. Regularly review and audit RBAC configurations.
• Secure Kubelet Configuration: Ensure kubelet configurations are hardened and restrict access to kubeconfig files on nodes.
• Network Segmentation: While Kubernetes namespaces provide logical separation, they do not inherently offer network isolation. Implement network policies to enforce strict network segmentation between namespaces and pods, preventing lateral movement even if a pod is compromised.
• ETCD Security: Secure etcd access rigorously. As it stores all cluster state, compromise of etcd means compromise of the entire cluster. Encrypt data at rest and in transit, and restrict network access.

1. Prevent Lateral Movement and Persistence:

• Service Account Tokens: Audit service account permissions. Ensure they only have the necessary privileges and are not overly permissive (e.g., capable of creating pods or deploying resources across the cluster).
• Node Selector Misuse: Restrict the ability of non-admin users or service accounts to define node selectors for arbitrary nodes.
• Static Pods and Cron Jobs: Monitor for the creation of unauthorized static pods or Kubernetes cron jobs that could be used for persistence. Implement policies that restrict who can define these.
• Metadata API Access: Restrict access to cloud provider metadata APIs (e.g., 169.254.169.254) from within pods. Implement IMDSv2 (Instance Metadata Service Version 2) or similar mechanisms to prevent credential theft.

1. Address the Knowledge Gap:

• Training and Education: Invest in continuous training for security, DevOps, and development teams on Kubernetes architecture, security best practices, and common attack vectors.
• Hands-on Practice: Utilize platforms like Kubernetes Goat to provide practical, hands-on experience for both red and blue teams to understand and defend against real-world attacks.

By adopting these defensive strategies, organizations can significantly reduce their exposure to Kubernetes-specific attacks and build more resilient cloud-native environments.

Key Takeaways

• Kubernetes Security is Not Traditional Security: Securing Kubernetes demands a deep understanding of its unique architecture, components, and native features. Applying traditional security concepts without adaptation will lead to significant blind spots and vulnerabilities.
• Attackers Leverage Native Features: Attackers frequently weaponize legitimate Kubernetes functionalities like node ports, kubelet configurations, service accounts, node selectors, static pods, and cron jobs for initial access, privilege escalation, lateral movement, and persistence.
• Runtime Security with eBPF is Crucial: Traditional detection methods are often too slow for dynamic cloud-native environments. eBPF-based tools like Cilium Tetragon offer real-time, granular syscall detection and prevention capabilities, which are essential for effective defense against container escapes and other runtime attacks.
• Defense-in-Depth is Paramount: No single security tool or configuration can protect a Kubernetes cluster. A layered security approach encompassing secure configurations, robust RBAC, network policies, image security, and strong runtime protection is indispensable.
• Knowledge is Your Strongest Defense: The significant knowledge gap in Kubernetes security is a primary vulnerability. Hands-on learning environments like Kubernetes Goat are vital for security professionals to understand attack paths and develop effective defensive strategies.
• Assume Flat Networking Without Policies: Never assume namespaces provide network isolation by default. Implement explicit network policies to enforce segmentation and prevent unauthorized lateral communication between pods and namespaces.

About the Speaker(s)

Madhu Akula is a prominent figure in the Kubernetes security community, known for his deep expertise and contributions to open-source projects. He has been a fortunate speaker at numerous prestigious security and DevOps conferences globally, including Black Hat, Def Con, and KubeCon. Akula is the creator of Kubernetes Goat, a widely recognized vulnerable playground designed to help individuals learn and practice Kubernetes security in a hands-on environment. His work focuses on researching and sharing insights into cloud-native security, helping practitioners understand the intricacies of attacking and defending Kubernetes clusters.