Tutorial: Hacking up a Storm With Kubernetes - Rory McCune, Datadog; Marion McCune & Iain Smart

Tutorial: Hacking up a Storm With Kubernetes

Speakers: Rory McCune, Datadog; Marion McCune; Iain Smart

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=8Q8sFzODEUo

Overview

This tutorial, "Hacking up a Storm With Kubernetes," presented primarily by Rory McCune of Datadog, offers a compelling journey into the intricacies of Kubernetes security through the lens of a hacking scenario. The talk aims to demystify complex security concepts by demonstrating how a seemingly innocuous developer attempting to deploy an application can escalate privileges and ultimately compromise a Kubernetes cluster. McCune's personal goal for the session was for every attendee to leave with at least one new insight into Kubernetes security, a goal he demonstrably achieved by revealing several lesser-known features and potential misconfigurations.

The core premise revolves around a developer facing admission control restrictions and their creative, albeit insecure, methods to bypass these controls. This narrative effectively illustrates common pitfalls in Kubernetes deployments, such as overly permissive RBAC, misconfigured service accounts, and overlooked security implications of standard features. By walking through a multi-step attack path, McCune highlights critical vulnerabilities and the defense-in-depth strategies necessary to secure modern containerized environments. The tutorial serves as a practical guide for both developers and security professionals to understand how attackers think and how to build more resilient Kubernetes clusters.

The talk is particularly relevant in today's cloud-native landscape, where Kubernetes has become the de facto orchestrator for containerized applications. As organizations increasingly adopt Kubernetes, the attack surface expands, and the nuances of its security model become paramount. This session provides invaluable insights into the practical exploitation of Kubernetes features, emphasizing the importance of granular permissions, robust admission policies, and continuous security education. It’s a vital resource for anyone looking to deepen their understanding of Kubernetes security and protect their deployments from sophisticated threats.

Background

▶ Watch: Tutorial introduction, goals, and speaker background (0:00)

Kubernetes, by its very nature as a distributed system designed for remote command execution, presents a complex security landscape. Securing a cluster involves multiple layers, starting with fundamental concepts like authentication, authorization, and admission control. Authentication verifies user or workload identity, often via OIDC for human users or service account tokens for workloads. Authorization determines what an authenticated entity is permitted to do, typically managed by Role-Based Access Control (RBAC). Finally, admission control acts as a crucial third gate, enforcing additional policies even after authentication and authorization have passed, preventing dangerous configurations from being deployed.

The problem explored in the tutorial stems from common deployment patterns and misconfigurations. Developers often require the ability to deploy workloads, which inherently grants them create pod rights. However, certain pod configurations, such as using hostPath volumes, hostNetwork, hostPID, or the privileged: true security context, can grant direct access to the underlying node. These configurations are highly dangerous as they can lead to immediate privilege escalation to root on the node, effectively compromising the entire cluster.

To mitigate such risks, Kubernetes introduced Pod Security Standards (PSS), which define three levels of security policies: Privileged, Baseline, and Restricted. The Baseline standard, used in the tutorial's scenario, is designed to block known privilege escalations by disallowing configurations like hostPath volumes and privileged containers. Prior to PSS, similar policies were often enforced by external admission controllers like Kyverno or OPA, or deprecated solutions like Pod Security Policies (PSPs). The challenge, as demonstrated, lies in ensuring these controls are not only in place but also resilient to bypass techniques, especially when combined with overly permissive RBAC configurations for service accounts or a misunderstanding of how Kubernetes objects and permissions interact.

Key Findings

▶ Watch: Technical requirements: Docker, Kind, and workshop repo (2:00)

The tutorial uncovered several critical findings and exploitable behaviors within Kubernetes, highlighting common security misconfigurations and less-understood features:

1. RBAC star includes escalate verb: A significant discovery was that granting star permissions within RBAC on certain resources (like roles or rolebindings) implicitly includes the escalate verb. This verb allows a user to create roles with permissions they do not currently possess, directly enabling privilege escalation. This means an attacker with star on RBAC objects can grant themselves arbitrary permissions within their scope (e.g., namespace-wide admin rights), even if the original star privilege was intended to be limited to basic CRUD operations.

1. Namespace Labels as Security Controls: The demonstration revealed that Kubernetes security controls like Pod Security Admission (PSA) are enforced via labels on namespaces. If an attacker gains sufficient permissions within a namespace (e.g., ns-admin rights), they can modify or remove these labels. By doing so, they can effectively disable namespace-level security policies, allowing previously blocked, insecure workloads to be deployed. This underscores the vulnerability of security controls implemented solely through labels if RBAC is not tightly controlled.

1. Unrestricted create pod leads to Node Root: A foundational security principle reiterated was that unrestricted create pod permissions, especially when combined with the ability to bypass admission controls, is equivalent to gaining root access on the underlying node. The tutorial demonstrated how deploying a pod with privileged: true, hostNetwork: true, and hostPID: true allows an attacker to chroot /host and achieve full root access on the host, illustrating the severe consequences of weak pod security policies.

1. Persistent Cluster Admin via admin.conf / super-admin.conf: Once root access to a control plane node is achieved, an attacker can extract persistent, non-expiring cluster administrative credentials, specifically the admin.conf and super-admin.conf files. The super-admin.conf is particularly dangerous as it belongs to the system:masters group, which has a hard-coded bypass for all authorization modules in the Kubernetes API server, making its privileges unrevocable via RBAC.

1. Unpatchable CVE-2020-8554 (Traffic Hijacking): The tutorial showcased the exploitation of a known, unpatchable Kubernetes CVE (CVE-2020-8554). This vulnerability allows an attacker to hijack external network traffic by creating a Kubernetes Service with externalIPs that match a target external IP address. Because Kubernetes Services are implemented using IPtables rules, an attacker can redirect traffic intended for an external service to an internal, attacker-controlled pod, enabling man-in-the-middle attacks.

Technical Deep Dive

▶ Watch: Setting the scene: Developer's challenge for the tutorial (4:30)

The technical deep dive into the attack path meticulously illustrates how an attacker can leverage seemingly minor misconfigurations to achieve significant privilege escalation. The scenario begins with a developer attempting to deploy a log reader application that uses a hostPath volume to read logs from a pod in the kube-system namespace. This deployment initially fails due to Pod Security Admission (PSA) enforcing the baseline policy in the dev namespace, which blocks hostPath volumes.

The first step in the attack involves understanding the developer's initial permissions. Using kubectl auth can-i --list, it's revealed that the developer has limited read-only rights, but crucially, can create pods and get serviceaccounts within their dev namespace. A critical observation is the presence of an rbac-manager service account in the dev namespace, which sounds like it might have elevated privileges.

Since the developer can create pods, they create a new pod (token-read) configured to use the rbac-manager service account. Kubernetes automatically mounts the service account's token into the pod at /var/run/secrets/kubernetes.io/serviceaccount/token. The attacker then execs into this token-read pod and extracts the JWT token using cat. This token is then decoded (e.g., using jwt.io or jq -R 'split(".") | .[1] | @base64d | fromjson') to confirm it belongs to rbac-manager.

With the rbac-manager token in hand, the attacker uses kubectl auth can-i --list --token <RBAC_MANAGER_TOKEN> to query its permissions. The revelation is that rbac-manager has star permissions on rolebindings and roles within the namespace. This star permission implicitly grants the escalate verb. The escalate verb is a powerful RBAC permission that bypasses the typical Kubernetes restriction preventing users from creating roles with more privileges than they currently possess.

Leveraging the escalate verb, the attacker creates a new Role named ns-admin with star permissions across all resources () and all verbs () within the dev namespace. They then create a RoleBinding to associate this ns-admin role with their original developer service account. This effectively grants the developer full administrative control over the dev namespace.

The final step in bypassing the initial problem is to disable PSA. Since PSA is enforced via labels on the namespace, and the developer now has ns-admin rights (including the ability to modify namespace objects), they can remove the PSA label. The command kubectl label namespace dev pod-security.kubernetes.io/enforce- --overwrite removes the enforcement label, effectively disabling PSA for the dev namespace. With PSA disabled, the original log reader application, which uses a hostPath volume, can now be successfully deployed and run.

Demo / Proof of Concept

▶ Watch: Hands-on: Creating the Kind Kubernetes cluster (5:00)

The tutorial featured several compelling live demonstrations that showcased the entire attack chain and further exploitation techniques.

1. Initial Blocked Deployment:

The first demo illustrated the problem: the read-log-file application, which attempts to mount a hostPath volume (/var/log from the node), failed to deploy in the dev namespace. kubectl describe pod revealed the admission control rejection, explicitly stating that the pod violated the baseline Pod Security Standard due to the use of hostPath. This set the stage for the subsequent privilege escalation.

1. Service Account Token Exploitation and RBAC Escalation:

This was the core of the demo. After identifying the rbac-manager service account, Rory McCune demonstrated creating a temporary pod that utilized this service account. He then execd into that pod, extracted the service account's JWT token from /var/run/secrets/kubernetes.io/serviceaccount/token, and decoded it to prove its authenticity. The kubectl auth can-i --list --token <token> command was then used to show the rbac-manager's permissions, revealing the critical star on RBAC resources, which implicitly included the escalate verb.

With the escalate verb, the demo proceeded to create a new Role named ns-admin with star permissions for all resources and verbs within the dev namespace. This ns-admin role was then bound to the developer's original service account. This step visibly transformed the developer's permissions from restricted to full ns-admin, as confirmed by another kubectl auth can-i --list check, which now showed "lots of stars."

1. Pod Security Admission Bypass:

The next logical step was to bypass the very control that initially blocked the deployment. Since the developer now had ns-admin rights, McCune demonstrated how to modify the dev namespace's labels. Specifically, the pod-security.kubernetes.io/enforce: baseline label was removed using kubectl label namespace dev pod-security.kubernetes.io/enforce- --overwrite. This action immediately allowed the previously failing read-log-file pod to start successfully, demonstrating a complete bypass of the admission control.

1. Cluster Compromise (Node Root Access):

Beyond namespace administration, the demo escalated further to full cluster compromise. A manifest for a node-root pod was applied. This pod was configured with privileged: true, hostNetwork: true, and hostPID: true – effectively disabling all container security boundaries. Upon execing into this node-root pod and running chroot /host, the terminal prompt changed, indicating root access to the underlying control plane node. This visually confirmed that unrestricted create pod rights, combined with security context bypasses, equate to full node compromise.

From the compromised node, the admin.conf and super-admin.conf files were extracted. These files contain client certificates that grant permanent, unrevocable cluster-admin access. The super-admin.conf was highlighted as particularly dangerous due to its system:masters group membership, which bypasses all RBAC authorization.

1. CVE-2020-8554 (Unpatchable Traffic Hijacking):

The final, advanced demo showcased the exploitation of CVE-2020-8554. A client pod was deployed, attempting to reach icanhazip.com (a service that returns the client's public IP). Initially, curl icanhazip.com from the client pod returned the expected external IP.

The attacker then deployed a Kubernetes Service with externalIPs set to the IP addresses of icanhazip.com, but configured to route traffic to an internal Nginx deployment. After applying this service, curl icanhazip.com from the client pod now returned the Nginx welcome page, demonstrating successful traffic hijacking. This illustrated how Kubernetes' reliance on IPtables for service implementation can be abused to redirect external traffic internally, even in fully patched clusters, underscoring the need for strict admission control on externalIPs usage.

Defensive Implications

▶ Watch: Verifying the Kind Kubernetes cluster setup (6:00)

The attack paths demonstrated in the tutorial highlight critical areas where Kubernetes clusters are vulnerable and provide clear guidance for strengthening defenses:

1. Strict Namespace Segregation and Least Privilege for Service Accounts:

The rbac-manager service account, with its broad permissions, should never have been in a developer-facing namespace. Privileged service accounts must be placed in dedicated, tightly controlled namespaces where only essential, trusted components can operate. Furthermore, service accounts should always adhere to the principle of least privilege, being granted only the absolute minimum permissions required for their function. Avoid granting star permissions to any service account, as this often includes unintended and dangerous verbs like escalate.

1. Robust Admission Control:

Admission controllers are the most effective gatekeepers for enforcing security policies.

• Pod Security Admission (PSA) should be configured to enforce baseline or restricted policies on all application namespaces. For highly sensitive workloads, restricted is preferable.
• For features not covered by PSA, such as the externalIPs in services (CVE-2020-8554), external admission controllers like Kyverno or OPA/Gatekeeper should be used. These can specifically block the creation of services with externalIPs unless absolutely necessary and restricted to specific, highly controlled namespaces. They can also enforce policies against privileged: true containers, hostNetwork, hostPID, and hostPath volumes, even if PSA is not fully configured or is bypassed at the namespace label level.
• If using Kubernetes 1.30+, ValidatingAdmissionPolicy offers a native, internal way to define and enforce custom policies.

1. Granular RBAC and Avoidance of star:

The escalate verb's inclusion in star permissions on RBAC resources is a critical takeaway. Cluster administrators must be extremely cautious when granting star permissions on roles or rolebindings, even within a namespace. Instead, explicitly list allowed verbs and resources. Regularly audit ClusterRoles, Roles, ClusterRoleBindings, and RoleBindings for overly permissive configurations. Tools like kube-audit or rbac-lookup can help identify such risks.

1. Preventing Unrestricted Pod Creation:

The ability to create pods with dangerous security contexts (e.g., privileged: true, hostNetwork: true, hostPID: true, hostPath volumes) must be strictly controlled. This is a primary function of PSA and other admission controllers. Preventing such pod configurations from being deployed is paramount to preventing node root access.

1. Secure Control Plane Nodes:

The demonstration of extracting admin.conf and super-admin.conf highlights the importance of securing control plane nodes. Physical and network access to these nodes must be extremely restricted. These configuration files contain powerful, non-expiring credentials that grant cluster-admin access, with super-admin.conf being particularly insidious due to its system:masters group membership which bypasses RBAC. Regular security audits of control plane nodes and strict access controls are essential.

1. Network Policies:

While not the primary focus of the CVE-2020-8554 demo, Network Policies can provide an additional layer of defense. Implementing a default-deny policy for both ingress and egress traffic in all namespaces, and then explicitly allowing only necessary communication, can help mitigate the impact of traffic hijacking or lateral movement attempts by compromised pods, even if an externalIPs service is created. For instance, blocking egress from a "bad" namespace to specific external IPs or internal services could prevent the hijacked traffic from reaching its destination.

Key Takeaways

• Kubernetes security is complex: Many subtle interactions and lesser-known features can lead to unexpected vulnerabilities, making continuous learning essential.
• RBAC's escalate verb is dangerous: Granting star permissions on RBAC objects (like roles or rolebindings) implicitly includes the escalate verb, allowing users to create roles with privileges they don't currently possess, leading to privilege escalation.
• Namespace labels are critical security controls: Policies like Pod Security Admission are enforced via namespace labels. If an attacker gains ns-admin rights, they can modify these labels to disable security controls.
• Unrestricted create pod is equivalent to node root: The ability to deploy pods with privileged: true, hostNetwork: true, hostPID: true, or hostPath volumes can grant immediate root access to the underlying node.
• Persistent cluster admin credentials can be extracted: Compromising a control plane node allows attackers to steal admin.conf and super-admin.conf files, providing permanent, unrevocable cluster-admin access.
• "Unpatchable" CVEs exist: Some Kubernetes vulnerabilities, like CVE-2020-8554 (traffic hijacking via externalIPs services), cannot be patched and require defensive configurations like admission controllers to mitigate.

About the Speaker(s)

The tutorial was primarily delivered by Rory McCune, a seasoned expert in container security with approximately nine years of experience in the field. Rory is actively involved in the Kubernetes community, contributing to Kubernetes SIG Security and helping maintain the CIS benchmarks for Docker and Kubernetes. His extensive background makes him a recognized authority on container and Kubernetes security best practices and common misconfigurations. He is also a prolific blogger on security topics, sharing insights at rorymccune.com.

While the schedule listed Marion McCune and Iain Smart as co-presenters, they were unfortunately unable to attend the conference. Their contributions to the tutorial's content and design are acknowledged, reflecting a collaborative effort in developing this in-depth security session.