Hacking Kubernetes

Hacking Kubernetes

Speakers: Benjamin Koltermann

Conference: Cloud Village

YouTube: https://www.youtube.com/watch?v=QGzFBtaiA7A

Overview

In his Cloud Village talk, "Hacking Kubernetes," Benjamin Koltermann provides a comprehensive exploration of common security vulnerabilities and misconfigurations within Kubernetes environments. The presentation is structured to first introduce fundamental Kubernetes concepts and then progressively delve into advanced attack vectors related to Role-Based Access Control (RBAC) and Admission Control, culminating in a series of hands-on, CTF-style practical labs. Koltermann's objective is to equip attendees with a deeper understanding of how attackers exploit Kubernetes and, crucially, how defenders can fortify their clusters against such threats.

The talk is particularly relevant for anyone involved in deploying, managing, or securing Kubernetes clusters, from developers and DevOps engineers to security professionals. It highlights that while Kubernetes offers powerful orchestration capabilities, its complexity can lead to subtle yet critical security oversights. By examining key architectural components, access mechanisms, and policy enforcement points, Koltermann underscores the importance of a layered security approach that goes beyond default configurations, emphasizing the principle of least privilege and robust validation.

The session distinguishes itself by blending theoretical explanations with concrete examples, including live demonstrations of exploitation techniques through an interactive Capture The Flag (CTF) platform. This practical focus allows participants to immediately apply their newfound knowledge, reinforcing the critical lessons on securing kubeconfigs, configuring RBAC effectively, implementing admission controllers for policy enforcement, and understanding various pod escape scenarios. The talk serves as a vital resource for understanding the attacker's mindset and building more resilient Kubernetes deployments.

Background

▶ Watch: Introduction to Kubernetes components and attack surfaces (0:00)

Kubernetes, at its core, is divided into a control plane (master nodes) and worker nodes. The control plane houses critical components like the Kubernetes API server (the primary interaction point), the cloud controller manager, the etcd database (for cluster state), and the scheduler and controller manager. On worker nodes, kubelet manages pods and containers, while kube-proxy handles network proxying for services. Access to the Kubernetes API is typically managed via kubeconfigs, which contain server URLs, Certificate Authorities (CAs), client certificates, and client keys, or service account tokens. The speaker warns that malicious plugins have been observed dumping kubeconfigs, emphasizing their critical importance as a secret.

Networking in Kubernetes is orchestrated by the Container Network Interface (CNI). Services route traffic to pods based on labels and selectors. An attacker who can control pod labeling can potentially hijack DNS or redirect traffic to malicious pods, though default load balancing in Kubernetes often relies on IP tables with probabilities, which can introduce a probabilistic element to traffic redirection. The networking model also uses an overlay network, meaning pod IPs might be directly reachable from the node's primary network in some cloud provider implementations (e.g., GKE), potentially bypassing traditional network controls.

Kubernetes storage is managed through the Container Storage Interface (CSI). A crucial security consideration is that Kubernetes volumes do not have default encryption. Encryption must be explicitly supported and configured via the chosen CSI driver. If a CSI driver does not offer encryption, volumes will remain unencrypted, even if encryption is desired.

The Container Runtime Interface (CRI) is responsible for running containers, handling image pulling, isolation, resource allocation, and lifecycle management. Popular runtimes include containerd and CRI-O. Specialized runtimes, such as those for confidential containers (e.g., using Intel SGX or AMD SEV), enhance security by running containers within encrypted micro-VMs, protecting data in memory. Understanding these foundational components is essential for identifying and mitigating security risks across the Kubernetes stack.

Key Findings

▶ Watch: Kubernetes networking: Attacking services via label manipulation (2:15)

The talk highlights several critical security findings and common misconfigurations that can lead to significant vulnerabilities in Kubernetes environments:

1. Service Account Token Exposure: Every pod, by default, has a service account token mounted under /var/run/secrets/kubernetes.io/serviceaccount/token. While these tokens have no permissions by default, if an attacker gains code execution within a pod, they can use this token to authenticate to the Kubernetes API. If the service account associated with the pod has any elevated privileges, this becomes a direct path for an attacker to interact with the cluster, potentially leading to privilege escalation or lateral movement.
2. RBAC Misconfigurations:

• Cluster-wide vs. Namespace-scoped Permissions: A common and dangerous misconfiguration is granting cluster-wide permissions (e.g., ClusterRole for reading secrets) when only namespace-scoped access is required. This can allow an attacker to read sensitive data across the entire cluster, even if their initial access was limited to a single namespace.
• Over-privileged Default Roles: While Kubernetes provides default ClusterRoles like admin, edit, and view, these can often be too broad. For instance, the edit role allows editing almost any resource, potentially enabling privilege escalation.
• Ignoring System Component Roles: When auditing RBAC, system component roles (prefixed with system:) are often highly privileged (e.g., kubelet creating pods). Defenders must understand that these are necessary for cluster operation and should be excluded from standard user/application RBAC audits to avoid false positives.

1. Admission Controller Bypass/Weakness:

• Lack of Default Encryption for Volumes: As previously mentioned, the absence of default volume encryption means that if the chosen CSI driver doesn't explicitly support and enable encryption, stored data is vulnerable.
• Performance Overhead of Third-Party Admission Controllers: While tools like Kyverno and Open Policy Agent (OPA) offer extensive policy enforcement, they can introduce significant performance overhead (up to a factor of 10 slower) compared to native Validating Admission Policies (VAPs) using Common Expression Language (CEL). This can lead to deployment bottlenecks in high-throughput environments.

1. "Unpatchable" Kubernetes CVEs: The talk mentions "four unpatchable CVEs" in Kubernetes due to system design. One specific example highlighted is the ability to perform a man-in-the-middle (MITM) attack by defining an externalIP field in a Kubernetes service. An attacker creating a service with a target IP in this field can intercept traffic intended for that IP address, effectively hijacking communications without needing to compromise the underlying network infrastructure.
2. Pod Escape Techniques: The presentation details a straightforward yet highly effective pod escape technique, often the first one tried in a penetration test. By configuring a pod with hostPID: true (allowing it to see host processes) and privileged: true (giving it access to host devices), an attacker can use nsenter to enter the mount namespace of the host's PID 1 (init process). This grants the attacker a root shell on the underlying host, completely compromising the worker node. This highlights the critical importance of restricting privileged containers and host namespaces.
3. Service Hijacking via Label Control: If an attacker can control the labels of pods, they can manipulate Kubernetes services to route traffic to their malicious pods by matching the service's selector. This can be used for various attacks, including data exfiltration or denial of service, though the probabilistic nature of Kubernetes load balancing might make it less reliable for direct interception without changing the service selector itself.

These findings collectively emphasize that while Kubernetes provides a robust platform, secure configuration requires deep understanding and continuous vigilance against both common misconfigurations and inherent design considerations.

Technical Deep Dive

▶ Watch: Bypassing network controls with Kubernetes overlay networks (4:15)

Koltermann’s deep dive focuses on two critical security mechanisms: Role-Based Access Control (RBAC) and Admission Control, illustrating how they function and where misconfigurations can introduce vulnerabilities.

Role-Based Access Control (RBAC)

RBAC is the primary mechanism for regulating access to the Kubernetes API. It enforces the principle of least privilege by defining granular permissions. A key aspect highlighted is the service account token mounted in every pod at /var/run/secrets/kubernetes.io/serviceaccount/token. While these tokens are valid credentials to the Kubernetes API, by default, new service accounts have no permissions other than the ability to create a self-subject-access-review to inspect their own permissions. A crucial defensive recommendation is to set automountServiceAccountToken: false on pods or service accounts if the application does not need to interact with the Kubernetes API, thereby removing a potential attack vector.

To enable RBAC, the API server configuration must have authorizers set to RBAC. RBAC distinguishes between cluster-scoped and namespace-scoped permissions. A Role defines permissions within a specific namespace (e.g., get or read pods in the default namespace), while a ClusterRole defines permissions across the entire cluster (e.g., reading secrets cluster-wide). Misconfiguring a ClusterRole to grant broad permissions (like secret reading) when only namespace-specific access is intended is a common and severe vulnerability.

Permissions are granted using RoleBindings (for Roles within a namespace) and ClusterRoleBindings (for ClusterRoles across the cluster) to subjects such as users, service accounts, or groups. The talk presents examples:

• A Role named pod-reader granting get, watch, list on pods in the default namespace.
• A ClusterRole named secret-reader granting get, watch, list on secrets cluster-wide.
• A RoleBinding binding pod-reader to a user Jane.
• A ClusterRoleBinding binding secret-reader to a group manager.

Kubernetes includes several default ClusterRoles:

• cluster-admin: Full administrative access to the cluster.
• admin: Full administrative access within a namespace, excluding cluster-level resources.
• edit: Allows editing most resources within a namespace, but not viewing or modifying roles or role bindings.
• view: Allows viewing most resources within a namespace, but explicitly excludes secrets.

System components also have roles, typically prefixed with system:, which are necessary for cluster operations (e.g., system:kubelet). These should be considered separately during security audits.

Admission Control

Admission controllers act as "bouncers" to the Kubernetes API server, intercepting requests after authentication and authorization but before objects are persisted to etcd. They can mutate (modify) or validate (reject) API requests. This allows for enforcing security policies, compliance, and injecting sidecar containers.

The API request flow involving admission controllers is:

1. User sends API request.
2. HTTP handler processes the request.
3. Authentication occurs.
4. Authorization occurs.
5. Mutating Admission Controllers (via webhooks) can modify the request.
6. Object Schema Validation ensures the mutated object is still valid Kubernetes YAML.
7. Validating Admission Controllers (via webhooks) can reject the request based on policy.
8. The object is finally stored in etcd.

Since Kubernetes v1.28, Pod Security Admission (PSA) and Pod Security Standards are built-in admission controllers. PSA defines three security profiles:

• Privileged: Unrestricted policy, allows any pod configuration.
• Baseline: Minimally restrictive, prevents known privilege escalations. Disallows hostPath volumes, hostPID, hostNetwork, privileged containers.
• Restricted: Heavily restricted, follows pod hardening best practices. Requires runAsNonRoot, disallows most volume types, and host namespaces. Deployments might break under this profile due to its strictness.

These standards can be enforced on a namespace level using labels, e.g., pod-security.kubernetes.io/enforce: restricted. PSA also supports audit (writes to audit logs) and warn (user-facing warning) modes.

For more fine-grained control, Kubernetes introduced Validating Admission Policies (VAPs), which use Common Expression Language (CEL) from Google. VAPs allow users to define custom validation rules directly within Kubernetes. For example, a VAP can restrict spec.replicas in a deployment to be less than or equal to 5. These policies are activated by Validating Admission Policy Bindings, which can scope the policy to specific namespaces using a namespaceSelector (e.g., only in namespaces labeled environment: test). The talk mentions that Mutating Admission Policies are also under development.

A critical performance note is the comparison between VAPs and third-party admission controllers like Kyverno or OPA. VAPs are significantly faster (up to 10x), making them preferable for high-throughput environments where rapid validation is crucial, as demonstrated by a Mercedes-Benz case study. Third-party tools can still work alongside VAPs, but their performance impact should be considered.

Demo / Proof of Concept

▶ Watch: Kubernetes storage security: CSI and the lack of default encryption (5:50)

The core of the practical demonstration involved a series of CTF-style labs hosted on a dedicated platform. Attendees were provided with an nc connection string, an API token, and a kubeconfig to interact with ephemeral Kubernetes clusters created on demand. The labs were designed to illustrate the vulnerabilities and exploitation techniques discussed in the theoretical sections.

The speaker outlined five distinct scenarios:

1. General 2 (Kubernetes Basics): This introductory challenge helped users familiarize themselves with kubectl. The flag was split into three parts:

• Reading a secret using kubectl get secret.
• Reading container logs from a specific namespace.
• Inspecting a Custom Resource Definition (CRD) named flags using kubectl get flags. Combining these parts revealed the full flag.

1. General 1 (Man-in-the-Middle): This lab focused on exploiting one of the "unpatchable CVEs" in Kubernetes. The challenge required executing a man-in-the-middle (MITM) attack using the externalIP field of a Kubernetes service. By creating a service and setting its externalIP to a target IP address, the attacker could intercept traffic intended for that IP within the cluster, demonstrating a critical design flaw.

1. RBAC 1 (Secret Exfiltration without get secret): This challenge demonstrated how to read secrets even without explicit get secret permissions. The scenario involved a user account with permissions to list pods, create pods, and delete pods, but not get secrets. The solution involved creating a new pod that mounts the secret from an existing pod (which the attacker could list and inspect to find the secret's name). The newly created pod, controlled by the attacker, would then print the mounted secret's content to its logs, which the attacker could then read using kubectl logs. This highlights how control over pod creation can bypass seemingly restrictive RBAC.

1. RBAC 2 (Namespace Hopping): This lab simulated a privilege escalation path involving lateral movement across namespaces. Starting in a default namespace with limited permissions, the attacker would find a pod, exec into it, and obtain a service account token. This token would grant access to a dev namespace, then a staging namespace, and finally a prod namespace where the flag was located. This illustrated how chaining misconfigured service accounts or roles can lead to broader cluster access.

1. Admission Challenge (Bypassing Admission Control): This advanced challenge required participants to define a privileged pod and then write their own admission rules to prevent such a pod from being created. The goal was to submit the correct admission controller configuration (e.g., a Validating Admission Policy) that would deny the creation of the privileged pod, showcasing the defensive power of admission control.

Beyond the structured labs, Koltermann also demonstrated a classic pod escape technique:

• The nsenter Pod Escape: Create a pod with hostPID: true (allowing the pod's processes to share the host's PID namespace) and privileged: true (granting broad capabilities, including access to /dev). Inside this privileged pod, an attacker can use nsenter --target 1 --mount --uts --ipc --net --pid -- /bin/bash. This command targets PID 1 (the init process on the host) and enters its various namespaces, crucially including its mount namespace. This effectively gives the attacker a root shell on the underlying worker node, demonstrating a complete compromise. Koltermann emphasized this is often the first technique tried in a cluster penetration test due to its simplicity and devastating impact if successful.

The practical nature of these demos underscored the theoretical concepts, providing tangible proof of concept for the vulnerabilities discussed.

Defensive Implications

▶ Watch: Container Runtime Interface (CRI) and its security implications (6:50)

The talk provides several critical defensive implications and actionable strategies for securing Kubernetes clusters:

1. Secure Kubeconfigs and Service Account Tokens:

• Treat kubeconfigs as highly sensitive secrets.
• Disable automountServiceAccountToken on pods and service accounts if the application does not explicitly need to interact with the Kubernetes API. This significantly reduces the attack surface if a pod is compromised.
• Implement least privilege for any service account that does require API access.

1. Robust RBAC Configuration:

• Principle of Least Privilege: Grant only the minimum necessary permissions to users, groups, and service accounts.
• Namespace Scoping: Prioritize namespace-scoped Roles and RoleBindings over cluster-wide ClusterRoles and ClusterRoleBindings wherever possible, especially for sensitive permissions like reading secrets.
• Audit RBAC Regularly: Continuously analyze policies and ACLs to detect misconfigurations in real-time. Differentiate between legitimate system component roles (prefixed with system:) and user/application roles during audits.
• Avoid Over-privileged Default Roles: Be cautious when using default ClusterRoles like admin or edit, as they can grant broad permissions.

1. Implement Strong Admission Control:

• Enforce Pod Security Standards (PSS): Utilize the built-in Pod Security Admission controller to enforce baseline or restricted profiles on namespaces. This prevents common privilege escalation techniques by disallowing privileged containers, host namespaces, certain volume types, and non-read-only filesystems.
• Leverage Validating Admission Policies (VAPs): Use VAPs with CEL for fine-grained, custom policy enforcement to meet specific organizational compliance or security requirements. Consider their performance benefits over third-party solutions for high-throughput environments.
• Use Admission Controllers for Key Security Policies: Enforce the usage of network policies, ensure security contexts are correctly applied (e.g., readOnlyRootFilesystem), and prevent the creation of insecure or non-compliant Kubernetes objects.
• Consider Third-Party Tools: While VAPs offer performance, tools like Kyverno and Open Policy Agent (OPA) provide extensive policy libraries and a mature ecosystem for complex policy management. Choose based on specific needs and performance considerations.

1. Secure Networking and Storage:

• Control Pod Labeling: Implement controls to prevent unauthorized modification of pod labels, which could lead to service hijacking.
• Ensure Storage Encryption: Verify that chosen CSI drivers support and are configured for volume encryption. Kubernetes does not encrypt volumes by default.
• Network Segmentation: Implement network policies to isolate workloads and restrict traffic flow between pods and namespaces.

1. Continuous Monitoring and Discovery:

• Map All Resources: Maintain a comprehensive inventory of all Kubernetes resources in the environment.
• Scan Continuously: Regularly scan clusters for misconfigurations, both anonymously and from multiple cloud service providers (CSPs), to catch various types of vulnerabilities.
• Audit Logs: Monitor Kubernetes audit logs for suspicious API activity, especially related to role bindings, service account usage, and pod creation.

1. Prevent Pod Escapes:

• Strictly disallow privileged: true containers, hostPID: true, hostNetwork: true, and other host-level access configurations through admission controllers. These settings are frequently abused for container escapes.

By proactively implementing these defensive measures, organizations can significantly reduce their Kubernetes attack surface and enhance the overall security posture of their containerized applications.

Key Takeaways

• RBAC is foundational but complex: Misconfigurations, particularly around cluster-wide permissions and over-privileged service accounts, are common entry points for attackers. Always apply the principle of least privilege.
• Service Account Tokens are critical attack vectors: Disable automountServiceAccountToken if applications don't need API access, or ensure associated roles have minimal permissions.
• Admission Controllers are powerful policy enforcers: Utilize built-in Pod Security Admission (PSA) and custom Validating Admission Policies (VAPs) to prevent insecure configurations before they are persisted.
• Performance matters for admission control: Native VAPs offer significant performance advantages over third-party tools like Kyverno or OPA for high-throughput environments.
• Kubernetes has inherent design "quirks": Be aware of "unpatchable" vulnerabilities, such as the externalIP service MITM, and design defenses around them.
• Pod escapes are a real and present danger: Restrict privileged containers and host namespace access rigorously through admission policies to prevent attackers from gaining root on worker nodes.

About the Speaker(s)

Benjamin Koltermann is a security expert with a deep understanding of Kubernetes. His expertise is evident in his contributions to best practices and practical security challenges. He has co-authored kube-bench related RBAC good practices, demonstrating his commitment to guiding the community toward more secure Kubernetes deployments. Furthermore, Koltermann is a co-author of the Kubernetes-specific CTF challenges at Defcon, showcasing his ability to design realistic and engaging scenarios that highlight critical vulnerabilities and effective exploitation techniques. His work focuses on bridging the gap between theoretical security knowledge and practical application in cloud-native environments.