Observing Escalation Paths in Kubernetes

Observing Escalation Paths in Kubernetes

Speakers: William Taylor, Security Consultant, Reverse

Conference: fwd:cloudsec

YouTube: https://www.youtube.com/watch?v=TlGIW10oEbo

Overview

In this insightful talk from fwd:cloudsec, William Taylor, a Security Consultant at Reverse, delves into the often-overlooked security implications of observability tools within Kubernetes environments. The presentation highlights a fundamental conflict between the architectural need for pervasive monitoring and the security principle of least privilege, demonstrating how commonly deployed observability agents can inadvertently introduce critical privilege escalation paths leading to potential cluster compromise. Taylor illustrates these vulnerabilities with real-world examples, particularly focusing on the Amazon CloudWatch observability add-on in EKS.

Taylor's talk is crucial for anyone involved in securing Kubernetes clusters, from cluster administrators and security engineers to developers of cloud-native applications. It sheds light on how seemingly innocuous operational tools, essential for managing complex distributed systems, can become entry points for attackers if their underlying permissions are not meticulously scrutinized. By examining specific configurations and permissions, the presentation offers concrete examples of how security weaknesses can be "unwittingly introduced" into an environment, even when using managed services.

The significance of this talk lies in its practical advice and timely warnings. While the specific vulnerabilities discussed have seen partial remediation, the underlying architectural patterns and security principles remain highly relevant. Taylor emphasizes the importance of understanding the cluster's threat model, scrutinizing the permissions of DaemonSets, and adopting a robust defense in depth strategy. The discussion extends beyond mere bug reporting, prompting a deeper consideration of how operational necessities can clash with security best practices, and offering guidance on navigating this complex landscape.

Background

▶ Watch: Introduction to Kubernetes privilege escalation problem (1:10)

Kubernetes, renowned for its power and flexibility, is also famously complex, with a multitude of moving parts that can make securing it a significant challenge. A cornerstone of Kubernetes security is privilege separation, which aims to isolate workloads and restrict their permissions to only what is absolutely necessary for their function. This adheres to the principle of least privilege, a fundamental security concept designed to minimize the impact of a compromise. In a typical Kubernetes cluster, different workloads are assigned specific service accounts with granular permissions defined by Kubernetes RBAC (Role-Based Access Control) roles and cluster roles. High-privilege accounts, often required for control plane components or sensitive worker-plane operations, are ideally segregated to limit their exposure.

However, the reality of managing large-scale Kubernetes clusters necessitates comprehensive monitoring and data collection. This is where observability tools come into play. These tools, which gather telemetry such as metrics, logs, and traces, are indispensable for cluster administrators to identify anomalies, manage resources, and troubleshoot issues. Their architecture commonly involves a management deployment and agents that are deployed to every node to collect data across all compute resources. This pervasive deployment is typically achieved using a DaemonSet, a Kubernetes resource that ensures a pod runs on each available node in the cluster.

The conflict arises because these observability agents often require elevated permissions to perform their data collection duties across the entire cluster. When a DaemonSet deploys an agent with high privileges onto every node, it can inadvertently create new escalation paths. If an attacker manages to compromise a low-privilege workload, escapes to the node, and then gains access to the highly privileged service account of an observability agent, they could potentially leverage those permissions to achieve cluster compromise. This scenario directly undermines the efforts to enforce privilege separation and minimize the blast radius of a successful attack, highlighting a critical tension between operational visibility and security posture.

Key Findings

▶ Watch: How observability tools introduce new escalation paths (4:00)

William Taylor's research identified two distinct, yet related, privilege escalation paths stemming from the deployment of common observability tools in Kubernetes, specifically focusing on the Amazon CloudWatch observability add-on in EKS (Amazon Elastic Kubernetes Service).

1. Cluster-Wide ConfigMap Update Privilege: The primary finding revolved around the cloudwatch-agent service account, which was granted a cluster role allowing it to update configmaps across the entire cluster. This permission was observed in conjunction with the deprecated EKS API and config map authentication mode. The vulnerability allowed an attacker who compromised the cloudwatch-agent's service account to modify the sensitive aws-auth ConfigMap. By adding themselves or an arbitrary IAM identity to the system:masters group within this ConfigMap, an attacker could effectively grant themselves cluster admin privileges, leading to full cluster compromise. Taylor noted that this was not a misconfiguration by the user but an out-of-the-box vulnerability when the add-on was enabled with the specific authentication mode. AWS later remediated this specific issue by updating the agent's permissions to a namespaced role for ConfigMap updates in versions 5.3 and later of the add-on.

1. nodes/proxy Permission for Pod Execution: The second, more pervasive finding involved the get permission on the nodes/proxy resource. This permission, often present in observability tools to access data like pod metrics and container logs, was shown to enable execution on any pod within the cluster, including sensitive ones. Taylor referenced a blog post by Graeme Helton (released January 2023) that detailed how this specific permission could be abused to execute commands on pods via the kubelet API. While this permission is often deemed necessary for observability, its broad scope creates an unintended escalation path. Unlike the ConfigMap issue, AWS's initial fix for the CloudWatch add-on did not address this nodes/proxy vulnerability, which remained present in later versions of the add-on. However, during the Q&A, a Kubernetes developer clarified that a safer, more granular set of permissions was introduced in Kubernetes 1.26 (GA in 1.36 by default) to replace the broad nodes/proxy permission, encouraging tool developers to migrate.

These findings underscore a critical challenge: the inherent tension between the need for comprehensive observability and the strict enforcement of least privilege in complex distributed systems like Kubernetes. Both issues highlight how standard, often managed, deployments can introduce significant security risks if their underlying permissions are not thoroughly understood and appropriately constrained.

Technical Deep Dive

▶ Watch: Real-world example: EKS CloudWatch add-on vulnerability (6:00)

The core of William Taylor's analysis centers on how specific permissions granted to the Amazon CloudWatch observability add-on in EKS could be exploited for privilege escalation. The attack chain relies on two key prerequisites in the EKS cluster: the authentication mode being set to EKS API and config map, and the observability add-on being enabled.

EKS Authentication Modes:

EKS offers two primary ways to manage IAM identity to Kubernetes RBAC mappings:

• EKS API: This mode uses access entries managed directly within the EKS service. It associates IAM identities with Kubernetes permissions. This is the newer, recommended approach.
• Config Map: This older, deprecated method uses the aws-auth ConfigMap within the cluster itself. This ConfigMap explicitly maps IAM users and roles to Kubernetes users and groups, which are then bound to cluster roles or roles via RBAC. Taylor emphasizes that while deprecated, this mode is still commonly found in clusters, often without explicit warnings during cluster creation in the AWS console. When both "EKS API and config map" are selected, the aws-auth ConfigMap still functions as the primary mechanism for IAM-to-RBAC mapping.

CloudWatch Observability Add-on Architecture and Permissions:

The CloudWatch add-on, a curated tool for EKS, deploys several resources into the cluster, including its own namespace, a management deployment, and crucially, two DaemonSets that act as agents. Both DaemonSets utilize the same cloudwatch-agent service account. This service account is assigned a cluster role, meaning its permissions apply across all resources in the cluster, not just within its namespace.

Taylor's investigation revealed that the cloudwatch-agent's cluster role had an update permission on configmaps cluster-wide. While the agent legitimately needed to update specific ConfigMaps within its own namespace for configuration purposes (which Taylor verified by observing configuration changes and testing permission removal), the cluster-wide scope was problematic. It allowed the agent to modify any ConfigMap in the cluster, including the critical aws-auth ConfigMap.

Exploitation of aws-auth ConfigMap:

The privilege escalation path leveraging this configmaps/update permission is as follows:

1. An attacker compromises an initial low-privilege workload.
2. The attacker escapes from the compromised pod to the underlying node.
3. On the node, the attacker gains access to the cloudwatch-agent service account token. Since the cloudwatch-agent runs as a DaemonSet on every node, its service account token is accessible on any node hosting the agent.
4. Using the cloudwatch-agent's token, the attacker, with the configmaps/update cluster-wide permission, modifies the aws-auth ConfigMap.
5. The attacker adds an entry to the aws-auth ConfigMap that maps an existing or newly created IAM identity (or even their own IAM identity if they possess one) to the system:masters group. For example, adding an entry like:

1. This grants the specified IAM identity cluster admin privileges, effectively compromising the entire EKS cluster.

The nodes/proxy Vulnerability:

Beyond the ConfigMap issue, Taylor also highlighted a separate, more fundamental vulnerability related to the nodes/proxy resource. The cloudwatch-agent (and many other observability tools) holds get permissions on nodes/proxy. This permission is often considered necessary for agents to collect data like pod metrics and container logs by directly accessing the kubelet API on the nodes.

However, as detailed in Graeme Helton's blog, the get nodes/proxy permission is overly broad. It allows an attacker to proxy requests to the kubelet on any node, including the /run endpoint. This can be abused to execute arbitrary commands on any pod running on that node, including sensitive control plane pods or other highly privileged workloads. This means that even if the cloudwatch-agent itself doesn't have other direct cluster-wide write permissions, its nodes/proxy permission still provides a potent avenue for privilege escalation once its service account is compromised.

Remediation and Lingering Concerns:

AWS addressed the configmaps/update vulnerability in CloudWatch add-on versions 5.3 and later by changing the cloudwatch-agent's ConfigMap permissions to a namespaced role, restricting its ability to only update ConfigMaps within its own namespace. This effectively closed the aws-auth ConfigMap escalation path.

However, the nodes/proxy issue was not addressed by this fix. The get nodes/proxy permission remained in the cloudwatch-agent's cluster role. A crucial update was provided during the Q&A: Kubernetes 1.26 introduced (and 1.36 made GA by default) more granular, safer permissions to replace the broad nodes/proxy permission. This allows observability tools to collect necessary data without granting the ability to execute on arbitrary pods. The onus is now on tool developers to adopt these new, safer permissions.

This deep dive illustrates how specific, seemingly justified permissions, when combined with architectural design choices (like DaemonSets and cluster-wide roles) and deprecated authentication mechanisms, can create critical security vulnerabilities in managed Kubernetes environments.

Demo / Proof of Concept

▶ Watch: EKS authentication modes: API vs. Config Map deprecation (6:30)

While William Taylor's presentation did not include a live demonstration, he clearly walked through the conceptual proof of concept for the identified privilege escalation paths. The "demo" was a detailed walkthrough of the attack flow, illustrating how an attacker could move from an initial compromise to full cluster control.

The primary escalation path demonstrated involved the Amazon CloudWatch observability add-on and its over-privileged cloudwatch-agent service account. The steps outlined were:

1. Initial Compromise: An attacker successfully compromises an external service running within the EKS cluster. This initial foothold could be achieved through various means, such as exploiting a web application vulnerability.
2. Pod-to-Node Escape: From the compromised pod, the attacker manages to escape to the underlying node. This is a common step in Kubernetes attacks and might involve container escape vulnerabilities or misconfigurations.
3. Service Account Token Acquisition: Once on the node, the attacker identifies and acquires the service account token associated with the cloudwatch-agent DaemonSet. Since the cloudwatch-agent runs on every node, its token would be present and accessible on the compromised node.
4. aws-auth ConfigMap Manipulation: Leveraging the cloudwatch-agent's service account token, which possessed cluster-wide update permissions on configmaps, the attacker modifies the aws-auth ConfigMap located in the kube-system namespace.
5. Cluster Admin Privilege Grant: The modification involves adding an entry to the aws-auth ConfigMap that maps an attacker-controlled IAM identity (or a newly created one) to the system:masters group. This action immediately grants the specified IAM identity cluster admin privileges within the EKS cluster.
6. Cluster Compromise: With cluster admin rights, the attacker gains full control over the EKS cluster, capable of deploying, modifying, or deleting any resource, accessing sensitive data, and potentially impacting the entire organization's cloud infrastructure.

Taylor also conceptually demonstrated the nodes/proxy vulnerability. If an attacker gains access to a service account with get nodes/proxy permission (like the cloudwatch-agent in its pre-Kubernetes 1.26 state), they could use this to proxy requests to the kubelet API on any node. This would allow them to execute arbitrary commands on any pod running on that node, bypassing normal RBAC controls for pod execution. While not a direct "cluster admin" grant, it provides a powerful lateral movement and execution capability that can lead to compromise of sensitive workloads or further escalation.

The strength of this "demo" lies in its clarity and directness, outlining a plausible and impactful attack vector that was observed in real-world client environments. It effectively illustrates how the convergence of a deprecated authentication mechanism, an essential operational tool, and overly broad permissions can create a critical security blind spot.

Defensive Implications

▶ Watch: Specifics of the EKS CloudWatch observability add-on (7:50)

The findings presented by William Taylor offer several critical defensive implications for securing Kubernetes clusters, particularly those running on EKS. These recommendations range from immediate remediation actions to broader architectural and operational shifts.

1. Update CloudWatch Add-on: For the specific aws-auth ConfigMap privilege escalation path, the primary mitigation is to ensure the CloudWatch observability add-on is updated to version 5.3 or later. AWS has remediated this issue by switching the cloudwatch-agent's ConfigMap permissions from a cluster-wide role to a namespaced role, preventing it from modifying the aws-auth ConfigMap. Cluster administrators should verify their add-on versions and upgrade proactively.

1. Address nodes/proxy Vulnerability:

• Kubernetes Version Upgrade: The most definitive fix for the nodes/proxy issue is to upgrade your Kubernetes cluster to version 1.26 or higher (GA in 1.36). As clarified during the Q&A, Kubernetes introduced more granular and safer permissions to replace the overly broad nodes/proxy permission.
• Tool Updates: Critically, developers of observability tools (and other DaemonSets) must update their manifests and Helm charts to utilize these new, safer permissions instead of the deprecated nodes/proxy. Cluster administrators should check if the observability tools they use have adopted these changes and encourage vendors to do so.
• Kubelet API Restriction (Cautious Approach): As a temporary or supplementary measure, restricting traffic to the kubelet API port within the cluster might mitigate the nodes/proxy exploit. However, Taylor warns that this could have significant knock-on effects on legitimate cluster operations and requires thorough testing and research before implementation. This is a complex mitigation as Kubernetes itself has historically marked nodes/proxy as a "won't fix" feature in older versions, meaning awareness and careful management are key.

1. Understand Your Threat Model: It's paramount for cluster owners to understand the threat model of their specific Kubernetes environment. While an escalation path might exist, its acceptability depends on the overall security posture. If perimeter defenses are extremely strong and there are no external services, the risk might be deemed acceptable. Conversely, highly sensitive data or critical applications necessitate stricter controls.

1. Scrutinize Managed Services and Add-ons: The talk highlights that vulnerabilities can be introduced even by managed services and official add-ons. The aws-auth ConfigMap issue, for instance, was not a user misconfiguration but an out-of-the-box vulnerability. Organizations should avoid blindly trusting managed services and always investigate the underlying permissions and configurations of any add-on or tool deployed to their cluster.

1. Observe and Audit Privileges: Regularly audit the permissions of all workloads, especially those deployed as DaemonSets. These tools, by their nature, run on every node and often require elevated privileges, making them prime targets for attackers. Pay close attention to observability tools, as their need for comprehensive data access often translates to broader permissions. Tools like kubeaudit or polaris can help identify over-privileged service accounts.

1. Enforce Principle of Least Privilege for DaemonSets: While challenging, strive to apply the principle of least privilege to DaemonSets. If possible, separate high-privilege functions into management pods and minimize the permissions of the agents running on individual nodes. For observability tool developers, this means designing architectures where the management component handles sensitive control operations, while agents only collect data with the absolute minimum necessary permissions.

1. Implement Defense in Depth: Taylor strongly advocates for defense in depth. While an attacker might find an escalation path, multiple layers of security controls (e.g., pod security policies, network policies, API security, perimeter protection, container runtime security) can prevent them from reaching or fully exploiting highly privileged targets. The goal is to ensure that even if one control fails, others are in place to limit the impact.

1. Deprecation Awareness: Be aware of deprecated features, such as the aws-auth ConfigMap authentication mode. While they might still function, they often come with known security risks or are not actively maintained to address new threat vectors. Migrate to recommended, modern alternatives (e.g., EKS access entries) whenever possible.

By implementing these defensive strategies, organizations can significantly reduce their exposure to privilege escalation attacks within Kubernetes, transforming residual risks into manageable ones.

Key Takeaways

• Observability Tools vs. Least Privilege: There's an inherent conflict between the need for comprehensive observability (often requiring broad permissions via DaemonSets) and the security principle of least privilege in Kubernetes.
• Managed Services Aren't Always Secure Out-of-the-Box: Even official add-ons and managed services (like the EKS CloudWatch add-on) can introduce critical vulnerabilities if their default permissions are overly broad, requiring diligent scrutiny from cluster owners.
• aws-auth ConfigMap is a High-Value Target: When used, the aws-auth ConfigMap is a critical component for IAM-to-RBAC mapping in EKS. Any workload with cluster-wide update permissions on ConfigMaps can potentially grant itself cluster admin privileges by modifying this map.
• nodes/proxy is a Persistent Risk: The get nodes/proxy permission, common in observability tools, can enable arbitrary command execution on pods via the Kubelet API. While Kubernetes 1.26+ offers safer, granular alternatives, older clusters and un-updated tools remain vulnerable.
• Prioritize DaemonSet Permissions: DaemonSets, by running on every node, are high-impact components. Their service account permissions must be meticulously reviewed and minimized, especially for observability agents.
• Defense in Depth is Non-Negotiable: A multi-layered security approach is essential. Even if an attacker finds an escalation path, other security controls (network policies, pod security, runtime security) should be in place to limit their lateral movement and overall impact.

About the Speaker(s)

William Taylor is a Security Consultant at Reverse. With approximately 15 years of experience in technology, he spent about a decade as an engineer before transitioning into security five years ago. He humorously describes his career shift by saying he "used to make things and now I break them ethically of course." His background as an engineer provides him with a deep understanding of system architecture and functionality, which he now applies to identifying and analyzing security vulnerabilities in complex environments like Kubernetes.