Don't trust Rufus, he's a mole - introducing KIEMPossible

Don't trust Rufus, he's a mole - introducing KIEMPossible

Speakers: Golan Myers, Security Researcher, Palo Alto Networks

Conference: Cloud Village

YouTube: https://www.youtube.com/watch?v=8IK7bxDbUOM

Overview

In his Cloud Village talk, "Don't trust Rufus, he's a mole - introducing KIEMPossible," Golan Myers, a Security Researcher at Palo Alto Networks, delves into the intricate and often opaque world of Kubernetes identity and access management (IAM). The presentation highlights a pervasive challenge for organizations: gaining clear visibility into who (or what) has what permissions within their Kubernetes clusters, what permissions they actually need, and how these permissions are truly being utilized. Myers introduces KIEMPossible, an open-source Golang-based tool designed to provide comprehensive Kubernetes Infrastructure Entitlement Management, helping security teams and DevOps engineers untangle this complexity.

The core problem, as framed by Myers, is the difficulty in identifying all entities with access, understanding their full permission sets (including those inherited indirectly), and distinguishing between granted permissions and those actively used. This lack of clarity creates a significant identity-related attack surface, making privilege escalation a common vulnerability in Kubernetes environments. KIEMPossible aims to address this by automating the collection, normalization, and analysis of Kubernetes IAM configurations and audit logs across various cloud providers and self-managed clusters.

This research and the accompanying tool are critical for modern cloud-native security. As Kubernetes adoption grows, managing access effectively becomes paramount. Over-privileged identities, whether human or machine, represent a prime target for attackers seeking to move laterally or escalate privileges. By shedding light on this hidden attack surface, KIEMPossible empowers defenders to implement a stronger least-privilege posture, reduce the blast radius of potential breaches, and ultimately enhance the overall security of their Kubernetes deployments.

Background

▶ Watch: Introducing the KIEMPossible tool and its flow (0:29)

The talk begins by illustrating a common pain point: a DevOps engineer, Kim, overwhelmed by security alerts stemming from an inability to understand the permissions granted to various identities in her Kubernetes clusters. This scenario underscores the fundamental challenge of Kubernetes IAM.

Kubernetes inherently provides several IAM objects:

• Service Accounts: These are the only identity objects that exist natively within a Kubernetes cluster.
• Users and Groups: While these can be referenced in bindings, they do not exist as explicit objects within Kubernetes itself, posing a significant discovery challenge.

Permissions are defined through:

• Roles and ClusterRoles: These are analogous to policies in AWS, holding the actual permissions (verbs like get, create, delete on resources like pods, secrets).
• RoleBindings and ClusterRoleBindings: These scope the permissions defined in Roles/ClusterRoles to specific identities (Service Accounts, Users, Groups) either at a namespace level (RoleBindings) or cluster level (ClusterRoleBindings).

Adding another layer of complexity are the cloud integrations that bridge Kubernetes identities with external cloud IAM systems. Examples include:

• AWS IAM (historically via aws-auth ConfigMap, more recently EKS Access Entries and Access Policies).
• GCP IAM for GKE.
• Azure AD for AKS.

Myers identifies three core problems that motivated the development of KIEMPossible:

1. Identifying Entities: How do we discover all users and groups with access, especially since they aren't native Kubernetes objects? The crucial "shadow user" problem arises when users gain permissions solely through group membership, and those groups are referenced in bindings, but the users themselves are never directly listed.
2. Identifying Permissions: How can we accurately determine the full set of permissions an entity possesses, considering wildcard definitions and the hierarchical nature of Kubernetes resources? Myers emphasizes the need to "flatten" permissions to their lowest possible level for practical analysis. This involves distinguishing between ephemeral resource permissions (e.g., pods, which can be terminated and re-created with different names, thus requiring namespace-level scope) and non-ephemeral resource permissions (e.g., secrets, which are static and can be scoped to specific resource names). Special verbs like bind, impersonate, and escalate also require careful consideration due to their powerful implications. Sub-resources (e.g., nodes/proxy as a sub-resource of nodes) further complicate permission mapping, as permissions on a parent resource implicitly grant access to its sub-resources, but not vice-versa.
3. Identifying Needed Permissions: Beyond what an entity can do, what does it actually need to do? This is critical for implementing least privilege. The proposed solution relies on analyzing audit logs to observe actual actions. This approach builds upon prior research, such as Daniel Gallalik's work on permissions that grant more permissions (e.g., exec into a pod with a mounted service account token), and Shay from Wiz's earlier talk on Kubernetes logging, which highlighted the challenges and inconsistencies across different cloud providers.

The speaker emphasizes that while machine identities (e.g., CI/CD pipelines, controllers) often have predictable access patterns, human identities are sporadic. He also briefly touches on the emerging challenge of AI potentially breaking these predictable patterns, a new area for security research.

Key Findings

▶ Watch: Understanding Kubernetes IAM objects and integrations (1:47)

The research underpinning KIEMPossible uncovered several critical findings regarding Kubernetes IAM and security:

• Lack of Native User/Group Enumeration: Kubernetes, by default, provides no direct API to list all users or group members. While service accounts are explicit objects, human users and external groups are only referenced in role bindings. This makes comprehensive identity discovery challenging.
• The "Shadow User" Problem: Users who derive their permissions solely through membership in a group that is then referenced in a RoleBinding or ClusterRoleBinding are effectively invisible to direct API queries. Their existence and permissions can only be inferred through audit logs by examining the groups claim within authentication material.
• Audit Logs as the Single Source of Truth for Usage: To determine needed permissions and identify "shadow users," audit logs are indispensable. They record actual actions, allowing for a comparison against granted permissions and revealing identities not directly discoverable via API calls.
• Inconsistent and Limited Audit Log Visibility in Managed Clusters: While self-managed Kubernetes clusters offer granular control over audit logging policies, managed cloud Kubernetes services (EKS, AKS, GKE) often provide limited visibility and control over what is logged. GKE, for instance, notably removes the groups claim from its logs, severely hindering group inheritance analysis for its users.
• Direct Kubelet Interactions Bypass API Server Security: Permissions like get and create on nodes/proxy allow direct interaction with the Kubelet on a node. This is a significant security blind spot because many authorization and logging checks are performed at the API server level. Direct Kubelet communication can bypass these checks and not be logged, allowing an attacker to perform actions on a node without leaving a trace in the standard audit logs.
• The Identity-Related Attack Surface is Vast: Myers unequivocally states that the identity-related attack surface in Kubernetes is "huge," and he has rarely encountered a cluster where privilege escalation wasn't possible through some form of identity compromise or misconfiguration. This underscores the widespread nature of over-privileging and the critical need for better IAM visibility.
• AI's Potential Impact on Identity Predictability: The introduction of AI into automated systems could fundamentally alter the predictable patterns often seen with machine identities, posing new challenges for security analysis based on historical log data.

Technical Deep Dive

▶ Watch: Challenge: Discovering users and group permissions effectively (2:58)

KIEMPossible's technical approach systematically addresses the challenges of Kubernetes IAM visibility by combining API introspection with audit log analysis. The tool, written in Golang, functions as a Kubernetes Infrastructure Entitlement Management solution.

Identifying Entities

1. Service Accounts: These are straightforward. As native Kubernetes objects, they can be listed directly using the Kubernetes API.
2. Users and Groups: The tool enumerates all RoleBindings and ClusterRoleBindings in the cluster. Subjects within these bindings are parsed to identify users and groups explicitly granted permissions.
3. "Shadow Users" and Group Inheritance: This is a crucial aspect. Since Kubernetes doesn't provide a way to list users within groups, KIEMPossible relies on audit logs. Most Kubernetes distributions include a groups claim in authentication materials within their audit events. By analyzing these logs, the tool can identify users who are part of a group and inherit permissions from it, even if they aren't directly specified in any binding. A significant caveat here is GKE, which omits the groups claim from its logs, making group inheritance analysis impossible for those clusters.

Identifying Permissions

The goal is to "flatten" permissions to the lowest possible, actionable level:

1. Enumeration and Normalization: KIEMPossible collects all Roles and ClusterRoles and processes their rules. Wildcard permissions (e.g., * on pods) are expanded into all relevant verbs and resources.
2. Scope Determination:

• For ephemeral resources (e.g., pods, deployments), the lowest practical scope is the namespace level, as their names can change frequently.
• For non-ephemeral resources (e.g., secrets, ConfigMaps), permissions can be scoped down to specific resource names. It's important to note that resource names require an exact match; no wildcards or regular expressions are supported for resource name scoping in Kubernetes RBAC.

1. Special Verbs: Verbs like bind, impersonate, and escalate are given special attention due to their ability to grant or expand privileges.
2. Sub-resources: The tool correctly handles sub-resources. If an entity has a permission (e.g., get) on a parent resource (e.g., nodes), it implicitly gains that permission on its sub-resources (e.g., nodes/proxy). However, the reverse is not true; a permission on a sub-resource does not grant permission on the parent.

Identifying Needed Permissions (Last Access)

This is achieved by integrating with and analyzing audit logs:

1. Log Ingestion:

• For self-managed clusters, KIEMPossible processes a specified audit log file.
• For cloud providers, it integrates with their respective logging services (e.g., AWS CloudWatch, Azure Monitor, GCP Cloud Logging).

1. Log Normalization: A critical step is normalizing the diverse log formats and structures across different providers to extract consistent information about actions, users, and resources.
2. Comparison and Mapping: The tool compares the permissions discovered from Kubernetes API objects (the "database permissions") with the actions observed in the audit logs ("log actions"). This allows it to:

• Identify which granted permissions are actually being used.
• Populate a last used time and last used resource for each permission.
• Identify permissions that have never been used, indicating potential over-privileging.
• Address the challenge of matching namespace-level ephemeral resource permissions (from the API) to specific pod actions (in logs).

KIEMPossible Tool Workflow

The tool's execution flow is comprehensive:

1. Cluster Type Detection & Authentication: Determines if it's a self-managed, AKS, EKS, or GKE cluster and authenticates accordingly (kubeconfig for self-managed, cloud provider SDKs for managed).
2. IAM Enumeration: Gathers all Roles, RoleBindings, ClusterRoles, ClusterRoleBindings, and ServiceAccounts.
3. Permission Calculation: Flattens and scopes all identified permissions.
4. Workload Collection (Optional): If configured, it collects information about all workloads (pods, deployments, etc.), mapping them to their respective Service Accounts and their original owners (e.g., a pod to its deployment). This aids in triage.
5. Log Ingestion: Connects to the appropriate logging service or reads the audit file.
6. AWS Access Entries Handling: Specifically processes the top five main AWS Access Entries, which define static permissions external to Kubernetes RBAC.
7. Group Inheritance: Uses the groups claim from audit logs to resolve group memberships and assign inherited permissions (excluding GKE).
8. Database vs. Log Comparison: Matches granted permissions with observed log actions to determine usage.
9. Output Generation: Populates last used time and last used resource fields. If the advice flag is passed, it provides an analysis of the riskiest identities and permissions.

The output consists of two main tables: one detailing cluster permissions with last used times and resources, and another providing workload identity mapping, linking workloads to their service accounts.

Demo / Proof of Concept

▶ Watch: Proposed approach for identifying entities and permissions (4:34)

During the talk, Golan Myers presented a pre-recorded demo of KIEMPossible in action. While the live demonstration faced technical difficulties, the intent was to showcase the tool's output and operational flow. The demo was described as "14,000 times sped up," indicating the volume of data processed and the efficiency of the tool in presenting its findings.

The demonstration aimed to visually represent the two primary outputs:

1. Cluster Permissions Table: This table would display a comprehensive list of all identified identities, their granted permissions (flattened to the lowest possible level), and crucial metadata such as the last used time and last used resource derived from audit logs. This allows users to quickly identify permissions that are over-privileged or unused.
2. Workload Identity Mapping: This output would illustrate the relationships between running workloads (like pods and deployments) and their associated Kubernetes Service Accounts, and potentially their corresponding cloud service accounts (though the latter is noted as a future extension). This mapping is vital for understanding the true blast radius of a compromised workload.

The demo was designed to convey the practical utility of KIEMPossible in providing tangible, actionable insights for minimizing the identity-related attack surface in Kubernetes environments.

Defensive Implications

▶ Watch: Flattening Kubernetes permissions to the lowest level (6:13)

The insights and capabilities offered by KIEMPossible have profound implications for strengthening Kubernetes security postures. Defenders should consider the following actions:

1. Implement a Strong Least Privilege Model: Leverage tools like KIEMPossible to gain granular visibility into actual permission usage. Regularly review and revoke unused or excessive permissions for both human and machine identities. This is paramount to minimizing the potential impact of a compromised identity.
2. Prioritize Audit Log Monitoring: Recognize that audit logs are a critical source of truth for understanding actual behavior and identifying "shadow users." Ensure comprehensive audit logging is enabled (where controllable) and that logs are ingested into a security information and event management (SIEM) system or a dedicated logging solution for continuous monitoring and analysis.
3. Understand Cloud Provider Logging Nuances: Be acutely aware of the differences in audit log content and accessibility across managed Kubernetes services (EKS, AKS, GKE). Specifically, the absence of the groups claim in GKE logs means group-based inheritance is invisible through this mechanism, necessitating alternative strategies for GKE environments.
4. Be Wary of Direct Kubelet Interactions: Understand that permissions granting access to nodes/proxy allow direct interaction with the Kubelet, bypassing the Kubernetes API server's authorization and logging mechanisms. Audit third-party tools and internal processes that utilize nodes/proxy to ensure they are legitimate and to identify potential blind spots. Consider implementing node-level logging or host-based security tools to detect suspicious Kubelet activity.
5. Regularly Scan for Identity Misconfigurations: Treat identity and access management as a dynamic security surface. Conduct periodic scans with tools like KIEMPossible to detect permission drift, newly created shadow users, or changes in usage patterns that might indicate over-privileging or compromise.
6. Educate Teams on Identity Hygiene: Foster a culture of security awareness among developers and operations teams regarding the creation and management of Kubernetes identities. Emphasize the risks of broad permissions and the importance of using dedicated service accounts for applications.
7. Leverage Open-Source Solutions: Actively explore and integrate open-source tools like KIEMPossible into your security toolkit. Community-driven efforts often address niche, yet critical, security challenges that commercial products might overlook.
8. Anticipate Emerging Threats: As highlighted by the speaker, the rise of AI could dramatically alter the predictability of machine identity behavior. Defenders should stay informed about new research in this area and adapt their monitoring and analysis strategies accordingly.

By proactively addressing the complexities of Kubernetes IAM, organizations can significantly reduce their attack surface and build more resilient cloud-native environments.

Key Takeaways

• Kubernetes identity management is inherently complex, especially when integrating with cloud provider IAM systems and dealing with indirect group-based permissions.
• Audit logs are indispensable for understanding actual permission usage and discovering "shadow users" who are not directly visible via Kubernetes API objects, though log quality and content vary significantly by cloud provider (e.g., GKE's lack of groups claim).
• Over-privileged identities represent a critical, often unaddressed, attack surface in Kubernetes clusters, making privilege escalation a common vulnerability.
• Tools like KIEMPossible are essential for gaining comprehensive, actionable visibility into who has what permissions, what they actually use, and identifying risky configurations across self-managed and managed Kubernetes environments.
• Direct Kubelet interactions (via nodes/proxy) bypass standard API server authorization and logging, creating a significant blind spot that defenders must specifically monitor for.
• A strong least-privilege approach, coupled with continuous auditing and a deep understanding of logging limitations, is vital for minimizing identity-related risk and securing Kubernetes deployments.

About the Speaker(s)

Golan Myers is a Security Researcher at Palo Alto Networks. His work primarily focuses on cloud security, particularly within Kubernetes environments, with a strong emphasis on identity and access management challenges. As he humorously noted at the beginning of his talk, he's "not very good with trust," a sentiment that clearly underpins his research into uncovering and addressing hidden trust relationships and over-privileges within complex systems like Kubernetes.