Command and KubeCTL: Kubernetes Security for Pentesters and Defenders

Command and KubeCTL: Kubernetes Security for Pentesters and Defenders

Speakers: Mark Manning

Conference: Cloud Village

YouTube: https://www.youtube.com/watch?v=FsfnA4OT5QQ

Overview

In this Cloud Village talk, Mark Manning, a seasoned expert in container security, delves into critical, real-world attack scenarios targeting Kubernetes environments, specifically focusing on vulnerabilities within container registries. The presentation aims to equip both pentesters with effective offensive strategies and defenders with actionable insights to harden their Kubernetes clusters. Manning challenges traditional network pentesting approaches when applied to ephemeral containerized systems, advocating for a more "operating system-like" perspective on Kubernetes security.

Manning highlights two primary attack vectors: the recovery of sensitive information from "deleted" files within container images using white out files, and the poisoning of trusted container images in registries to achieve widespread compromise. He introduces regge pillage, a new tool designed to uncover these hidden vulnerabilities, alongside demonstrating a proof-of-concept for backdooring HashiCorp Vault images. The talk underscores the often-overlooked supply chain risks associated with container registries and provides practical steps for mitigation, emphasizing the importance of robust image signing and verification practices in modern CI/CD pipelines.

The core message resonates with the evolving landscape of cloud-native security, urging practitioners to move beyond superficial network scans and understand the intricacies of container orchestration. Manning’s insights are particularly relevant for organizations struggling with shadow IT, misconfigured staging environments, or those seeking to mature their software supply chain security. By exposing these prevalent weaknesses and offering concrete defensive strategies, the talk serves as a vital guide for anyone involved in securing Kubernetes deployments.

Background

▶ Watch: Introduction and talk overview (0:00)

Mark Manning begins by reframing Kubernetes not merely as a collection of IPs and services, but as an operating system for the cloud. In this analogy, the Kubernetes API acts as the kernel, managing security controls and RBAC (Role-Based Access Control). This perspective is crucial because, unlike traditional network pentesting where fixed IP addresses and long-lived services are targets, Kubernetes components like pods are ephemeral. A reported vulnerability on an EngineX instance at IP 1.2.3.4 might be irrelevant minutes later as the pod is redeployed. Understanding Kubernetes as a dynamic, API-driven system is fundamental to effective security.

The talk then transitions to the pervasive problem of misconfigured container registries. Manning notes that during reconnaissance phases of a penetration test, tools like Shodan or Censys frequently reveal publicly exposed registry information. A common scenario involves organizations establishing a secure, private registry for production but failing to extend the same rigor to staging or development environments. This often leads to "shadow IT" registries or inadequately secured staging registries, sometimes even with public access due to the perceived difficulty of managing authentication for multiple development teams. These less secure registries become prime targets for attackers seeking to inject malicious code into the software supply chain.

Container registries themselves are described as sophisticated "blob stores with a front end," often built upon object storage like S3. While the Docker Registry API provides basic functionality, it leaves critical security features, such as garbage collection and tag immutability, as optional implementations. This design choice has led to a disparity in security capabilities across different registry solutions. For instance, Harbor is cited as an example of a registry that has "bolted on a bunch of security controls," whereas many free or less mature registries lack these essential protections. This foundational lack of inherent security features in some registries, combined with common operational oversights, creates significant attack surfaces that Manning proceeds to explore.

Key Findings

▶ Watch: Real-world scenario: insecure public registries (2:20)

The talk uncovers several critical findings related to Kubernetes security, particularly concerning container registries and image integrity:

1. Recovery of "Deleted" Files via White Out Files: Manning demonstrates that files "removed" from a Docker image during its build process are not truly erased from the image's layers. Instead, the overlay file system mechanism uses white out files (prefixed with .wh.) to logically hide these files. This means sensitive data, such as private keys or configuration files, copied into an image and then supposedly deleted, can still be extracted from previous layers within the registry. This represents a significant, often overlooked, data leakage vulnerability.

1. Registry Write Access as a Critical Attack Vector: The presentation highlights that gaining write access to a container registry, whether through exposed tokens, compromised CI environments, or misconfigured permissions, can lead to devastating supply chain attacks. Attackers can poison trusted images, replacing legitimate software with backdoored versions.

1. Image Poisoning of Ubiquitous Services (e.g., HashiCorp Vault): Manning illustrates a potent attack where a backdoored HashiCorp Vault image is pushed to a compromised registry. Given that Vault often runs as an init container on every Kubernetes node to manage secrets, poisoning its image allows an attacker to compromise both the Vault server and individual pods, exfiltrating all secrets accessed by the service. This "two-for-one deal" showcases the high impact of targeting foundational infrastructure components.

1. Exploiting Tag Immutability Weaknesses: A key enabler for image poisoning is the lack of tag immutability in many registries. This allows attackers to overwrite existing image tags (e.g., hashicorp/vault:latest or even specific version tags that resemble digests) with their malicious versions, even if the underlying digest changes. This bypasses a critical integrity check that many users might assume is in place.

1. The "Kaminsky Method" for Covert Data Exfiltration: Manning introduces a technique, dubbed the "Kaminsky method" (a shout-out to Dan Kaminsky), involving the exfiltration of secrets over DNS. By embedding malicious code in a backdoored image that sends sensitive data to a custom DNS server (Xfill.antree.com), attackers can bypass traditional network monitoring that might not scrutinize DNS queries as closely for data exfiltration.

1. Disparity Between Perceived and Real-World Risks: Manning concludes by challenging the common perception of Kubernetes security risks. While "sexy" exploits like container breakouts are often highlighted, real-world compromises more frequently stem from fundamental issues like overprivileged RBAC, misconfigurations, cryptojacking, or even accidental kubectl delete commands in the wrong context. He stresses the importance of focusing defensive efforts on these more probable and impactful vulnerabilities.

Technical Deep Dive

▶ Watch: Registry architecture and common vulnerabilities (3:50)

Recovering Hidden Files with White Out Files

The mechanism of container image layers is central to understanding the white out file vulnerability. When a Docker image is built from a Docker file, each command (e.g., FROM, COPY, RUN) typically creates a new layer. These layers are essentially tarballs representing changes to the file system. If a file is added, a new layer is created containing that file. If a file is modified, a new layer is created with the updated version.

The challenge arises when a file is "deleted." The overlay file system, which combines these layers to present a single, coherent file system to the running container, needs a way to know that a file from a previous layer should no longer be visible. This is achieved through white out files. When a RUN rm <file> command is executed, a new layer is created containing a special file named .wh.<file> (e.g., .wh.env.file). The overlay file system interprets this special prefix to mean "hide the file named <file> from any underlying layers." While the file is hidden from the running container, the original file still exists in its respective layer within the registry.

Manning introduces regge pillage, a tool he co-developed (initially in Python, then rewritten in Golang by Josh Mackin), specifically designed to exploit this behavior. regge pillage can be pointed at a container registry and instructed to hunt for these .wh. files. Upon finding a white out file in a specific layer (e.g., layer 5), it then traverses previous layers (4, 3, 2, 1) to locate and extract the original, "deleted" file. This provides a targeted method for recovering sensitive data that developers believed was securely removed.

As a practical example, Manning describes a Docker file that copies a signing_key and a docker_config file, uses them for a signing operation, and then explicitly removes them. A typical thought process might be: "I needed the private key for signing, but I don't want it stored in the final container, so I'll delete it." However, by using regge pillage with its white out file hunting feature, an attacker can extract these supposedly deleted files, potentially compromising the signing key and other credentials. The tool also includes filters to remove noise from common cleanup operations like apt remove cache files, focusing on deliberately hidden sensitive data.

Registry Image Poisoning and Secret Exfiltration

The second major technical deep dive focuses on exploiting write access to container registries to poison trusted images. Manning targets HashiCorp Vault, a ubiquitous open-source secret management engine, often deployed with an init container on every node in a Kubernetes cluster. This architecture makes Vault an attractive target, as compromising its image can lead to control over both the Vault server and the secrets accessed by individual pods.

The attack leverages a common misconfiguration or lack of feature: the absence of tag immutability in container registries. Without tag immutability, an attacker with write access can overwrite an existing image tag (e.g., hashicorp/vault:latest or even a version tag like hashicorp/vault:1.9.0) with a malicious image, even if the underlying digest (the SHA256 hash of the image content) changes. This can happen if a CI environment is compromised or if a staging registry has overly permissive write permissions.

Manning's proof-of-concept involves creating a backdoored Vault image. The malicious payload is a simple patch that adds a function to exfiltrate all secrets accessed by Vault. This exfiltration is performed using what he terms the "Kaminsky method," which involves sending the secrets over DNS to a custom, attacker-controlled DNS server (Xfill.antree.com). The speaker notes that the specific tool used for this is a modified version of the Exodus custom DNS server. He even openly states that this backdoored image is hosted on DockerHub, underscoring the risk of blindly trusting public registries. The simplicity of the patch demonstrates that sophisticated covert channels are not always necessary for effective compromise, especially when targeting common infrastructure.

The attack flow is straightforward:

1. Attacker gains write access to the registry.
2. Attacker pushes a backdoored hashicorp/vault image, overwriting a commonly used tag (e.g., latest).
3. A Kubernetes cluster's Kublet pulls the compromised image (e.g., for a new Vault server deployment or an init container on a pod).
4. The backdoored Vault instance runs, and as it accesses secrets, it exfiltrates them via DNS queries to the attacker's server.

Post-Exploitation Tools for Kubernetes

Once initial access is gained, Manning highlights several essential tools for navigating and further exploiting a Kubernetes cluster:

• K9s: Described as the "best tool" for interacting with Kubernetes. It's a statically compiled text-based UI that allows users to easily view pods, logs, and drop into shells. Its static compilation makes it ideal for "living off the land" scenarios during a pentest, as it doesn't require installing dependencies on the cluster. It also offers features for auditing RBAC.

• Krew: This is the plugin system for kubectl, extending its functionality with over 500 plugins. Manning mentions net-forward for reverse shell capabilities and access-matrix for RBAC auditing (specifically, the who-can functionality to determine which entities can perform specific actions).

• Paradis: Presented as the "Metasploit of Kubernetes," Paradis (made by Guardians and Jay Beiel, among others) is an open-source exploitation framework for Kubernetes clusters, offering point-and-shoot exploitation capabilities.

These tools collectively provide powerful capabilities for both offensive and defensive operations within a Kubernetes environment, from initial reconnaissance and exploitation to post-compromise lateral movement and privilege escalation.

Demo / Proof of Concept

▶ Watch: Introducing the 'Regge Pillage' security tool (5:00)

Mark Manning presented two distinct proof-of-concept demonstrations to illustrate his key findings:

1. Recovering White Out Files with regge pillage:

• Setup: A Docker file was constructed that intentionally copied sensitive files (a signing_key and a docker_config) into an image during the build process. Immediately after, a RUN rm command was used to delete these files, simulating a common developer practice to "clean up" sensitive data from the final image.
• Execution: The image was built and pushed to a local registry.
• Attack: Manning then demonstrated regge pillage targeting this image in the registry. The tool was configured to specifically hunt for white out files and extract their original counterparts from underlying layers.
• Result: The demonstration successfully showed regge pillage identifying and extracting both the signing_key and the docker_config files, proving that even "deleted" files remain recoverable from container image layers if not handled correctly. This visually confirmed the vulnerability of sensitive data persisting in image history.

1. Backdooring HashiCorp Vault and Exfiltrating Secrets:

• Setup: A backdoored version of HashiCorp Vault was prepared. This modified Vault image included code to exfiltrate tokens and other secrets via DNS queries to an attacker-controlled domain (Xfill.antree.com). The speaker mentioned that this malicious image is publicly available on DockerHub, highlighting the ease with which such attacks can be mounted.
• Execution: A Kubernetes cluster was set up to run the backdoored Vault server. The demonstration showed the server starting and performing its normal operations, but in the background, it was quietly sending data.
• Attack: On the attacker's side, a DNS server was configured to listen for incoming queries to Xfill.antree.com.
• Result: As the backdoored Vault server initialized and processed requests (which would involve accessing secrets), the attacker's console displayed a stream of exfiltrated tokens and other sensitive information, captured directly from the DNS queries. This vividly illustrated the effectiveness of image poisoning combined with DNS-based exfiltration, demonstrating how an attacker can gain access to critical secrets within a compromised Kubernetes environment.

Both demonstrations provided concrete evidence of the vulnerabilities discussed, transitioning from theoretical concepts to practical, repeatable attack scenarios, thereby reinforcing the urgency of implementing robust defensive measures.

Defensive Implications

▶ Watch: Regge Pillage features: secrets and misconfiguration hunting (6:00)

The vulnerabilities highlighted by Mark Manning – particularly the persistence of "deleted" files in image layers and the risk of registry poisoning – demand specific and robust defensive strategies for Kubernetes environments.

Mitigating White Out File Vulnerabilities

To prevent the recovery of sensitive files via white out files, defenders should:

• Consolidate COPY and RUN rm Commands: The root cause of the white out file issue is having the COPY command (introducing a sensitive file) and the RUN rm command (deleting it) in separate layers of the Docker file. By combining these operations into a single RUN command, the sensitive file exists only within the temporary build context of that single layer and is effectively cleaned up before a new layer is committed. For example, instead of:

The file is created and deleted within the same layer, preventing its persistence in previous layers.

• Utilize Multi-Stage Builds: Multi-stage builds are a powerful technique where intermediate build stages, which might contain sensitive build artifacts or tools, are discarded. Only the final, stripped-down artifacts are copied into the final production image. This significantly reduces the attack surface by ensuring that build-time secrets or temporary files never make it into the deployable image.
• Use ARG for Sensitive Build Arguments: For credentials or tokens needed only during the build process, consider passing them as build arguments using ARG and docker build --build-arg. These arguments are not persisted in the final image layers, unlike files copied into the image.

Securing Container Registries and Supply Chain

To counter image poisoning and ensure the integrity of container images, several measures are critical:

• Implement Image Signing and Verification: This is paramount for software supply chain security. Manning strongly advocates for signing all container images and verifying these signatures before deployment. Tools like Cosign (part of the Sigstore project) offer keyless signing, which simplifies the signing process by not requiring long-lived private keys to be managed manually. This ensures that images have not been tampered with since they were built and signed by a trusted entity.
• Enforce Out-of-Band Signing: The signing operation itself must be performed outside the normal build process. If the build environment is compromised, and the signing key is present during the build, an attacker can sign their malicious image with the legitimate key. Integrating with frameworks like SLSA (Supply-chain Levels for Software Artifacts) can help define and enforce secure build and signing processes, separating concerns to protect critical signing material.
• Enable Tag Immutability: Registries should be configured to enforce tag immutability. This feature prevents existing image tags from being overwritten, ensuring that once an image is published with a specific tag (e.g., hashicorp/vault:1.9.0), that tag always points to the original, untampered digest. This directly mitigates the image poisoning attack demonstrated with HashiCorp Vault.
• Implement Strong Registry Authentication and Authorization: Access to container registries must be tightly controlled with robust authentication and authorization mechanisms. Least privilege principles should be applied, ensuring that only necessary entities (e.g., CI/CD pipelines) have write access, and human users typically only have read access.
• Avoid the latest Tag in Production: While convenient for development, using the latest tag in production is highly risky. It's mutable and can easily be overwritten with a new, potentially malicious, image. Always pin to specific, immutable image digests or version tags that are verified.

General Kubernetes Security Best Practices

Beyond registry-specific defenses, Manning emphasizes broader Kubernetes security principles:

• Address Overprivileged RBAC: This is cited as the most common and impactful real-world vulnerability. Regularly audit and refine RBAC policies to ensure users and service accounts only have the minimum necessary permissions. Tools like access-matrix (a kubectl plugin) can aid in this.
• Improve kubectl Context Management: Prevent accidental deletions or modifications by implementing safeguards for kubectl usage, such as requiring explicit context confirmation or using separate contexts for production environments.
• Monitor for Cryptojacking: While not as "sexy" as other exploits, cryptojacking remains a prevalent real-world threat. Implement robust monitoring for unusual resource consumption or outbound network traffic patterns from pods.

By diligently implementing these defensive measures, organizations can significantly enhance the security posture of their Kubernetes clusters and protect their software supply chain from sophisticated attacks.

Key Takeaways

• Hidden Data in Layers: Files "deleted" from Docker images are often recoverable from earlier layers via white out files, posing a significant risk for exposed credentials or sensitive data.
• Registry as a Critical Attack Vector: Misconfigured or compromised container registries with write access are prime targets for image poisoning, allowing attackers to inject malicious code into the software supply chain.
• Image Signing is Non-Negotiable: Implementing robust image signing and verification using tools like Cosign and adhering to frameworks like SLSA is crucial to ensure the integrity and authenticity of container images before deployment.
• Specialized Tools are Essential: Effective Kubernetes security (both offensive and defensive) requires specialized tools like regge pillage for registry analysis, K9s for cluster interaction, Krew for kubectl extensions, and Paradis for exploitation.
• Prioritize Real-World Risks: While advanced exploits are interesting, defenders should focus resources on mitigating common, high-impact risks such as overprivileged RBAC, misconfigurations, and cryptojacking, which are far more prevalent in real-world compromises.
• Enforce Registry Best Practices: Utilize tag immutability, strong authentication, and avoid mutable tags like latest in production to prevent image tampering and ensure a secure supply chain.

About the Speaker(s)

The talk was delivered by Mark Manning, a distinguished professional with a deep background in container security. He gained significant experience in the field while working for a pentesting company specializing in container security. Following this, Mark contributed to building and hardening the secure Kubernetes offering at Snowflake. His expertise also extends to secure build technologies, having worked with G Visor and QEMU. Currently, Mark Manning is working at Janguard, where he focuses on securing containers from the ground up and strengthening the overall software supply chain.