Quickstart for a Breach! When Official Installations Expose Your K8 and Your Cloud

Quickstart for a Breach! When Official Installations Expose Your K8 and Your Cloud

Speakers: Michael, Microsoft; Yossi, Microsoft

Conference: Cloud Village

YouTube: https://www.youtube.com/watch?v=wifLvZlAX4A

Overview

This talk, presented by Yossi Suman and Michael Kachinski from the Microsoft Defender for Cloud research team, delves into a critical and often overlooked security vulnerability: default misconfigurations in popular cloud-native applications deployed on Kubernetes. The speakers highlight how even well-known and widely used applications, when installed with their default settings via official channels like Helm charts, can inadvertently expose sensitive data or grant attackers full control over a Kubernetes cluster and potentially the underlying cloud environment.

The core message is that while cloud-native security has matured, a new generation of "out-of-the-box" misconfigurations still poses a severe threat. These issues are not due to user error but stem from the application's default settings, making them highly exploitable, often within minutes of deployment. The research team's findings underscore that misconfigurations, rather than software vulnerabilities, are responsible for over 50% of attacks against containerized applications, demanding a shift in defensive strategies.

This article will explore the specific examples, the methodology used to uncover these flaws, and the profound implications for defenders. It emphasizes the necessity for organizations to scrutinize default configurations, map their external attack surface, and implement robust authentication and behavioral monitoring strategies to counter this pervasive threat.

Background

▶ Watch: Introduction to Cloud-Native Misconfigurations (0:00)

The landscape of cloud-native security has evolved significantly, yet historical patterns of misconfiguration continue to manifest in new forms. The speakers begin by recalling infamous examples from earlier generations of Kubernetes deployments. Helm, the package manager for Kubernetes, in its early versions, included a server-side component called Tiller. Tiller possessed permissive, cluster-wide permissions but critically lacked authentication. This meant any pod within the cluster could communicate with Tiller, allowing an attacker with even limited cluster access to achieve full control. This severe risk ultimately led to Tiller's deprecation. Similarly, older versions of the Kubernetes Dashboard often shipped without authentication, enabling privilege escalation if an attacker gained initial access, especially when users mistakenly exposed it to the internet. Kubeflow, a popular framework for ML pipelines in Kubernetes, also historically suffered from installations that lacked built-in authentication, leading to widespread exploitation campaigns. The common thread among these examples was the combination of high privileges and absent or weak authentication, leaving workloads vulnerable to compromise with minimal user error.

While these specific issues have largely been addressed, the Microsoft research team's work reveals that the problem of "out-of-the-box" misconfigurations persists. These are situations where applications are completely exploitable simply by installing them with their default configurations, without any user mistakes. To understand these vulnerabilities, a brief overview of Kubernetes services and Helm is essential.

Kubernetes Services provide a stable and consistent way to access applications, acting as a permanent front door and load balancer for multiple instances of a workload. There are three primary types discussed:

• ClusterIP: Assigns a virtual IP accessible only within the cluster. Used for internal communication between containers.
• NodePort: Opens a static port on every node in the cluster, making the service reachable from outside the cluster if node networking rules permit.
• LoadBalancer: The most common method for exposing applications to the internet in managed Kubernetes services (e.g., AKS in Azure, GKE in Google Cloud). It automatically assigns a public IP address and uses a cloud provider-managed load balancer. The vast majority of external attacks observed target workloads exposed in this manner.

Helm is the de facto package manager for Kubernetes, utilizing charts—reusable and versioned sets of Kubernetes configuration files. When a Helm chart is installed, these configuration files are deployed to the cluster. Crucially, Helm charts come with default configurations defined in a values.yaml file, which users can override. The problem arises when these default values, unless explicitly overwritten, contain "surprises" that lead to severe security vulnerabilities. This understanding forms the foundation for the research into default misconfigurations.

Key Findings

▶ Watch: Focus: Exploitable Default Installations (Zero Mistakes) (2:30)

The research was ignited in late 2024 when the team identified suspicious, identical network activity across multiple Kubernetes clusters, all running Apache Pinot. This led to the discovery that these clusters were exposing Pinot's controller externally via a LoadBalancer service, with no default authentication. This incident served as a stark reminder that "default misconfigurations are still a thing today in 2025," even in popular applications.

Recognizing the scale of the potential problem, Michael Kachinski initiated a systematic hunt for similar vulnerabilities. The methodology involved scanning GitHub and Artifact Hub for YAML manifests and Helm charts that exposed applications externally. An initial "naive approach" of broadly scanning for all YAMLs exposing LoadBalancer services proved impractical due to the sheer volume and limitations of GitHub's code search.

The strategy was refined to focus specifically on Helm charts, targeting the values.yaml file, which contains default configurations. To overcome GitHub's 1,000-result limit per search, Michael employed creative techniques, such as splitting searches by filtering results based on specific criteria like port numbers (e.g., searching for ports starting with '1', then '2', and so on). This allowed for a much broader collection of potential targets.

In total, the team scanned over 12,000 files belonging to 6,700 unique repositories. From these, more than 300 charts with at least 500 GitHub stars were manually deployed and validated. The key findings from this extensive research revealed several recurring patterns of exploitable misconfigurations:

• Hardcoded Passwords: Some charts included passwords directly baked into the values.yaml file.
• No Authentication at All: Many applications were exposed with no authentication mechanism whatsoever.
• Self-Registration Without Validation: Surprisingly, some applications allowed any external user to self-register and gain access without any form of validation.

These fundamental misconfigurations, often present in charts intended for testing or quick starts, still exposed clusters to significant risk. The research confirmed that the combination of internet exposure and weak or non-existent authentication remains the primary vector for these "out-of-the-box" vulnerabilities.

Technical Deep Dive

▶ Watch: Kubernetes Service Types for Application Exposure (3:45)

The research uncovered several compelling case studies demonstrating how official installations can expose Kubernetes and cloud environments.

Apache Pinot

The initial trigger for the research, Apache Pinot, a data store for low-latency analytics and real-time querying, exemplifies the "exposure + no auth" pattern. When deployed using its official Helm chart from its GitHub repository with default settings, it automatically creates two LoadBalancer services: one for the controller (the "brain" of the framework, handling tables, schema, and orchestration) and another for the broker (handling queries). The critical flaw is that the Helm chart's default configuration explicitly enables these LoadBalancer services, and more importantly, there is no default authentication mechanism. An attacker accessing the exposed controller dashboard can gain full control over the application, query data, and potentially manipulate the data store. The relevant section in the Helm chart allows users to disable this exposure, but it's enabled by default.

Meshery

Meshery is an open-source tool designed for managing service meshes, offering a drag-and-drop interface to create and share templates of resources and applications. It can also integrate with cloud accounts to deploy resources. When deployed on a Kubernetes cluster, Meshery can manage cluster resources, acting as an advanced Kubernetes dashboard. The team found that Meshery, when installed via its official Helm chart (or even its mesheryctl CLI tool), is exposed by default with a LoadBalancer. The most alarming vulnerability is that upon accessing the external IP address, the Meshery dashboard presents a sign-up screen, allowing anyone to self-register without any validation. Once registered and logged in, the user is greeted with a dashboard displaying comprehensive cluster information. Crucially, Meshery's deployment is configured with full cluster admin privileges. During the demo, a pod was created with shared PID and network namespaces with the host, demonstrating the extensive control. Furthermore, if a user has integrated their cloud account with Meshery, an attacker gaining access also inherits those cloud permissions, significantly escalating the potential impact.

Selenium Grid

Selenium is a widely used open-source framework for browser automation and testing, with Selenium Grid enabling parallel test execution across different browsers. The speakers highlighted a history of campaigns targeting Selenium (e.g., by Kado Security, Wiz, and Aqua's perfCDL campaign), prompting the question of why it's so frequently attacked. The answer lies in its default lack of authentication. Many Helm charts and manifests, including one found in the official Kubernetes repository (using a NodePort), expose Selenium externally without requiring any authentication.

Attackers exploit Selenium's functionality to execute arbitrary commands. A lesser-known technique involves using the chrome option set binary method in conjunction with a flag called utility command prefix. This flag allows attackers to prepend malicious commands before the Chrome browser starts, executing a file they've created (e.g., osman.tss, a malicious JavaScript file disguised as CSS). This obscure technique, mentioned only once in 2020 regarding "pawning ABAS browsers," demonstrates attackers' sophisticated knowledge of the platform's inner workings.

Prometheus and Grafana

Prometheus is an open-source monitoring system that collects and stores metrics in a time-series database, while Grafana is a visualization tool that creates interactive dashboards from Prometheus data. The research identified an interesting Helm chart under the Prometheus community repo that deploys both. While Grafana typically requires authentication, the values.yaml file in this chart hardcoded the default password, often as admin/admin or kube-operator. The speakers noted a pending pull request aimed at fixing this issue, indicating awareness but also the persistence of the problem.

A Shodan search revealed tens of thousands of exposed Prometheus and Grafana instances. Prometheus, by design, often assumes it's running on an internal network and doesn't require authentication. While Grafana does, the prevalence of misconfigured default passwords means that even a small percentage of these exposed dashboards (e.g., 0.5-1%) translates to over 1,000 potentially exploitable instances.

Internal Exposure: The Hidden Threat

Beyond external exposure, the talk also shed light on the dangers of internally exposed applications using ClusterIP services. In Kubernetes, by default, every pod can communicate with any other pod; there are no inherent firewalls or isolation between them unless Network Policies are explicitly configured. Namespaces, by themselves, do not provide network boundaries.

To track internally exposed applications, attackers can employ several techniques:

1. Brute-forcing DNS: ClusterIP services have a predictable DNS name format: service-name.namespace. Attackers can use wordlists to discover internal services.
2. Scanning IP Ranges: Attackers can scan the internal ClusterIP range for unique ports associated with common services (e.g., Redis on 6379, PostgreSQL on 5432).
3. Reverse DNS Lookup: This powerful technique, available on managed Kubernetes services like AKS and EKS, allows an attacker from a compromised pod to perform reverse DNS queries on internal IP addresses to discover service names.

Michael demonstrated this with a script that performs reverse DNS scanning across a cluster's IP range, quickly identifying internal ClusterIP services.

• Redis: An exposed Redis instance was found via reverse DNS. The port was open, and the attacker could connect from a "bridge pod" because Redis, until recent versions, did not enforce authentication by default.
• Elasticsearch: The popular Bitnami repo's Elasticsearch chart was shown to have its security features disabled by default. Using reverse DNS from a compromised pod, the service was identified, and a connection was established without authentication.

These internal misconfigurations highlight that initial access to a single pod can quickly lead to widespread privilege escalation within the cluster if internal services are not properly secured.

Demo / Proof of Concept

▶ Watch: Helm Charts: Default Configurations and Surprises (5:40)

Throughout the presentation, the speakers provided compelling demonstrations and proof-of-concept scenarios for their findings:

For Apache Pinot, screenshots illustrated the exposed controller dashboard, showing how an unauthenticated user could interact with the application, query data, and effectively take full control. The Helm chart's values.yaml snippet was displayed, clearly showing the type: LoadBalancer and the absence of authentication settings.

The Meshery demonstration was particularly impactful. Screenshots showed the initial login/sign-up screen, followed by the ability to self-register an account without any validation. Once logged in, the Meshery dashboard was visible, displaying comprehensive cluster information. A key part of the PoC involved using Meshery's drag-and-drop interface to create a new pod with the host's PID and network namespaces shared, illustrating the high level of control. The speaker then queried the application's role, confirming it possessed full cluster admin privileges.

For Selenium Grid, the attack vector was detailed using specific flags and commands. While not a live demo, the explanation of using chrome option set binary and utility command prefix with a malicious script (osman.tss) provided a clear technical PoC of how attackers exploit unauthenticated Selenium instances.

The Prometheus and Grafana case included a screenshot from Shodan, revealing the vast number of exposed instances online. The values.yaml file from the Prometheus community Helm chart was shown, highlighting the hardcoded adminPassword for Grafana, demonstrating how default credentials lead to easy exploitation.

Finally, the internal exposure section included a live-script demonstration. Michael showed a script performing reverse DNS scanning from a simulated "bridge pod" within a cluster, successfully identifying internal ClusterIP services. He then used nmap to confirm open ports on the discovered Redis instance and demonstrated connecting to both the unauthenticated Redis and Elasticsearch (from the Bitnami chart) from the compromised pod, effectively proving the ease of internal lateral movement and privilege escalation.

Defensive Implications

▶ Watch: First Finding: Apache Pino Default Misconfiguration (6:50)

The findings presented by Michael and Yossi have profound implications for security practitioners and organizations operating cloud-native environments.

First, the research unequivocally shows that despite advancements in cloud-native security, exploitable misconfigurations are still prevalent in 2025, even in popular, widely used applications. These issues affect organizations of all sizes, from small startups to large enterprises. The critical distinction is that these misconfigurations often come "out of the box," meaning they are not due to user error but rather to reliance on insecure default settings. This makes it crucial for teams to move beyond simply trusting default configurations and to deeply understand what they are deploying.

The speed of exploitation is another alarming factor. The speakers noted cases where misconfigurations were exploited in less than 10 minutes from deployment, underscoring the urgency of addressing these issues proactively. Furthermore, the focus should not solely be on initial external access; internal misconfigurations, such as unauthenticated ClusterIP services, can facilitate rapid privilege escalation once an attacker gains a foothold within the cluster.

A stark statistic highlights the scale of the problem: more than 50% of attacks against containerized applications originate in misconfigurations, not software vulnerabilities. This means that even organizations with top-tier vulnerability scanners and automated blocking of vulnerable applications are only doing "half of the job." A comprehensive security strategy must equally prioritize configuration hardening.

To mitigate these risks, defenders should focus on two primary areas: internet exposure and authentication.

Mitigations

1. Internet Exposure:

• Map the External Surface: Organizations must maintain a clear and up-to-date map of all applications exposed to the internet. While some exposure is necessary, identifying unintended or misconfigured exposure is critical.
• Monitor Kubernetes Audit Logs: A common pattern observed is users manually changing service configurations from ClusterIP to LoadBalancer for debugging or testing, then forgetting to revert the change. Monitoring Kubernetes audit logs for service configuration changes, specifically from ClusterIP to LoadBalancer, can help identify and remediate these forgotten exposures.

1. Authentication in Cloud-Native Applications:

• Scrutinize Application Manifests: Defenders must meticulously review application manifests for problematic settings, such as allowAnonymousAccess or the presence of default/easy-to-guess credentials.
• Leverage LLMs for Credential Detection: With advancements in Large Language Models (LLMs), it's now easier to identify easy-to-guess credentials within configuration files. Integrating LLM-based scanning into CI/CD pipelines can help catch these before deployment.

Detections

Identifying the exploitation of misconfigurations can be challenging due to the lack of generic rules. However, the speakers presented an insightful approach based on container behavior:

• Predictable Container Activity: Most containers are designed to run a single application and exhibit highly consistent behavior. Large-scale analysis showed that containers run, on average, 4.5 different processes, and less than 1% of containers launch new processes after their initial bootstrap.
• Behavioral Monitoring with eBPF: By tracking the behavior of containers, for example, using eBPF tools, organizations can identify anomalies such as the execution of new or unexpected processes, unusual network connections, or unauthorized command execution. Such deviations from baseline behavior can be strong indicators of misconfiguration exploitation or other threats. This proactive behavioral monitoring provides a robust layer of defense against attacks that leverage default misconfigurations.

Key Takeaways

• Default Misconfigurations Persist: Even in 2025, popular cloud-native applications can be severely insecure by default, leading to "out-of-the-box" exploits without user error.
• Rapid Exploitation: Misconfigured applications can be compromised extremely quickly, often within minutes of deployment, making immediate remediation critical.
• Exposure + Weak Auth = Breach: The primary pattern for exploitation is the combination of internet exposure (e.g., via LoadBalancer services) and weak or non-existent authentication mechanisms.
• Internal Threats Matter: Beyond external exposure, internally exposed services (e.g., ClusterIP services without authentication) pose significant risks for privilege escalation once an attacker gains initial access to the cluster.
• Don't Trust Defaults: Organizations must actively review and harden all default configurations, map their external attack surface, and enforce robust authentication across all deployed applications.
• Behavioral Detection is Key: Since over 50% of container attacks stem from misconfigurations, not vulnerabilities, implementing behavioral monitoring (e.g., using eBPF tools) to detect anomalous container activity is crucial for identifying exploitation attempts.

About the Speaker(s)

The talk was presented by Yossi Suman and Michael Kachinski. Both speakers are members of the research team at Microsoft Defender for Cloud. Their work focuses on uncovering and understanding security threats within cloud-native environments, particularly in Kubernetes, and developing strategies for detection and mitigation.