Testing and Fuzzing the Kubernetes Admission Configuration
Benjamin Koltermann (Security Architect for Cloud and Kubernetes · Avolens), Maximilian Rademacher (Vulnerability Researcher and Developer · Avolens)
TROOPERS23 · Day 2 · Attack & Research · Attack & Research
Overview
In this insightful talk from TROOPERS, Benjamin Koltermann and Maximilian Rademacher of Avolens delve into the critical, yet often overlooked, area of Kubernetes admission controller security. Their presentation, titled "Testing and Fuzzing the Kubernetes Admission Configuration," highlights the inherent power and complexity of admission controllers, which act as crucial gatekeepers for all API requests entering a Kubernetes cluster. The speakers argue that while these controllers are vital for enforcing security policies and compliance, their testing methodologies often lag behind, leaving clusters vulnerable to misconfigurations, bypasses, and even denial-of-service conditions.

Key moments
- 4:10 Understanding Kubernetes Admission Chain and Webhooks
- 6:30 Enforcing Security Policies with Admission Controllers
- 14:40 Fuzzing Kubernetes objects using OpenAPI schemas
- 20:40 Identifying logic bugs and runtime crashes
- 24:20 Fine-tuning fuzzing with KubeFuzz constraint language
- 43:00 Two admission controller bugs (logic and crash) revealed
- 50:20 Live demo: KubeFuzz finds logic and crash bugs
- 53:00 Q&A: Fail-open policies and fuzzing DoS risks
Testing and Fuzzing the Kubernetes Admission Configuration
Speakers: Benjamin Koltermann (Security Architect for Cloud and Kubernetes, Avolens), Maximilian Rademacher (Vulnerability Researcher and Developer, Avolens)
Conference: TROOPERS
YouTube: https://www.youtube.com/watch?v=pnadBOMAIo8
Overview
In this insightful talk from TROOPERS, Benjamin Koltermann and Maximilian Rademacher of Avolens delve into the critical, yet often overlooked, area of Kubernetes admission controller security. Their presentation, titled "Testing and Fuzzing the Kubernetes Admission Configuration," highlights the inherent power and complexity of admission controllers, which act as crucial gatekeepers for all API requests entering a Kubernetes cluster. The speakers argue that while these controllers are vital for enforcing security policies and compliance, their testing methodologies often lag behind, leaving clusters vulnerable to misconfigurations, bypasses, and even denial-of-service conditions.
The core of the talk introduces KubeFuzz, an open-source tool developed by the speakers to automate the testing and fuzzing of Kubernetes admission configurations. KubeFuzz addresses the unique challenges of fuzzing in a Kubernetes environment by leveraging the cluster's OpenAPI specification to generate syntactically and semantically correct inputs. Through a live demonstration, Koltermann and Rademacher illustrate how KubeFuzz can uncover subtle logic bugs and critical runtime crashes in admission controllers, which could otherwise lead to unauthorized resource deployments or complete API unavailability.
This presentation is highly relevant for anyone involved in Kubernetes security, platform engineering, or application development within a Kubernetes ecosystem. It underscores the necessity of robust security testing for custom resources and admission webhooks, providing both a theoretical understanding of the problem space and a practical, open-source solution. The findings emphasize that even seemingly minor flaws in admission logic can have significant security implications, making tools like KubeFuzz indispensable for maintaining the integrity and security of modern containerized environments.
Background
▶ Watch: Understanding Kubernetes Admission Chain and Webhooks (4:10)
Kubernetes, as a leading container orchestration system, is designed to manage hundreds or thousands of containers, making manual management impractical. A key feature of Kubernetes is its API-driven architecture, where "everything is an API object." This design allows for extensive customization and extension through Custom Resource Definitions (CRDs), enabling users to define their own API resources (e.g., Elasticsearch, GitLab objects) and interact with them using standard Kubernetes tools like kubectl.
At the heart of Kubernetes security lies the admission chain, a sequence of checks that every API request must pass after authentication and authorization. This chain comprises mutating admission controllers and validating admission controllers. Mutating controllers can modify an API object before it's persisted (e.g., adding a specific name to a pod), while validating controllers enforce policies by accepting or rejecting an object based on its content (e.g., preventing privileged pods or containers running as root). These controllers effectively act as "bouncers" for Kubernetes API calls, ensuring that only compliant manifests enter the cluster and are persisted in the etcd database.
Admission controllers are powerful tools for enforcing organizational compliance and security policies. They can restrict dangerous configurations, inject default values, and validate custom resources managed by operators. For instance, GKE Autopilot clusters use admission controllers to restrict privileged pods, and the Elasticsearch Cloud on Kubernetes employs them to validate Elasticsearch cluster configurations. Attackers view these controllers as high-value targets; bypassing their logic could grant access to critical infrastructure or allow the deployment of malicious components, such as custom kernel modules via a Kernel Module Operator.
Despite their critical role, testing admission controllers effectively presents significant challenges. While manual testing and traditional unit tests are viable, they often fall short in comprehensively exploring the complex state space and potential edge cases that can arise from varied input combinations. This gap in mature testing methodologies is precisely why advanced techniques like fuzzing become essential. Fuzzing, in this context, aims to automatically generate a vast array of inputs to uncover unexpected program behaviors, including crashes or subtle logic bugs, that human-crafted tests might miss.
Best practices for admission controllers include running them inside the Kubernetes cluster to minimize external communication, writing them in performant languages like Go or Rust (avoiding less robust options like Bash), and crucially, setting their failure policy to Fail. A failurePolicy: Fail ensures that if the Kubernetes API cannot communicate with the admission controller, the object is rejected. Conversely, failurePolicy: Ignore (or "fail-open") would allow the object to pass through as if the controller never existed, creating a significant security bypass risk if the controller becomes unavailable or crashes.
Key Findings
▶ Watch: Fuzzing Kubernetes objects using OpenAPI schemas (14:40)
The talk's primary finding is the critical need for robust, automated testing of Kubernetes admission controllers, a need currently unmet by widely adopted mature methods. The speakers identify fuzzing as a powerful solution to this problem, capable of uncovering subtle yet severe vulnerabilities that traditional testing approaches might miss.
The introduction of KubeFuzz, an open-source tool, represents a significant contribution. It demonstrates that intelligent, Kubernetes-aware fuzzing can be practically applied to admission configurations. Key findings regarding KubeFuzz's capabilities include:
- Automated Input Generation: KubeFuzz successfully leverages the Kubernetes cluster's OpenAPI specification (available at
/openapi/v2) to generate syntactically correct and semantically meaningful YAML/JSON inputs for any resource, including custom ones. This overcomes a major hurdle in fuzzing structured data. - Discovery of Logic Bugs: The tool effectively identifies logic flaws in admission controller code. A prominent example demonstrated was a missing lower bound check for a
timestampfield, allowing a manifest withtimestamp: 0to be accepted when it should have been rejected. Such bugs can lead to the deployment of non-compliant or malicious resources. - Detection of Runtime Crashes: KubeFuzz can trigger and detect admission controller crashes. In the demo, providing a negative integer to a field expected to be an unsigned 64-bit integer (
u64) caused a Rustunwrap()panic, leading to the admission controller thread crashing. These crashes, especially when the admission webhook is configured with afailurePolicy: Ignore(fail-open), can result in a complete bypass of all validation checks. - Identification of Coverage Points: The tool intelligently uses various feedback mechanisms, including acceptance/rejection status, HTTP return codes (e.g.,
403,400), and crucially, unique admission controller messages, to determine "interesting" inputs and guide the fuzzing process. This allows KubeFuzz to explore different logic paths within the controller. - Highlighting
dryRunImportance: The talk underscores the critical role of thedryRunflag inkubectlcommands, which allows testing admission controllers without actually persisting resources in the etcd database. This is essential for safe and high-volume fuzzing in a live cluster.
Overall, the key findings emphasize that admission controllers, despite their security-critical function, are susceptible to common programming errors and logical flaws. KubeFuzz provides a targeted, efficient, and automated way to expose these vulnerabilities, thereby strengthening the security posture of Kubernetes clusters.
Technical Deep Dive
▶ Watch: Fine-tuning fuzzing with KubeFuzz constraint language (24:20)
Fuzzing, at its core, involves automatically generating inputs for a program to observe unexpected behaviors like crashes or logic bugs. The basic fuzzing loop consists of generating a new sample (often based on an existing "corpus" of interesting inputs), feeding it to the program, observing the outcome, and deciding if the result is "interesting" enough to be added to the corpus. For Kubernetes admission controllers, this process introduces unique challenges that KubeFuzz is designed to address.
1. Input Generation: Understanding Kubernetes Resources
The first major challenge is generating valid and meaningful Kubernetes resource manifests (YAML or JSON). Simply generating random strings would not yield useful inputs, as Kubernetes API requires a specific structure. KubeFuzz solves this by leveraging the Kubernetes API itself. Every Kubernetes cluster exposes an OpenAPI specification at the /openapi/v2 endpoint. This specification describes all available resources and their schemas.
KubeFuzz uses tools like openapi-to-json-schema to convert this OpenAPI specification into JSON schemas. These schemas act as blueprints, detailing the types, formats, and constraints for each field within a Kubernetes object (e.g., type: string, format: int32, enum values). By understanding these schemas, KubeFuzz can generate inputs that are both syntactically correct (valid YAML/JSON structure) and semantically correct (adhering to specified types and formats, at least partially). For example, if a schema specifies an enum for a field, KubeFuzz will generate values from that enum. It also attempts "smart generation" for common fields like port (e.g., generating a 16-bit integer).
To further refine input generation, KubeFuzz introduces a constraint language. This allows users to fine-tune the fuzzing process by specifying which fields of a resource (identified by group, version, and kind) should be targeted. Users can define JSON paths to specific fields (e.g., spec.containers[0].securityContext.privileged), set min_occurrences and max_occurrences for arrays, overwrite values (e.g., privileged: true), mark fields as required, or even specify regex patterns for string generation. This targeted generation is crucial because randomly generating an entire complex resource would rarely yield semantically meaningful inputs that could trigger specific admission controller logic.
2. Efficient Execution: Interacting with the Kubernetes API
The second challenge is efficiently feeding these generated inputs to the admission controllers. Since admission controllers operate on API requests, KubeFuzz interacts directly with the Kubernetes API. The tool, written in Rust, uses the kube-rs crate for authenticated communication, supporting various authentication schemes via a kubeconfig file.
A critical aspect of efficient and safe execution is the use of the dryRun flag. When submitting a resource, KubeFuzz sets dryRun=All (or dryRun=Client or dryRun=Server). This tells the Kubernetes API to process the request through the entire admission chain (authentication, authorization, mutating webhooks, validating webhooks) but without actually persisting the resource in the etcd database. This prevents the cluster from being flooded with thousands of ephemeral resources during fuzzing, which would be both inefficient and potentially harmful.
The primary bottleneck for execution speed is often network latency, as KubeFuzz is performing fuzzing over the web API. Despite this, the approach is necessary to test the actual admission chain as it operates in a live cluster, including potential interactions between multiple admission controllers.
3. Determining Interesting Results: Coverage Metrics
The final challenge is determining which generated inputs are "interesting" and should be added to the corpus for further mutation. Unlike traditional code fuzzers that use code coverage, KubeFuzz relies on feedback from the Kubernetes API and the admission controllers themselves. "Interesting" inputs are those that trigger new or unexpected behaviors. KubeFuzz uses several metrics:
- Acceptance/Rejection Status: Was the resource accepted or rejected by the API? This is a fundamental differentiator.
- HTTP Return Codes: Different HTTP status codes (e.g.,
200 OK,403 Forbidden,400 Bad Request) can indicate distinct processing paths or errors. - Denial of Service/Internal Errors: If the API reports an internal error or failure to call a webhook (e.g., "could not call the admission webhook due to an internal error"), this signifies a critical event, potentially a crash or DoS, making the input highly interesting.
- Admission Controller Messages: This is arguably the most valuable coverage metric. When an admission controller rejects a resource, it often returns a custom message explaining the reason. KubeFuzz treats unique admission controller messages as distinct coverage points. For example, if a controller says "pod name not allowed" for one input and "privileged container detected" for another, these represent different logic branches being hit. Finding new, unique messages indicates that the fuzzer has explored a new state or code path within the admission controller.
By combining these metrics, KubeFuzz can build a corpus of inputs that collectively explore the various logical states and error conditions of the target admission controller, guiding mutations towards increasingly interesting or problematic inputs.
Hurdles and Future Improvements:
During development, several hurdles were encountered:
- Untyped CRD Fields: Many CRD specifications lack full type descriptions, forcing KubeFuzz to guess or users to manually specify types, hindering generality.
- Ambiguous Types: Kubernetes quantities (e.g., memory limits) often rely on textual descriptions rather than explicit types, making automatic generation difficult.
- Undocumented Constraints: Native Kubernetes objects sometimes have internal min/max values not described in their schemas.
- Missing Required Fields: Some required fields are not explicitly marked as such in schemas.
- Semantic Properties: Complex semantic rules (e.g., field A only exists if field B is present) are not described in schemas. A future plugin system for post-processing generated resources is planned to address this.
- Webhook Side Effects: Admission controllers might have side effects (logging, database writes) that are hard to control or revert during high-volume fuzzing.
- Non-Static Error Messages: Dynamic error messages (e.g., including a specific pod name) can artificially inflate coverage, making it harder to identify truly distinct logic paths.
- Statefulness: If an admission controller is stateful, fuzzing results might not be reproducible, as the controller's internal state could change between requests.
Future improvements for KubeFuzz include enhancing the coverage system by processing optional warnings from webhooks, analyzing the delta of changed resources after mutating webhooks, and implementing a plugin system for post-processing to enforce complex semantic properties or custom formats (like checksums or signatures).
Demo / Proof of Concept
▶ Watch: Two admission controller bugs (logic and crash) revealed (43:00)
The live demonstration showcased KubeFuzz's ability to uncover critical bugs in a custom admission controller. The scenario involved a Kubernetes cluster running an admission controller named Troopers2023Talks designed to validate custom resources representing conference talks.
The Custom Resource Definition (CRD):
The Troopers2023Talks CRD defined a Troopers2023Talk object with the following structure:
title: A string.speaker: An array of objects, each with aname(string) andcanTalk(boolean).status: An object containing:accepted: A boolean.track: An enum (e.g., "Blue", "Red").timestamp: An integer.
The Admission Controller Logic (with deliberate bugs):
Maximilian presented a simplified Rust-based admission controller for this CRD, which contained two specific bugs:
- Logic Bug (Missing Lower Bound Check): The controller validated the
status.timestampto ensure it was greater than a specific UTC timestamp. However, it lacked a lower bound check. This meant that while future timestamps were validated, very old or zero timestamps were not explicitly rejected, leading to an illogical acceptance. - Runtime Crash (Negative Timestamp): The
status.timestampfield was loaded usingtimestamp.parse().unwrap()as an unsigned 64-bit integer (u64). If a negative integer was provided, theparse()operation would fail, and the subsequentunwrap()call would cause the controller's thread to panic and crash.
KubeFuzz in Action:
- Schema Retrieval: First,
KubeFuzz get schemaswas executed against the Kubernetes API's/openapi/v2endpoint, downloading all resource definitions, including theTroopers2023TalksCRD schema. - Constraint Configuration: A
constraint.yamlfile was created to focus KubeFuzz on theTroopers2023Talksresource. It specified that thespecfield should always be generated and constrained thespeakersarray to contain 1-2 entries. - Fuzzing Execution: The
KubeFuzz fuzzcommand was run, pointing to the downloaded schemas and the constraint configuration. The fuzzer started generating and submitting manifests withdryRunenabled.
- The UI displayed real-time metrics: inputs per second, total inputs, unique coverage hits, and counts of accepted and error responses.
- The fuzzer quickly started identifying "new coverage hits" as it explored different logic paths within the admission controller, indicated by varying admission controller messages.
- After a short period,
KubeFuzzfound both types of bugs.
Demo Results:
- Logic Bug Found: KubeFuzz generated and an accepted manifest where the
status.timestampwas0. This demonstrated the missing lower bound validation, allowing an invalid talk schedule to be accepted into the cluster. - Runtime Crash Found: KubeFuzz also generated an input that resulted in an error response. Examination of this input revealed a negative value for
status.timestamp. This input caused the admission controller to panic, exposing theunwrap()vulnerability.
The implications of these findings were highlighted: a logic bug could lead to non-compliant data being stored, which an operator might then blindly process. A runtime crash, especially with a failurePolicy: Ignore (fail-open) configured, would bypass all subsequent validation, allowing any malicious object to be deployed. The demo vividly illustrated KubeFuzz's effectiveness in finding such critical vulnerabilities, even in a simple admission controller.
Defensive Implications
▶ Watch: Q&A: Fail-open policies and fuzzing DoS risks (53:00)
The findings from this talk have significant implications for anyone operating or developing within a Kubernetes environment. Admission controllers are the last line of defense before resources are persisted, and their integrity is paramount.
- Prioritize Admission Controller Testing: Manual testing and basic unit tests are insufficient for complex admission logic. Organizations must adopt sophisticated, automated testing methodologies like fuzzing to uncover subtle logic bugs and runtime vulnerabilities. Tools like KubeFuzz should be integrated into CI/CD pipelines for continuous security validation.
- Strict Failure Policies are Crucial: The choice of
failurePolicyis a critical security decision. Always configure admission webhooks withfailurePolicy: Fail(also known as "fail-close") whenever possible. This ensures that if the admission controller is unavailable, crashes, or returns an error, the API request is rejected, preventing potential bypasses. WhilefailurePolicy: Ignoremight seem appealing for availability, the security risk of allowing unvalidated or malicious resources to pass through is generally unacceptable. - Validate All Inputs, Even Post-Admission: The demo highlighted that an operator might "blindly trust" inputs that have passed the admission chain. This is a dangerous assumption. Even if an admission controller is in place, downstream components (operators, applications) should perform their own input validation and sanity checks, as admission controllers can be bypassed or misconfigured.
- Ensure Robust CRD Schemas: Developers of Custom Resource Definitions should strive for fully typed and descriptive schemas. Ambiguous types, undocumented constraints, and missing "required" markers make it difficult for automated tools to generate meaningful inputs and for admission controllers to reliably validate. Clear schemas improve both security and maintainability.
- Address Side Effects and Statefulness: When developing admission controllers, be mindful of side effects (e.g., external API calls, database writes) and statefulness. High-volume fuzzing can exacerbate issues related to these, potentially causing external systems to be overloaded or leading to non-reproducible test results. Design admission controllers to be stateless and idempotent where possible, or implement safeguards to manage side effects during testing.
- Design for Clear Error Messages: Admission controllers should emit static, descriptive error messages. Non-static messages (e.g., including specific resource names) can undermine the effectiveness of fuzzing tools that rely on message uniqueness for coverage, making it harder to identify distinct logic paths.
- Consider Differential Fuzzing: For organizations running multiple admission controllers or different versions of the same controller, differential fuzzing can be used to compare their behavior. This can uncover parser differentials or inconsistencies in policy enforcement across different implementations or configurations.
- Regularly Review and Update Admission Configurations: Admission controllers, like any security policy, are not "set it and forget it." Regularly review their configurations and logic, especially after Kubernetes upgrades, CRD changes, or the introduction of new applications, and re-run fuzzing tests to ensure continued effectiveness.
By adopting these defensive measures and leveraging specialized tools like KubeFuzz, organizations can significantly enhance the security posture of their Kubernetes clusters, mitigating the risks posed by faulty admission controller logic and configurations.
Key Takeaways
- Admission Controllers are Critical Gatekeepers: Kubernetes admission controllers are powerful and essential components for enforcing security policies and compliance within a cluster, acting as the final line of defense for API requests.
- Fuzzing is a Powerful Testing Method: Automated fuzzing, as demonstrated by KubeFuzz, is an effective and efficient way to uncover subtle logic bugs and critical runtime crashes in Kubernetes admission controllers that often go undetected by traditional testing.
- Kubernetes API Schemas Enable Intelligent Fuzzing: KubeFuzz leverages the cluster's OpenAPI specification to generate syntactically and semantically correct inputs, overcoming a major challenge in fuzzing structured Kubernetes resources.
- Failure Policy is a Security-Critical Configuration: The
failurePolicysetting dictates how the API behaves when an admission controller is unavailable or crashes. ConfiguringfailurePolicy: Fail(fail-close) is generally the safer choice to prevent security bypasses, despite potential availability concerns. - Vulnerabilities Have Real-World Impact: Logic bugs (e.g., missing validation checks) can lead to the deployment of non-compliant resources, while runtime crashes can enable complete policy bypasses, especially in fail-open configurations, posing significant risks to cluster integrity and data security.
- Continuous and Comprehensive Testing is Essential: Given the complexity and criticality of admission controllers, continuous and comprehensive testing with tools like KubeFuzz should be an integral part of the Kubernetes security lifecycle.
About the Speaker(s)
Benjamin Koltermann is the CEO of Avolens, a company specializing in cloud and Kubernetes security, where he works as a Security Architect for Cloud and Kubernetes. He is an active participant in the cybersecurity community, known for his involvement as a CTF (Capture The Flag) player with teams like fluxfingers, sauercloud, and parqueto. Benjamin is also a regular attendee at the Kubernetes SIG Security meetings, contributing to the security documentation and best practices for the Kubernetes ecosystem. His Twitter handle is @parqueto.
Maximilian Rademacher serves as a Vulnerability Researcher and Developer at Avolens. Like Benjamin, he is also a CTF player for fluxfingers. Maximilian is the primary author and developer of KubeFuzz, the open-source tool presented in this talk, demonstrating his expertise in vulnerability research and practical security tool development for Kubernetes. His Twitter handle is @ADD_tunnel.
Reviews
Dr. Zero (Offensive Security Researcher) — SOLID
Koltermann and Rademacher built a genuinely useful fuzzer for Kubernetes admission controllers that avoids trashing your etcd database. Using dryRun to safely hit the live admission chain while using webhook error messages as a coverage signal is smart, pragmatic engineering. It won't win a Pwnie for theoretical math, but it's a tool every K8s auditor needs in their belt tomorrow.
Heather Calloway (CISO) — SOLID
This is a practical, highly effective look at why custom Kubernetes security controls fail and how to test them. The researchers introduce an open-source fuzzer that targets admission controllers, exposing how easily a crashed webhook can bypass your entire cluster security model. If your platform team writes custom Kubernetes admission rules, they need to watch this and integrate the tooling.