Encryption, Identities, and Everything in Between; Building Se... Lior Lieberman & Igor Velichkovich

Encryption, Identities, and Everything in Between; Building Secure Kubernetes Networks

Speakers: Lior Lieberman, Engineer Lead, Google; Igor Velichkovich, Engineer, Stealth Startup

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=Q15XbASxHM0

Overview

In their KubeCon EU talk, "Encryption, Identities, and Everything in Between; Building Secure Kubernetes Networks," Lior Lieberman and Igor Velichkovich address critical shortcomings in current Kubernetes network security paradigms and advocate for a shift towards identity-driven enforcement. The presentation highlights how relying on mutable IP addresses and labels for network policy decisions leaves Kubernetes clusters vulnerable to sophisticated attacks, drawing parallels to infamous real-world breaches. Lieberman and Velichkovich argue that a lack of robust, immutable identity within the network stack, coupled with an absence of industry-wide standardization, creates significant security gaps that current NetworkPolicy and even newer AdminNetworkPolicy mechanisms struggle to bridge.

This talk is essential for anyone involved in securing Kubernetes environments, from platform engineers and security architects to application developers. It systematically breaks down the limitations of existing policy enforcement, introduces the concept of cryptographic identity for workloads, and explores how leading solutions like Cilium and Istio are attempting to solve these challenges, albeit without universal standardization. The speakers emphasize the urgent need for community-driven standardization efforts to establish common identity formats, distribution mechanisms, and interoperability semantics to truly achieve robust, scalable, and predictable network security in cloud-native ecosystems.

Background

▶ Watch: Introduction and the Debug Pod security story (0:00)

The talk opens with a compelling, albeit "made-up," story that perfectly encapsulates a common vulnerability pattern in unsecured Kubernetes environments. An ingress-exposed proxy pod, a rate limiter, and a temporary debug pod are deployed. The debug pod, intended for short-term troubleshooting, is granted broad, unrestricted network access. After fixing the rate limiter issue, the debug pod is forgotten, leaving a public-facing ingress open. An attacker compromises this debug pod and, due to a complete lack of network segmentation, moves laterally across the network to steal sensitive user, payment, and transaction information. This scenario, the speakers underscore, is not hypothetical; it mirrors numerous high-profile data breaches.

Igor Velichkovich then presents several real-world examples to drive this point home:

• Target (2013): Attackers infiltrated Target's network through a weak link – an HVAC contractor using inadequate anti-malware software. A lack of network segmentation allowed them to move from the contractor's system to Target's core network, stealing 70 million people's PII and 40 million credit card numbers. This incident directly led to the widespread adoption of chip and EMV cards in the US.
• Equifax (2017): A vulnerability in Apache Struts, a popular Java web application framework, served as the initial entry point. Again, insufficient network segmentation allowed attackers to exfiltrate 148 million people's personal information, including social Security numbers and financial records.
• Starwood Hotels and Resorts (later Marriott, 2014-2018): Multiple long-term breaches occurred where hackers installed keyloggers and Trojans, compromising 339 million personal data records. Critically, Marriott acquired Starwood, and despite a 10-month security assessment, the ongoing breach was missed. Upon integration, the lack of proper segmentation allowed the breach to spread from the compromised Starwood network to Marriott's infrastructure.

These incidents highlight a recurring pattern: a "weak link" exploited, followed by widespread compromise due to a fundamental absence of network segmentation. The core problem, as the speakers emphasize, is the inability to effectively secure network traffic within and between applications.

Kubernetes, in its current state, offers several primitives for network control:

• NetworkPolicy v1: This has been a stable API for over seven years. It allows application owners (namespace admins, developers) to define ingress and egress rules for their pods. Policies are namespace-scoped and typically use label selectors or CIDRs to identify target pods and allowed sources/destinations. While widely adopted and implemented by most CNIs (Container Network Interfaces), it's designed for granular control within an application's microservices (e.g., frontend to backend).
• AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP): Introduced to address gaps in NetworkPolicy v1, these are cluster-scoped policies designed for cluster administrators. ANP allows for non-overridable global rules, supporting deny and pass actions in addition to allow, and introducing priority for policy evaluation. BANP specifically tackles the "implicit deny" problem by allowing administrators to define default behavior for traffic not covered by other policies. Both are currently v1alpha APIs and require explicit CRD installation and CNI implementation.
• RBAC (Role-Based Access Control): While primarily for API access control, RBAC is also a critical component of overall cluster security, controlling who can perform what actions, including managing network policies.

Despite these offerings, significant gaps remain:

• Implicit Deny Confusion: NetworkPolicy v1 only specifies what's allowed, leading to confusion about default deny behavior.
• No Cluster Admin Support (initially): NetworkPolicy v1 is namespace-scoped, lacking a global enforcement mechanism for administrators. ANP addresses this but adds complexity.
• IP-level, Not Identity-level: Policies primarily operate at the IP layer, implicitly using IPs as an identity derived from labels. This is a critical flaw, as IPs can be reassigned, and labels can be manipulated.
• Scalability Issues: In large clusters with thousands of churning pods and complex label sets, constantly updating IP-based rules on every node for eventual consistency becomes an immense challenge.
• Limited North-South Traffic Support: Controlling external (to/from the cluster) traffic is not robustly supported by core policies.
• Increased Complexity: The introduction of ANP and BANP creates layers of policies, making it difficult to reason about effective traffic flow and troubleshoot issues.

The central thesis of the talk is that the root cause of these issues lies in the fundamental reliance on ephemeral, mutable identifiers (IPs, labels) rather than strong, immutable cryptographic identities for network security.

Key Findings

▶ Watch: Kubernetes' current network security offerings (4:00)

The core message of the talk revolves around the inherent weaknesses of traditional Kubernetes network policies and the urgent need for a paradigm shift towards identity-based security. The speakers identify several key findings:

1. IPs and Labels are Insufficient for Network Security: Current NetworkPolicy mechanisms primarily rely on IP addresses and label selectors to define and enforce traffic rules. The speakers argue that this approach is fundamentally flawed because:

• Mutability of Labels: Labels, often used as proxies for identity, can change at runtime. Kubernetes components like kubelet have privileges to modify labels, and there have been past CVEs and attack vectors exploiting this. If a security policy is tied to a mutable label, an attacker who can modify a label could potentially gain unauthorized access.
• IP Reassignment and Caching Issues: When pods churn (start/stop), IPs can be reassigned. If network policy enforcement mechanisms cache IP-to-label mappings, a new pod with different labels might inherit the network access of a previous pod that occupied the same IP, leading to security bypasses.
• Not a True Identity: Labels and IPs are operational identifiers, not cryptographic identities. They do not provide strong assurance about the workload's authenticity or authorization.

1. The Need for Immutable Identity: To build truly secure Kubernetes networks, policies must be tied to an immutable, verifiable identity that persists throughout a pod's lifecycle and cannot be easily manipulated. This identity should be fixed for the pod's lifetime, ensuring predictable policy enforcement.

1. ServiceAccount as a Stronger Identity Primitive: The speakers propose the Kubernetes ServiceAccount as a more suitable primitive for identity. ServiceAccounts are designed to be an identity for pods, are tied to RBAC (meaning they have defined permissions), and are significantly harder to manipulate than labels. A pod cannot change its own ServiceAccount at runtime without a restart, making it a more robust security boundary. If a pod is compromised, escalating privileges by changing its ServiceAccount would require a restart, potentially alerting operators or invalidating the compromise.

1. Scalability Challenges of Label-Based Enforcement: Beyond security, relying on label selectors for policy enforcement in large, dynamic clusters presents significant scalability and performance issues. Queries for subsets of pods based on overlapping labels are computationally expensive for both the data plane and control plane. Enforcing a strict order of evaluation for complex policies across a constantly churning environment of thousands of pods, each requiring IP rule updates, leads to eventual consistency problems and potential security delays.

1. Transmitting Identity in Network Packets: The most radical and promising finding is the idea of embedding a pod's verifiable identity directly into the network packet. If the identity could be transmitted as part of the networking packet itself, it would eliminate the need for expensive, error-prone, and eventually consistent mappings between labels and IPs. This would allow for more precise, real-time, and secure policy enforcement directly at the network layer, independent of the pod's IP address.

1. Lack of Standardization Hinders Adoption and Security: Despite the clear need and the existence of advanced solutions, there is a profound lack of standardization in how identity is defined, distributed, and enforced across different Kubernetes network security tools. This leads to fragmentation, interoperability issues, and potential security gaps as organizations cobble together disparate solutions. Key areas lacking standardization include:

• What constitutes a standard identity (e.g., pod labels, ServiceAccount names, SPIFFE IDs)?
• What is the standard format for this identity (e.g., JWTs, X.509 certificates, SVIDs)?
• How are identities distributed and issued (e.g., via kubelet as proposed in KEP-437: Pod Certificates)?
• What is a common root of trust?
• How do L3/L4 and L7 enforcement mechanisms interoperate and compose policies?

These findings collectively highlight a critical juncture in Kubernetes security, urging a move beyond IP-centric network segmentation to a more robust, identity-aware, and standardized approach.

Technical Deep Dive

▶ Watch: Key gaps and limitations of Network Policy v1 (5:30)

The technical deep dive into Kubernetes network security begins by dissecting the fundamental problems with current approaches, primarily the reliance on IPs and labels, and then explores how leading projects attempt to address these issues, ultimately revealing the profound need for standardization.

The Identity Crisis in Kubernetes Networking

The central technical challenge identified is the lack of a robust, immutable identity for workloads at the network layer. As Lior Lieberman quotes John Howard, a Kubernetes maintainer, "the strengths of the checkpoint vary substantially based on how we identify employees." Similarly, the strength of network policies depends entirely on the reliability of the "identity" used.

• Labels as a Flawed Identity Proxy: While NetworkPolicy v1 uses label selectors to group pods and define policy, these labels are not true identities. They are metadata.
• Mutability: Labels can change at runtime. The kubelet has privileges to modify labels, and this has been a source of security vulnerabilities. If a policy relies on a label, a compromised pod that can alter its own label could bypass security controls.
• Dynamic Nature: When pods churn, IP addresses are reassigned. If network policies are cached based on IP-to-label mappings, a new pod might inherit the network access rights of a previous pod that held its IP, even if its own labels dictate different permissions. This "eventual consistency" can lead to temporary, but critical, security holes.
• Scalability Cost: Querying for subsets of pods based on labels is computationally expensive, especially with overlapping selectors and a large number of pods. The control plane constantly maps labels to IPs, distributes these mappings to the data plane, and updates rules on every node. This process is resource-intensive and prone to delays.

• ServiceAccount as a Stronger Identity: The speakers advocate for using the ServiceAccount as the primary identity for pods.
• Immutability: A ServiceAccount is fixed for a pod's lifetime. Changing a pod's ServiceAccount requires a restart, which makes it a much stronger security boundary. A compromised pod cannot easily escalate privileges by simply changing its ServiceAccount.
• Tied to RBAC: ServiceAccounts are inherently linked to RBAC roles, providing a clear definition of permissions and responsibilities. This makes them a natural fit for defining network-level authorization.

The Vision: Identity in the Network Packet

The ideal solution proposed is to transmit the pod's verifiable identity directly as part of the network packet itself. This eliminates the need for complex, expensive, and potentially insecure label-to-IP mappings. If the identity is present in the packet, network enforcement points can make real-time, precise authorization decisions based on cryptographic proof, rather than relying on an inferred identity from an IP address. This would drastically improve security, scalability, and predictability.

Existing Solutions and Their Approaches

The talk then examines two prominent projects that provide advanced network segmentation in Kubernetes, highlighting their different approaches to identity and enforcement:

1. Cilium Network Policy:

• Enrichment of NetworkPolicy: Cilium extends core Kubernetes NetworkPolicy with its own CiliumNetworkPolicy CRD.
• L3/L4 Enforcement: Provides granular control at Layer 3 (IP) and Layer 4 (Port/Protocol). While it offers useful abstractions like predefined entities or integration with AWS Security Groups, its underlying enforcement still boils down to IP addresses.
• L7 Enforcement: Crucially, Cilium supports Layer 7 (Application) policies. This functionality is enforced using a node-local Envoy proxy, which intercepts and inspects traffic. The CNI level itself cannot enforce L7 policies; it requires an application-aware proxy.
• Cluster-wide Policies: CiliumClusterwideNetworkPolicy offers administrators a way to define global rules, similar to AdminNetworkPolicy.
• Egress/Ingress and Deny Support: Cilium policies support both ingress and egress rules, as well as explicit deny actions.
• Identity: Cilium introduces its own concept of "identity" which is derived from pod labels and ServiceAccounts but is ultimately mapped to a numerical identity that is then used for policy enforcement within the Cilium data plane (e.g., eBPF). While an improvement, it still relies on the initial label derivation.

1. Istio Authorization Policies:

• Service Mesh Approach: Istio, as a service mesh, takes a fundamentally different approach. It focuses on L4 TLS and L7 enforcement.
• Identity via Certificates: Istio's primary mechanism for identity is through mTLS (mutual Transport Layer Security). Each workload is issued an X.509 certificate by Istio's control plane. This certificate embeds the workload's identity, typically its ServiceAccount in the Subject Alternative Name (SAN) field, often using the SPIFFE format (Secure Production Identity Framework for Everyone).
• Enforcement: Policy enforcement happens at the proxy level (either sidecar or ambient mode). When two services communicate, their Envoy proxies establish an mTLS connection, verify each other's certificates, and extract the identity. Authorization policies then use this verified identity (e.g., payments-frontend ServiceAccount) to determine if traffic to a destination (e.g., payments-backend) is allowed.
• External Authorization: Istio supports delegating authorization decisions to external services, such as OPA (Open Policy Agent) or custom webhooks, allowing for dynamic and complex policy evaluations.
• Deny and Cluster-wide: Istio provides support for deny policies and cluster-wide policies (by placing them in the istio-system namespace).
• No Egress Support (natively in AuthorizationPolicy): A notable gap highlighted is the lack of native egress policy support within Istio's AuthorizationPolicy CRD, which focuses primarily on ingress to a service.

The Standardization Conundrum

Despite the advanced capabilities of projects like Cilium and Istio, the speakers emphasize that the lack of standardization is a critical barrier. Key questions that remain unanswered and require community consensus include:

• Standard Identity: What is the universally accepted identity for a workload? Is it a pod label, a ServiceAccount name, or a SPIFFE ID?
• Standard Format: How should this identity be represented? JWTs, X.509 certificates, or SVIDs (SPIFFE Verifiable Identity Documents)? Each has its pros and cons regarding scope (L7 vs. L4/L3), complexity, and revocation.
• Identity Distribution and Issuance: How should identities be issued and distributed to pods? The Kubernetes Pod Certificates KEP (KEP-437) proposes that kubelet could issue X.509 certificates to pods, standardizing this crucial step. But will the ecosystem adopt this?
• Common Root of Trust: How do different components and clusters establish a common trust anchor for verifying these identities?
• Interoperability: How do L3/L4 network policies and L7 authorization policies compose and interoperate seamlessly? This is a daily challenge for engineers dealing with layered security.

The technical deep dive reveals a landscape where robust solutions exist but operate in silos, preventing a unified, secure, and scalable approach to Kubernetes network segmentation. The call for standardization is not merely an academic exercise but a critical need for practical security implementation.

Demo / Proof of Concept

▶ Watch: Introducing Admin Network Policy for cluster-wide control (7:00)

The talk did not feature a live technical demonstration or a traditional proof of concept. Instead, the speakers effectively used a "made-up story" at the beginning of the presentation as a conceptual demonstration of the vulnerabilities that arise from a lack of identity-driven network segmentation. This narrative, involving a forgotten debug pod with unrestricted access, served to illustrate the real-world impact of the security gaps they later discussed. The subsequent examples of Target, Equifax, and Marriott further cemented this conceptual understanding by highlighting how similar weaknesses have led to catastrophic breaches in large enterprises. While not a code-driven demo, this narrative approach effectively conveyed the problem statement and the necessity for the solutions proposed.

Defensive Implications

▶ Watch: Admin Network Policy: deny, priority, and pass actions (8:00)

The insights from Lieberman and Velichkovich's talk carry significant defensive implications for organizations operating Kubernetes clusters. Moving beyond traditional IP-based network policies to an identity-aware, defense-in-depth strategy is crucial for mitigating the types of breaches discussed.

Immediate Actionable Steps:

1. Admission Control for Labels: If your existing network policies rely on labels for defining security boundaries, it is imperative to implement admission controllers to prevent unauthorized or accidental label changes at runtime. Tools like Kyverno or Gatekeeper can enforce policies that restrict who can modify labels on pods, or even validate the format and values of labels used for security purposes. This ensures that the "identity" derived from labels remains consistent and cannot be exploited by a compromised workload.
2. Enable and Verify MTLS for Identity Transmission: For solutions that leverage identity (like Istio), mTLS (mutual Transport Layer Security) is the backbone for transmitting and verifying that identity securely. If you are using a service mesh or similar technology, ensure that mTLS is fully enabled and enforced across your cluster. As the speakers explicitly state, "if you disable MTLS, if you're disabling encryption in your cluster, you cannot enforce authorization policy." Without mTLS, identities cannot be cryptographically verified, rendering identity-based authorization policies ineffective. Understand the implications of your configuration choices on the security posture.

Adopting a Zero Trust and Defense-in-Depth Posture:

The ultimate goal for platform engineering teams should be to facilitate a Zero Trust environment with defense-in-depth principles. This means:

• Preventing Impersonation with MTLS: Every service should present its own verifiable cryptographic certificate. mTLS ensures that only authenticated and authorized services can communicate, making it significantly harder for a compromised debug pod (or any other workload) to impersonate a legitimate service.
• Identity-Based Authorization Policies: Implement authorization policies that strictly define what each workload (based on its verified identity, e.g., ServiceAccount) is allowed to access. A debug pod, for instance, should only have access to the specific resources it needs to inspect (e.g., dump headers from a specific ingress), not the entire cluster's sensitive data. This prevents lateral movement even if the pod is compromised.
• Least Privilege for ServiceAccounts: Ensure that ServiceAccounts are granted only the absolute minimum API access required for the workload to function. The debug pod in the opening story, for example, should not have had any API access beyond its specific debugging task. This limits the blast radius of a compromised pod.
• Ease of Use for Developers: Platform engineering teams must make it easy for developers to operate in a security-conscious manner. This involves providing secure defaults, clear guidelines, and tools that simplify the implementation of identity-based security without hindering developer velocity.

Advocating for Standardization:

Beyond immediate technical fixes, defenders should actively engage with and advocate for standardization efforts within the Kubernetes community:

• Kubernetes Pod Certificates KEP (KEP-437): This Kubernetes Enhancement Proposal is working towards standardizing the issuance of X.509 certificates to pods via kubelet. Contributing feedback or tracking its progress is vital, as it could provide a foundational, standardized mechanism for cryptographic identity in Kubernetes.
• SIG Network Policy API Group Discussions: Participate in discussions within the SIG Network Policy API group on GitHub and Slack. There's an ongoing discussion about gathering use cases for identity-based policies from a user experience perspective, not just security. Input from practitioners on how identity-based policies could simplify reasoning about workloads (compared to ephemeral IPs) is highly valuable.
• Community Engagement: Join the relevant Slack channels (e.g., #sig-network and #sig-auth) to contribute ideas, ask questions, and help drive the standardization of identity, format, distribution, root of trust, and interoperability semantics.

By taking these defensive steps and actively participating in standardization efforts, organizations can move towards a more resilient and secure Kubernetes network architecture that is better equipped to prevent the widespread data breaches of the past.

Key Takeaways

• Current Kubernetes Network Policies are Insufficient: Relying on mutable IP addresses and labels for network segmentation in NetworkPolicy v1 creates critical security gaps and scalability challenges, leading to vulnerabilities demonstrated by real-world breaches like Target and Equifax.
• Immutable Identity is Crucial: Robust network security in Kubernetes requires policies to be tied to a strong, immutable, and verifiable identity for workloads, such as the ServiceAccount, which is harder to manipulate and provides predictable policy enforcement.
• Transmit Identity in Network Packets: The ideal approach is to embed cryptographic identity directly into network packets, eliminating the need for complex and eventually consistent IP-to-label mappings, thereby improving security, scalability, and real-time enforcement.
• Advanced Solutions Exist, But Lack Standardization: Projects like Cilium and Istio offer advanced network segmentation and identity-aware policies (Cilium at L3/L4/L7, Istio via mTLS at L4/L7), but their differing approaches to identity, format, and enforcement layers highlight a critical lack of industry standardization.
• Standardization is Paramount for Future Security: Community-driven efforts, such as the Kubernetes Pod Certificates KEP (KEP-437) and discussions within the SIG Network Policy API group, are essential to standardize identity formats (SPIFFE, SVIDs), distribution mechanisms, roots of trust, and interoperability between L3/L4 and L7 enforcement.
• Implement Defense-in-Depth: Immediately implement admission control for labels (e.g., Kyverno, Gatekeeper), ensure mTLS is enabled for identity transmission, and apply identity-based authorization policies with the principle of least privilege to create a resilient, Zero Trust security posture.

About the Speaker(s)

Lior Lieberman is an Engineer Lead at Google. His work focuses on developing and leading engineering initiatives, likely with a strong emphasis on cloud-native technologies and security within Google's extensive infrastructure.

Igor Velichkovich is an Engineer at a stealth startup, where he is currently focused on AI infrastructure. His role suggests expertise in cutting-edge technologies and the challenges of securing complex, high-performance computing environments.