Effective building blocks for securing...

Effective building blocks for securing...

Speakers: Shrikant Pandhare, Sagiv Sheelo

Conference: BSidesSF 2024

YouTube: https://www.youtube.com/watch?v=Q_lMKSEzK-E

Overview

This talk, presented by Shrikant Pandhare and Sagiv Sheelo from Snap, delves into the intricate journey of evolving a cloud-native infrastructure from a monolithic application to a highly secure, multi-tenant Kubernetes platform. The speakers share Snap's experiences, challenges, and the effective security controls they implemented to safeguard shared Kubernetes clusters operating across multiple public clouds (AWS and Google Cloud). The core focus is on establishing robust isolation and least privilege principles in an environment where diverse services, owned by different teams with varying threat profiles, co-exist on the same underlying infrastructure.

The presentation highlights the critical need for strong platform security as a foundational element for any application, especially in a rapidly scaling, multi-cloud environment. It addresses the inherent security risks introduced by multi-tenancy in Kubernetes, which was not originally designed with secure multi-tenancy as a default. By detailing their architectural evolution and the specific security controls built, Snap provides a blueprint for other organizations grappling with similar challenges in securing their shared compute environments.

The insights shared are particularly relevant for security engineers, platform architects, and infrastructure teams looking to enhance the security posture of their Kubernetes deployments. The talk emphasizes a proactive, defense-in-depth approach, covering everything from admission control and identity management to supply chain security and developer access patterns, all while navigating significant organizational and technical hurdles.

Background

▶ Watch: Overview of single-tenant Kubernetes architecture (cluster per service) (03:00)

Snap's platform journey began on Google App Engine, a platform that facilitated rapid bootstrapping and scaling to millions of users. However, approximately five years into its life, as Snap grew to encompass multiple user-facing products and backend services, the limitations of a monolithic architecture became apparent. These limitations included challenges with scalability and developer velocity. Concurrently, Snap made a strategic decision to adopt a multi-cloud strategy, necessitating a platform that could abstract away cloud-specific complexities and operate seamlessly across providers like AWS and Google Cloud. This led to the adoption of Kubernetes as the chosen abstraction layer for their first service mesh platform.

Initially, Snap implemented a single-tenant service mesh platform. This architecture utilized open-source Envoy for both API Gateway functionality at the edge and as sidecars for individual service deployments. An in-house service configuration portal, named Switchboard, was responsible for synchronizing control plane artifacts such as service-to-service and Gateway routes, along with other transport parameters. A key design choice at this stage was to provision a cluster per service. This decision was made to limit the blast radius and as a cautious step, given that Kubernetes was being adopted for the first time at a very large scale. The lack of feature parity between cloud providers for managed Kubernetes services also influenced some trade-offs.

However, as the number of services grew, this "cluster per service" model led to an explosion in the number of clusters that needed to be managed and secured. Operations became decentralized, lacking a Site Reliability Engineering (SRE) model, which meant reliance on individual engineering teams for cluster management. This often resulted in delayed cluster upgrades and patching. The "path to production" story was weak, making it difficult to enforce a golden path CI/CD pipeline, leading to challenging, error-prone, and redundant service rollouts. From a security perspective, services had broad IAM permissions, operators had unrestricted production access, and deployments lacked provenance, making integrity against unknown compromises difficult to ascertain. The large, heterogeneous fleet also complicated vulnerability response and made it expensive to deploy unified tooling for detection and response. These accumulating limitations and security concerns ultimately prompted a re-evaluation of the mesh platform and a strategic shift towards a multi-tenant Kubernetes model.

Key Findings

▶ Watch: Defining multi-tenancy and inherent security concerns in shared environments (06:00)

The transition from a single-tenant to a multi-tenant Kubernetes architecture at Snap was driven by a clear set of problems and objectives, leading to several key findings regarding security risks and necessary controls.

The primary problem statement for building a multi-tenant platform was to construct an abstraction that provisions all Kubernetes primitives for multi-tenancy and standardizes the developer experience across multiple clouds (Google Cloud and AWS). The core objectives were to create an opinionated platform that enforces best practices for CI/CD, observability, and security, thereby minimizing drift and heterogeneity, and to centralize infrastructure to enable high-velocity infrastructure changes.

The shift to multi-tenancy, where multiple services from different teams share the same Kubernetes cluster and potentially the same nodes, introduced significant security risks:

• Privilege Escalation: Kubernetes was not originally built for secure multi-tenancy. Lack of default isolation can lead to privilege escalation between tenants. For instance, control plane permissions might not be restricted to a single tenant's namespace, and pods could be configured to access nodes, allowing service owners to abuse this lack of isolation.
• Overly Permissive Identities: By default, Kubernetes often uses a node identity for all services, which can grant broad permissions (e.g., Google Cloud's Compute Engine service account with editor permissions). This violates the principle of least privilege, as services would share permissions to cloud resources they don't need.
• Unauthorized Access: Developers often have broad access to their clusters by default, risking inadvertent or malicious access to other services or their logs. Direct developer access to shared clusters poses a risk to all co-located services.
• Lack of Isolation Guarantees: Kubernetes, by design, does not provide strong isolation guarantees between pods on the same node. A compromised pod could potentially access and manipulate other pods on the same node, even if they belong to a different service or namespace.
• Supply Chain Risks: Without strong controls, the integrity of deployed code and images could be compromised, allowing malicious code to reach production.

To mitigate these risks, Snap identified the need for robust security controls across three main areas:

1. Proactive blocking of known bad configurations and enforcement of policies for service placement based on sensitivity.
2. Implementation of per-service and per-component identities to ensure least privilege.
3. Limiting developer access to only their service namespaces and logs, and restricting direct access to the Kubernetes control plane.

These findings underscored that a successful multi-tenant Kubernetes platform requires a comprehensive, layered security approach that goes beyond default Kubernetes configurations.

Technical Deep Dive

▶ Watch: Mitigation: Custom admission controller enforcing Pod Security Standards for ... (11:00)

Snap's approach to securing their multi-tenant Kubernetes clusters involved implementing a suite of robust security controls, addressing the identified risks across various layers of the platform.

1. Privilege Escalation Mitigation via Admission Controller:

To prevent privilege escalation between co-tenant services, Snap deployed a custom admission controller. This Kubernetes mechanism intercepts and validates requests to the Kubernetes API before they are persisted. Snap's admission controller enforces security policies based primarily on Kubernetes Pod Security Standards. These standards provide best practices for hardening pods.

Specifically, the admission controller:

• Restricts the usage of privileged containers: These containers have access to all Linux capabilities, which could allow a compromised container to break isolation.
• Prevents pods from using the host namespace: Access to the host namespace would allow a pod to access the underlying node and all other containers running on it, directly breaking isolation.

By enforcing these rules, the risk of privilege escalation between services in multi-tenant clusters is significantly reduced.

2. Granular Pod Exec Permissions:

Recognizing that pod exec can be a risky debugging tool, especially for containers with elevated permissions, Snap implemented a targeted approach. They limit pod exec permission exclusively to non-privileged, service-owned containers. This ensures that developers can still debug their services when necessary, but critical system containers or those with elevated privileges cannot be used as a path for privilege escalation. All actions performed on production are also audited.

3. Least Privilege Identities with Workload Identity Federation:

Addressing the issue of overly permissive identities, Snap moved away from using the default node identity. In Google Cloud, this default is often the Compute Engine service account with broad editor permissions, which violates the principle of least privilege. Snap implemented Workload Identity Federation to deploy pods with a cloud identity distinct from the node identity. This allows for:

• Per-service identities: Each service runs with only the specific permissions it needs (e.g., a "friend service" only has access to the "friends database").
• Per-Kubernetes component and internal infrastructure add-on identities: Ensuring that even platform components operate with least privilege.

This shifts identity access from the node to identities linked to pod service accounts, which are then associated with specific cloud identities. The node itself runs with only the permissions required for normal Kubernetes cluster operations, further reducing the attack surface.

4. Hardened Access via Golden CI/CD Path and RBAC:

To control developer access to shared clusters and prevent unauthorized changes, Snap implemented a golden CI/CD path. This ensures that:

• All deployments and changes to the cluster are tracked and audited, providing a clear trail of "who did what and when."
• Role-Based Access Control (RBAC) is used to limit service owners' direct access to their clusters. Developers are restricted to non-mutating actions within their own namespaces. This prevents out-of-band changes to production, reinforcing the CI/CD pipeline as the sole path for modifications.

5. Zero-Trust Access to Kubernetes Control Planes:

Kubernetes control planes are often internet-facing by default, posing a significant external attack risk. Snap leverages its internal BeyondCorp zero-trust access pattern by using an authentication proxy built at the edge. This proxy provides context-aware, secure access. Support for Kubernetes was integrated into this proxy, and its usage is enforced by limiting control plane access to its specific IPs. This control is transparent to developers, who continue to use kubectl as usual, with their contexts configured to route through the auth proxy and set default service namespace access.

6. Software Supply Chain Security with Image Signature Validation:

To ensure the integrity and provenance of deployed code, Snap's CI/CD pipeline signs digests of built images. Their admission controller then validates these signatures for all pod container images. This guarantees that only approved and verified code, built through their trusted CI/CD platform, is allowed to be deployed on clusters, significantly reducing the risk of malicious or unapproved code reaching production. A key challenge in this implementation was migrating from mutable image tags to immutable image digests, requiring updates across all platform components, but the security guarantees were deemed worth the effort.

7. Service Log Isolation:

In shared Google Cloud projects, services often emit logs to a common Stackdriver logging sink, lacking per-tenant isolation. To address this, Snap configures GCP log sinks and buckets that route logs to service-owned projects. This ensures per-service isolation, meaning each service's logs are ingested directly into its respective project, preventing unauthorized access to logs from other service owners. These log sink and bucket configurations are automated during the service onboarding process to a shared cluster.

8. Co-Tenancy Policy for Blast Radius Control:

Despite all the technical controls, the security team felt multi-tenancy still presented a significant risk. To further reduce the blast radius, Snap developed a co-tenancy policy. This policy evaluates services based on attributes such as access to sensitive data, internet exposure, and service tier during workload onboarding. It then limits which services can be deployed to which clusters, ensuring that highly sensitive services are not co-located with less trusted ones, or that services with high internet exposure are isolated. This policy adds an additional layer of logical isolation and risk management.

9. Proactive Vulnerability Management:

To mitigate against new kernel vulnerabilities that could undermine container isolation and allow a service to escape its boundary, Snap enforces the usage of auto-upgrading node pools where possible. They also maintain the ability to upgrade the entire fleet at any given moment. A dedicated pipeline alerts on-call teams about disclosed security bulletins, ensuring that critical vulnerabilities are patched promptly.

These controls, when combined, form a comprehensive security architecture that provides strong isolation between services, secures access to them, and hardens the entire path to production in Snap's multi-tenant Kubernetes environment.

Demo / Proof of Concept

▶ Watch: Mitigation: Workload Identity Federation for per-service least privilege iden... (13:00)

The talk provided a detailed architectural overview and explanation of the security controls implemented at Snap, including how they function within the platform. However, the presentation did not include a live demonstration or a specific proof of concept of these controls in action. The speakers focused on describing the design, implementation, and challenges encountered during their journey.

Defensive Implications

▶ Watch: Organizational and technical challenges encountered during multi-tenant platf... (22:00)

The journey and solutions presented by Snap offer several critical defensive implications for organizations operating or planning to adopt multi-tenant Kubernetes clusters:

1. Embrace a Zero-Trust Model for Access: Limit direct developer access to production clusters. Instead, enforce a golden CI/CD path for all deployments and changes. Implement a zero-trust authentication proxy for all control plane access, ensuring context-aware and audited connections. This minimizes the attack surface from internal threats and ensures accountability.
2. Implement Strong Admission Control: Leverage Kubernetes admission controllers to enforce security policies at deployment time. Proactively block known bad configurations, such as privileged containers, host namespace access, or unapproved capabilities. Basing these policies on Pod Security Standards provides a strong foundation for hardening workloads.
3. Adopt Least Privilege Identities: Move away from default, overly permissive node identities. Implement Workload Identity Federation or similar mechanisms to provision per-service identities with only the necessary permissions. Extend this principle to all Kubernetes components and infrastructure add-ons.
4. Harden the Software Supply Chain: Integrate image signature validation into your CI/CD pipeline and admission control. Ensure that only images built and signed by trusted sources are allowed to run in production. Transitioning to immutable image digests instead of mutable tags is a crucial step for integrity.
5. Ensure Data Isolation: In multi-tenant environments, logs and other sensitive data must be isolated per tenant. Configure logging solutions (e.g., GCP log sinks) to route logs to service-owned projects or buckets, preventing unauthorized access to co-tenant data.
6. Develop a Co-Tenancy Policy: Don't treat all workloads equally. Implement a policy to evaluate services based on their sensitivity, internet exposure, and business criticality. Use this policy to strategically collocate services on clusters, limiting the blast radius by isolating high-risk or highly sensitive applications.
7. Prioritize Proactive Vulnerability Management: Automate node pool upgrades and maintain the capability to rapidly patch the entire fleet. Establish clear processes and alerting for security bulletins to ensure prompt remediation of critical vulnerabilities that could undermine container isolation.
8. Manage Behavioral Change: Recognize that implementing strict security controls often requires changes in developer behavior (e.g., moving away from direct production access). Provide clear, easy-to-use alternative paths (like robust CI/CD) and "break glass" procedures for critical situations to foster adoption and trust.
9. Account for Cloud Provider Differences: Even with Kubernetes as an abstraction, be prepared for subtle differences in networking, IAM, and other infrastructure components between cloud providers (AWS, GCP). These differences can significantly impact engineering cost and complexity when designing multi-cloud solutions.

By adopting these defensive strategies, organizations can build a more resilient and secure multi-tenant Kubernetes platform, effectively mitigating the inherent risks of shared compute environments.

Key Takeaways

• Multi-tenancy in Kubernetes requires explicit security controls: Kubernetes was not designed for secure multi-tenancy out-of-the-box, necessitating robust, layered security measures to prevent privilege escalation and ensure isolation between co-located services.
• Least privilege is paramount for identities: Moving from broad node identities to per-service identities via Workload Identity Federation is crucial for minimizing the attack surface and adhering to the principle of least privilege.
• A strong CI/CD pipeline is the backbone of production security: Enforcing a golden CI/CD path with image signature validation and strict RBAC for developer access ensures provenance, integrity, and auditability of all deployments.
• Admission controllers are critical enforcement points: Custom admission controllers, leveraging Pod Security Standards, can proactively block insecure configurations and prevent known privilege escalation paths before they reach production.
• Organizational and technical challenges are inevitable: Migrating existing services, managing developer behavioral changes, securing leadership buy-in, and handling bespoke service requirements across multiple cloud providers are significant hurdles that require patience and strategic planning.
• Continuous evolution towards managed services is the future: Decoupling service deployments from underlying infrastructure management, with automated patching and upgrades, enhances security (especially supply chain security) and allows engineering teams to focus solely on business logic.

About the Speaker(s)

Shrikant Pandhare is an Engineering Manager for the Infrastructure Security team at Snap. His work focuses on securing Snap's service mesh platform, which is built on Kubernetes. He has experience with cloud-native infrastructure across both AWS and Google Cloud public clouds.

Sagiv Sheelo is a Security Engineer working on the Infrastructure Security team at Snap, alongside Shrikant. His expertise lies in cloud and Kubernetes security controls, contributing to the design and implementation of the security measures discussed in the talk.