Zero Trust at Shopify Scale: Automating MTLS Across Thousands of Serv... Dani Santos & Michelle Mali

Zero Trust at Shopify Scale: Automating MTLS Across Thousands of Services

Speakers: Dani Santos, Senior Infosec Engineer, Shopify; Michelle Mali, Infosec Engineer, Shopify

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=T-nN86wTebM

Overview

In this insightful talk, Dani Santos and Michelle Mali pull back the curtain on Shopify's extensive journey to implement mutual TLS (MTLS) at an unprecedented scale. Addressing an audience at KubeCon EU, the speakers delve into the complexities of securing internal service-to-service communication across an infrastructure comprising millions of pods, thousands of nodes, and hundreds of Kubernetes clusters. The presentation emphasizes the foundational role of Zero Trust principles in modern security architectures, highlighting how Shopify has embraced these tenets to fortify its vast and dynamic environment.

The core of their discussion revolves around establishing robust workload identities using the Spiffy (Secure Production Identity Framework for Everyone) standard and automating the entire certificate lifecycle. They detail Shopify's strategic choices, including leveraging Google Cloud's Certificate Authority Service, implementing custom solutions where off-the-shelf tools fell short, and meticulously planning for certificate rotation and observability. This talk is crucial for organizations grappling with distributed systems, cloud-native complexities, and the imperative to move beyond traditional perimeter-based security models.

Shopify's experience provides a practical blueprint for tackling the challenges inherent in large-scale Zero Trust adoption. By sharing their architectural decisions, technical deep dives into custom tooling, and lessons learned from real-world scenarios, Santos and Mali offer invaluable insights for security engineers, platform architects, and developers aiming to secure their own cloud-native deployments. The talk not only showcases Shopify's innovative approach but also underscores the critical need for scalable, automated, and observable security mechanisms in today's rapidly evolving digital landscape.

Background

▶ Watch: Talk introduction and agenda overview (0:00)

The concept of Zero Trust, pioneered by John Kindervag in 2009, posits that "trusting is a vulnerability" and security must be designed with the strategy "never trust, always verify." This philosophy has seen a dramatic surge in interest, with a 13% rise noted in O'Reilly's 2025 technology trends, largely driven by the seismic shifts of the 2020 pandemic. The widespread adoption of remote work, bring-your-own-device (BYOD) policies, and accelerated migration to cloud infrastructure rendered traditional perimeter-based security models, often likened to a "castle-and-moat," increasingly ineffective. In a scattered workforce and distributed system, the perimeter becomes porous, if not entirely dissolved.

At the heart of Zero Trust is the imperative to verify the identity of every entity requesting access, whether human or non-human. Non-human entities, referred to as workloads—such as VMs, containers, applications, and services—represent the vast majority of identities in modern infrastructure, outnumbering human identities by an approximate ratio of 45 to 1. Authenticating these machine identities presents unique challenges: they cannot rely on traditional security measures like Multi-Factor Authentication (MFA), their creation is often spontaneous and decentralized, they tend to accumulate excessive permissions, and they frequently outlive their intended purpose. Furthermore, a small misconfiguration in their permissions can lead to critical system outages.

Just as passports serve as authenticated identity documents in the physical world, X.509 certificates function as digital identity documents for workloads. These certificates, issued by trusted Certificate Authorities (CAs), employ cryptographic signatures to establish authenticity, much like watermarks on a passport. However, managing machine identities at scale is complicated by the heterogeneous nature of modern infrastructure, which often involves multiple technology stacks, a mix of cloud services and on-premise systems, and a rapid pace of development. This complexity necessitates a standardized, interoperable approach to identity management.

This is where Spiffy (Secure Production Identity Framework for Everyone), a CNCF graduated project, comes into play. Spiffy provides an open-source framework designed to deliver a secure and standardized way to manage identity in cloud-native environments. A Spiffy ID, represented as a URI, uniquely identifies workloads. Shopify structures its Spiffy IDs to include rich metadata such as the trust domain, workload identity pool, Google service account identity, and corresponding project, all embedded within the certificate's Subject Alternative Name (SAN). While Spire is the reference implementation for Spiffy, Shopify opts for Google Cloud Certificate Authority Service and identity reflection for federated workloads, allowing them to leverage Spiffy benefits without running Spire agents directly.

Key Findings

▶ Watch: Why MTLS is crucial for workload identity (2:20)

Shopify's journey to automate MTLS at scale revealed several critical findings and strategic decisions:

• Spiffy Adoption with Cloud Provider Integration: Instead of deploying Spire agents, Shopify successfully integrates Spiffy IDs using Google Cloud Certificate Authority Service and identity reflection. This approach provides standardized workload identities across their infrastructure while offloading the operational burden of managing dedicated Spire agents.
• Custom Solutions for Ecosystem Gaps: When faced with a mismatch between Kafka's default requirement for Distinguished Names (DNs) for ACLs and Google CA Service's provision of SAN URIs for Spiffy IDs, Shopify developed a custom Kafka principal builder. This highlights the necessity of in-house development when open-source or vendor solutions don't perfectly align with specific infrastructure needs.
• Strategic Choice of Ingress NGINX over Service Mesh: For the majority of its millions of pods, Shopify deliberately chose Kubernetes native Ingress NGINX for internal service-to-service MTLS over more complex service mesh solutions like Istio. This decision was driven by the significant overhead (CPU, memory, operational burden) that sidecar proxies would introduce at Shopify's extreme scale, demonstrating a pragmatic approach to balancing security with operational efficiency.
• Automation as a Mandate for Certificate Management: The talk strongly emphasizes the critical need for automated certificate issuance, renewal, and rotation. Shopify's experience with manual key management leading to partial outages (e.g., a Cloudflare key expiring after four months) underscores that manual processes are error-prone and unsustainable at scale. This led to the development of a custom Golang tool for client certificate management, complementing Cert Manager for GKE workloads.
• Three-Tier PKI for Enhanced Security: Shopify implemented a three-tier Public Key Infrastructure (PKI) hierarchy (Root CA, Intermediate CA, Leaf Certificates). This architecture provides an essential buffer of security, allowing for revocation and rotation of intermediate CAs without compromising the root of trust, significantly improving resilience and incident response capabilities.
• Comprehensive Observability for Proactive Issue Detection: A robust observability stack, incorporating Grafana, Prometheus, Tempo, and GCP Logs Explorer, is fundamental to Shopify's certificate management. Custom metrics from their in-house tools, combined with kube_job_status_failed metrics, provide early warnings for certificate issuance or renewal failures, preventing potential outages.
• Orchestrated Deployment for Reliability: To ensure certificates are available before services start, Shopify utilizes Argo CD Sync Waves (and previously Carvel Kapp) to enforce strict deployment order. This guarantees that certificate provisioning jobs complete successfully before dependent applications are initialized.

Technical Deep Dive

▶ Watch: Introduction to Spiffy for secure workload identity (4:45)

Shopify's implementation of Zero Trust through automated MTLS is a masterclass in adapting industry standards and building custom solutions to meet extreme scale.

Workload Identity with Spiffy and Google Cloud CA Service

At Shopify, Spiffy IDs are the cornerstone of workload identity. Rather than deploying Spire agents, Shopify leverages Google Cloud Certificate Authority Service combined with identity reflection. This means their workloads, federated through Google's IAM, obtain Spiffy IDs directly from Google's managed CA service, streamlining the identity issuance process. The Spiffy URI structure is carefully designed to embed critical metadata: spiffe://<trust_domain>/<workload_identity_pool>/<google_service_account>/<project_id>. This rich structure, captured in the certificate's SAN, allows for fine-grained access control decisions.

A significant challenge arose with Kafka infrastructure. By default, Kafka's Access Control Lists (ACLs) are managed using the certificate's Distinguished Name (DN). However, certificates issued via Google's CA Service in reflected Spiffy mode prioritize the SAN URI, making the DN mutable and unverifiable. To bridge this gap, Shopify developed a custom Kafka principal builder. This builder parses the Spiffy ID from the SAN URI and maps it to appropriate Kafka ACLs, effectively allowing Spiffy-based authorization for Kafka. While open-source solutions and Kafka Improvement Proposals (KIPs) exist for similar problems, Shopify opted for a tailored in-house implementation.

Service-to-Service Authentication Mechanisms

Shopify evaluated several approaches for internal service-to-service authentication at its vast scale:

• Service Mesh (e.g., Istio): Service meshes provide strong workload identity, automatic certificate rotation, and MTLS via sidecar proxies. While appealing for their comprehensive security features, they introduce significant complexity and operational overhead. At Shopify's scale of millions of pods, the extra CPU and memory consumption of sidecars, along with the management burden of an additional control plane, proved prohibitive for general adoption. They do use Envoy proxies for specific, targeted use cases, such as in Elasticsearch.
• Cloud Provider Solutions (e.g., Google's Application Load Balancers - ALB): Google Cloud's internal ALBs offer managed MTLS, providing custom MTLS headers to backend services. Benefits include managed service resilience, integration with other cloud services (IAM, monitoring), and built-in security features like DDoS protection. However, the drawbacks include vendor lock-in, restriction to provider-defined features, and potentially escalating costs with high traffic volumes.
• Kubernetes Native (Ingress NGINX): Shopify has relied on Ingress NGINX since 2018 for its Kubernetes ingress. Despite exploring newer options like the Gateway API, Ingress NGINX remains a primary tool for internal MTLS. It supports various authentication methods, but Shopify is actively migrating away from basic auth due to its inherent security flaws (unverified identity, shared long-lived credentials, lack of automated rotation).

Shopify's preference for Ingress NGINX for MTLS is configured using specific annotations:

• nginx.ingress.kubernetes.io/auth-tls-secret: Points to a Kubernetes secret containing the client CA certificates.
• nginx.ingress.kubernetes.io/auth-tls-verify-depth: Sets the verification depth for the certificate chain (e.g., 2 for root and subordinate CA).
• nginx.ingress.kubernetes.io/auth-tls-verify-client: Set to on to enforce client certificate verification.
• nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: Crucially, this passes the entire raw client certificate to the upstream service.

The challenge with auth-tls-pass-certificate-to-upstream is that it forwards the full PEM-encoded client certificate as a URL-encoded string, which is cumbersome to parse for extracting the Spiffy URI needed for ACLs. During a "hack day," Shopify engineers experimented with a Lua script (assisted by GPT) to dynamically extract the SAN Spiffy URI from the client certificate and forward it as a cleaner request header. While functional, the solution was acknowledged to be a "hack" due to code quality concerns, and was not upstreamed. For services written in Golang, URL query unescape is used to simplify parsing.

Certificate Management and Rotation

Managing certificates at Shopify's scale necessitates a robust PKI and automated tools:

• Three-Tier PKI Hierarchy: Shopify employs a three-tier PKI structure:

1. Root CA: Kept offline and highly secured, solely used to sign intermediate CAs.
2. Intermediate CA: The workhorse, signing all leaf certificates. This layer provides a critical security buffer, allowing for revocation and rotation without touching the root of trust.
3. Leaf Certificates: Issued to individual services for day-to-day MTLS authentication.

• CA Rotation: This is a complex but vital process. Shopify initiates root CA rotation one year before expiry and subordinate CA rotation six months before expiry, using alert rules to notify teams.
• Root CA Rotation: Involves creating a new root CA, adding its certificate to trust stores, creating new subordinate CAs signed by the new root, and finally deleting the old root CA.
• Subordinate CA Rotation: Involves creating a new subordinate CA, disabling existing subordinate CAs, and deleting the expired subordinate CA.
• Client and Server Certificate Management:
• Cert Manager: A CNCF graduated project, Cert Manager automates the issuance and renewal of certificates for many of Shopify's GKE workloads. It integrates seamlessly with various cloud providers; Shopify transitioned from HashiCorp Vault to Google Certificate Authority Service using Google CA's issuer.
• In-house Golang Tool: For non-Kubernetes environments (VMs, local development), serverless applications (Google Cloud Run), and clusters without Cert Manager, Shopify developed a lightweight, in-house Golang tool. This binary can be run as a sidecar container, a background process, or a one-time job. For Cloud Run, it's deployed as a scheduled job to handle renewals, storing certificates in Google Secret Manager. The tool supports multiple storage options: local file system, GCS bucket, Google Secret Manager, and Kubernetes secrets, requiring specific IAM permissions.
• Custom Tool for Client Certificates: After evaluating both Cert Manager and their custom tool, Shopify decided to standardize on their custom tool for all client certificates, adopting a pattern where jobs and cron jobs handle creation and updates.
• Deployment Sequencing with Argo CD Sync Waves: A critical aspect is ensuring certificate data is available before a service container initializes. Shopify uses Argo CD Sync Waves (similar to Carvel Kapp) to manage deployment order. An initial job runs once to create the Kubernetes secret containing the certificates, which is then mounted by the application. Subsequent renewals are handled by cron jobs.
• Cron Job Scheduling: To prevent a "thundering herd" problem with thousands of concurrent renewal requests, cron jobs run every six hours (four times a day) but use a deterministic randomization function based on the service name to stagger their execution times.
• Renewal Logic: The custom tool attempts to renew a certificate when it reaches 50% of its lifetime. This aggressive renewal schedule, combined with four daily runs, ensures that any issuance failures trigger alerts well in advance of expiry (e.g., 20 alerts over 5 days for a 10-day certificate), providing ample time for remediation.

Observability Stack

Shopify's observability stack is crucial for monitoring the certificate lifecycle and preventing outages:

• Grafana: For visualizing metrics, logs, and traces through customizable dashboards.
• Prometheus: A time-series database for collecting and querying metrics.
• Tempo: For application traces.
• GCP Logs Explorer: For ingesting and analyzing CA-related logs.

Both Cert Manager and Shopify's in-house tool expose Prometheus-format metrics. For ephemeral jobs, which terminate before Prometheus can scrape them, Shopify uses two strategies:

1. kube_job_status_failed metric: An early warning system from kube-state-metrics (part of kube-prometheus-stack) to detect job failures.
2. StatsD with Exporters: Shopify's in-house tool instruments metrics using StatsD, which immediately pushes metrics via UDP. These are then converted to Prometheus format by StatsD exporters and consumed by Prometheus, allowing for reliable capture of issuance failures and request durations, even from short-lived processes.

Demo / Proof of Concept

▶ Watch: Shopify's Spiffy implementation with Google Cloud (6:05)

The talk included a compelling demo showcasing Shopify's MTLS solution in action within a local Kind cluster, featuring two services (A and B) communicating via Ingress NGINX.

The demo's synopsis outlined the following flow:

1. A key pair is generated by the cert tool.
2. The cert tool authenticates with Google's IAM Workload Identity Federation, obtaining an ID token exchanged for an access token via the Security Token Service.
3. Using the access token, the cert tool makes a Certificate Signing Request (CSR) in reflected Spiffy mode to Google CA Service to get a Spiffy URI as workload identity.
4. The cert tool stores the certificate data (subordinate/leaf bundle, private key, shared root CA) in a Kubernetes secret named mtlserts.
5. The cert tool pushes metrics to a Prometheus Push Gateway before termination, ensuring Prometheus scrapes them.
6. Prometheus rules evaluate these metrics, generating alerts for issues like failed requests.
7. Metrics and alerts are visualized in Grafana.

The demo began by illustrating a common problem: applying multiple Kubernetes manifest files in the wrong order leads to errors. The solution demonstrated was Argo CD, which uses sync wave annotations to ensure resources are deployed in the correct sequence. For instance, the namespace was set with sync wave: -1 (prioritized), the certificate provisioning job with sync wave: 1, deployments with sync wave: 2 (dependent on the job), and services/ingresses with higher sync waves. This ensures that the mtlserts secret is created and available before the application pods attempt to start.

Next, an Access Control List (ACL) demonstration was performed. Service A successfully accessed the /internal/spiffy endpoint on Service B, with the demo highlighting the Spiffy ID used for authentication. However, when Service A attempted to access /internal/Z, it was met with an "Access denied" error, despite using a valid certificate. This showcased the ACL in action, demonstrating that authentication (valid certificate) does not automatically grant authorization; fine-grained access control based on the Spiffy ID is enforced.

Finally, a failure scenario was presented. Users reported application issues, leading the presenter to Grafana. An alert from the cert tool indicated a "certificate issuance" error. Further investigation of the job's status in the terminal confirmed it never completed. Examining the pod logs revealed the root cause: the service account responsible for issuing certificates lacked the correct IAM permissions, triggering the alert and preventing certificate renewal. This effectively demonstrated the end-to-end monitoring and alerting capabilities that Shopify has built into its system to proactively detect and diagnose certificate-related issues.

Defensive Implications

▶ Watch: Broader MTLS implementation challenges at Shopify's scale (7:50)

Organizations can draw several critical defensive implications from Shopify's extensive MTLS and Zero Trust implementation:

• Embrace Zero Trust and MTLS: The transition from perimeter-based security to a Zero Trust model is non-negotiable for modern, distributed infrastructures. MTLS should be the primary mechanism for authenticating non-human entities (workloads) in service-to-service communication.
• Standardize Workload Identity with Spiffy: Adopting Spiffy provides a robust, standardized, and interoperable framework for workload identity across heterogeneous environments. Leverage cloud provider integrations (like Google Cloud CA Service with identity reflection) to simplify deployment and management where possible.
• Strategically Choose Authentication Solutions: Evaluate service mesh, cloud provider, and Kubernetes native solutions based on your organization's specific scale, complexity, and operational tolerance. Shopify's choice of Ingress NGINX over a full-blown service mesh for widespread MTLS highlights the importance of balancing security benefits against performance overhead and management burden.
• Automate Certificate Lifecycle Management: Manual certificate management is a major vulnerability. Invest in tools like Cert Manager or develop custom solutions (like Shopify's Go tool) to automate issuance, renewal, and rotation of both CA and leaf certificates. This eliminates human error and prevents outages due to expired certificates.
• Implement a Robust PKI Hierarchy: A three-tier PKI (Root, Intermediate, Leaf CAs) significantly enhances security and resilience. It provides layers of defense, allowing for more agile certificate rotation and revocation without compromising the ultimate root of trust.
• Prioritize Comprehensive Observability: Establish a robust monitoring and alerting stack for the entire certificate lifecycle. Collect metrics (e.g., expiry dates, renewal failures), logs, and traces. Utilize tools like Prometheus, Grafana, and kube_job_status_failed metrics to detect and alert on certificate issues proactively, preventing them from escalating into service outages.
• Enforce Deployment Sequencing: Use orchestration tools like Argo CD Sync Waves or Carvel Kapp to ensure that certificate provisioning jobs complete before dependent applications start. This prevents race conditions and ensures services always have valid certificates at bootstrap.
• Implement Fine-Grained ACLs: Beyond mere authentication, enforce granular Access Control Lists (ACLs) based on the rich metadata embedded in Spiffy IDs. This ensures that even authenticated services are only authorized to access the specific resources they require, adhering to the principle of least privilege.
• Plan for CA Rotation Diligently: CA rotation is a complex, high-stakes operation. Develop detailed, well-tested procedures and initiate the process well in advance of expiry (e.g., one year for root CAs, six months for subordinate CAs) to ensure a smooth transition.

Key Takeaways

• Zero Trust and MTLS are foundational: The accelerated adoption of Zero Trust principles makes mutual TLS (MTLS) essential for secure service-to-service communication, moving beyond outdated perimeter security.
• Spiffy is the standard for workload identity: Spiffy provides a robust, standardized, and interoperable framework for identifying and authenticating workloads in cloud-native environments.
• Authentication solutions must scale with your organization: There's no one-size-fits-all. Carefully evaluate the trade-offs between complexity, overhead, and security when choosing solutions like service meshes, cloud provider offerings, or Kubernetes native approaches like Ingress NGINX.
• Automated certificate management is non-negotiable: Manual processes are prone to error and lead to outages. Invest in open-source tools like Cert Manager or develop custom solutions for automated issuance, renewal, and CA rotation.
• Robust PKI and meticulous CA rotation planning are critical: A three-tier PKI enhances security, and thorough planning for CA rotation, initiated well in advance of expiry, is vital to prevent widespread trust issues.
• Comprehensive monitoring and alerting prevent outages: Implement an observability stack (e.g., Prometheus, Grafana) to proactively detect certificate issuance or renewal failures, providing early warnings before services are impacted.

About the Speaker(s)

Michelle Mali is an Infosec Engineer at Shopify, where she has been a key contributor to the MTLS project for nearly five years, focusing on securing Shopify's vast infrastructure. Her work involves ensuring the integrity and security of internal communications at an immense scale.

Dani Santos is a Senior Infosec Engineer at Shopify, having joined the team in 2020. His recent projects are centered around increasing the adoption of MTLS for internal service authentication and leveraging attested identities for Access Control Lists (ACLs), driving forward Shopify's Zero Trust initiatives.