How We Solved TLS at Scale: Self-Service, Multi-Tenant Cert-manager - Erik Godding Boye & Tim Ramlot

How We Solved TLS at Scale: Self-Service, Multi-Tenant Cert-manager

Speakers: Erik Godding Boye, Cert-manager Maintainer; Tim Ramlot, CyberArk, Cert-manager Maintainer

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=gWgagjHtnlE

Overview

This presentation by Erik Godding Boye and Tim Ramlot, both maintainers of the CNCF-graduated cert-manager project, delves into the intricate challenges and sophisticated solutions for managing TLS certificates at scale within a multi-tenant Kubernetes and OpenShift environment. Specifically, Erik shares his journey and the advanced setup he engineered for a critical client, Statnett, the Norwegian Transmission System Operator (TSO) responsible for Norway's power grid. The talk highlights how cert-manager, augmented by other ecosystem projects like trust-manager and approval-policy, can be leveraged to provide self-service certificate provisioning to independent teams while maintaining stringent security and policy enforcement.

The core problem addressed is the secure authentication of thousands of microservices interacting with critical systems like Kafka, particularly in an air-gapped, highly regulated environment. The speakers meticulously explain the evolution from an initial state of no authentication to a robust, automated, and policy-driven MTLS solution. This article will dissect their architectural decisions, technical implementations, and the lessons learned, offering invaluable insights for platform engineers and security professionals grappling with similar large-scale, multi-tenant TLS requirements.

The significance of this work extends beyond mere certificate management; it underpins the security posture of critical national infrastructure. By enabling autonomous, secure TLS for numerous independent teams, the solution not only mitigates significant security risks (like unauthenticated Kafka access) but also streamlines development workflows and ensures operational resilience, even in the face of complex challenges like root Certificate Authority (CA) rotations.

Background

▶ Watch: Introduction to TLS at scale and cert-manager (0:00)

Before the implementation discussed in this talk, Statnett faced a challenging security landscape. Their environment comprised multiple Kubernetes clusters and Kafka clusters running on virtual machines, serving numerous independent teams (tenants) deploying thousands of microservices. A critical flaw was the lack of authentication for Kafka clusters, meaning sensitive data was accessible to anyone with network access to the clusters. The imminent launch of a major project for automatic energy market balancing necessitated an immediate and robust solution for data security.

The team explored Kafka's native authentication mechanisms, narrowing down to SASL (requiring integration with an identity provider and a registration process) and SSL/MTLS (authenticating using mutual TLS). Given the organization's traditional operational model with many manual processes, MTLS was initially preferred to avoid dependency on a potentially cumbersome identity provider registration process. A critical discovery during this phase was that Kafka does not support certificate revocations. This limitation immediately highlighted the necessity for certificates with relatively short lifetimes, further emphasizing the need for robust automation.

The requirements quickly crystallized: automate MTLS provisioning for thousands of microservices, enable self-service for tenants to provision their own certificates without manual intervention, and manage the short-lived certificates efficiently. An initial proof-of-concept (PoC) explored Spiffy and Spire, open-source projects tailored for workload identities. While promising, Spire encountered significant challenges running on their target OpenShift clusters due to conflicts with SELinux policies. Although many of these issues are now resolved, at the time, this roadblock necessitated a pivot.

The team then turned to cert-manager, a CNCF graduated project designed for X.509 certificate management on Kubernetes and OpenShift. Operating in an air-gapped environment meant the common use case of obtaining certificates from Let's Encrypt was not viable. However, the team retained a key goal from their Spiffy/Spire exploration: to issue certificates containing X.509 SVIDs (Secure Verifiable Identity Documents) with URI attributes. These URIs would encode crucial workload identity information, such as the trust domain (representing the cluster) and, critically, the namespace where the workload resided. This was essential for the multi-tenant model, where authorization in Kafka (using Kafka ACLs) would depend on the workload's namespace identity to prevent tenants from accessing unauthorized data.

The inherent challenge with cert-manager in a multi-tenant setup was two-fold: preventing tenants from obtaining certificates they were not entitled to and enforcing the specific X.509 SVID format required for Kafka ACLs. This laid the groundwork for the two-phase solution presented in the talk.

Key Findings

▶ Watch: Choosing MTLS for Kafka authentication; short cert lifetimes (4:20)

The talk presents a compelling journey through two distinct phases, dubbed V1 and V2, each addressing the escalating complexity of TLS management at scale within a multi-tenant, critical infrastructure environment.

V1 Solution: Rapid Deployment with Controlled Access (Acknowledging Corners Cut)

• Initial Security Posture: To rapidly secure the environment, a highly restrictive approach was adopted. All RBAC permissions for direct access to cert-manager custom resources (CRs) were disabled for tenants. The ingress-shim was also disabled to prevent users from obtaining certificates simply by annotating Ingress resources. Similarly, CSI drivers were blocked, as they too could lead to certificate requests bypassing central control.
• Controlled Access: Instead of direct tenant interaction, a custom Application Operator and a new Identity Provider controller were introduced. These controllers watched service accounts annotated by users. Upon detection, the Identity Provider would interact with cert-manager to provision a certificate matching that service account's context.
• Problematic Trust Anchor: For establishing trust, the CA.CRT field from the cert-manager-issued secret was directly mounted into workload pods. While functional for the immediate need, this approach was identified as fundamentally flawed by the speakers. It created significant hurdles for root CA rotation without downtime and incorrectly used the client's CA chain to verify the server's identity.

V2 Solution: Robust, Policy-Driven Automation with Seamless Trust Management

• Addressing Root CA Rotation: The critical flaw of V1's trust anchor management was resolved by integrating Trust Manager, another project from the cert-manager ecosystem. Trust Manager allows for the dynamic management of trust bundles, enabling multiple CAs to be trusted simultaneously. This facilitates graceful root CA rotation by allowing new CAs to be introduced and old ones phased out without application downtime.
• Unlocking Cert-manager with Policy Enforcement: To move beyond the restrictive V1 lockdown, cert-manager CRs, the ingress-shim, and CSI driver support were re-enabled. This was made possible by the introduction of Approval Policy, a cert-manager plugin designed for defining and enforcing granular policies on certificate requests.
• Common Expression Language (CEL) Integration: A key contribution by Erik was the addition of CEL support to Approval Policy. This allowed for highly flexible and dynamic policy definitions, such as ensuring that the URI or DNS names in a certificate request precisely match the namespace in which the request originated, thereby enforcing multi-tenant isolation.
• Enhanced CA Backend: The in-cluster CA issuer, initially used in V1 (and acknowledged as less secure), was replaced by an external HashiCorp Vault PKI engine. This provided a more secure and robust signing backend for cert-manager, with additional policy layers enforced within Vault itself.
• Operational Resilience: The importance of comprehensive monitoring (using Prometheus and Grafana) for certificate expiration, CA health, and policy adherence was emphasized. The use of the Reloader Kubernetes project was also highlighted to ensure applications automatically reload certificates and trust bundles after renewal or rotation, preventing downtime.

In essence, the journey from V1 to V2 demonstrates a pragmatic evolution from a tightly controlled, somewhat brittle solution to a highly automated, flexible, and secure architecture capable of supporting self-service TLS at massive scale in a multi-tenant, air-gapped environment, all while adhering to strict security requirements for critical infrastructure.

Technical Deep Dive

▶ Watch: First attempt with Spiffy/Spire; OpenShift sLinux issues (6:20)

The technical implementation for solving TLS at scale involved a sophisticated interplay of several Kubernetes-native tools and custom controllers, evolving significantly from an initial, more restrictive approach to a flexible, policy-driven system.

Initial Kafka Authentication and Workload Identity

The core problem was to authenticate thousands of microservices connecting to Kafka clusters. The chosen method was Mutual TLS (MTLS), which required each workload to present a client certificate and trust the Kafka server's certificate. Since Kafka doesn't support certificate revocation, certificates needed short lifetimes, demanding automation.

To enable fine-grained authorization using Kafka ACLs, the client certificates needed to carry specific identity information. The team adopted the concept of X.509 SVIDs from the Spiffy project. These SVIDs encode workload identity as a URI attribute within the certificate, following a format like spiffe://trust-domain/ns/namespace-name/sa/service-account-name. For Statnett, the trust-domain identified the cluster, and namespace-name was crucial for multi-tenancy, ensuring tenants could only obtain certificates valid for their own namespaces. The service-account-name was less critical for multi-tenancy as it's within the same namespace.

V1: Controlled Cert-manager Access

In the initial V1 setup, a ClusterIssuer linked to the organization's private PKI (a shared common root CA) was deployed. To prevent tenants from issuing arbitrary certificates:

• RBAC permissions for cert-manager's custom resources (Certificate, CertificateRequest, Issuer, ClusterIssuer) were disabled for regular users.
• The ingress-shim, which automatically provisions certificates for Ingress resources, was disabled.
• CSI drivers for certificate injection were also blocked.

To allow legitimate certificate provisioning, a custom Identity Provider controller was developed. This controller watched Kubernetes ServiceAccount resources for a specific annotation. When an annotated ServiceAccount was detected, the controller would:

1. Create a Certificate resource for that ServiceAccount, ensuring the URI SAN (Subject Alternative Name) matched the namespace and cluster context.
2. Cert-manager would then process this Certificate request, submit a CertificateSigningRequest to the configured ClusterIssuer (which, at the time, was an in-cluster CA Issuer, later migrated to Vault), and finally issue the certificate.
3. The issued certificate, along with its private key and the CA chain, was stored in a Kubernetes Secret. Workload pods (e.g., Kafka clients) would then mount this secret, accessing tls.crt (client certificate), tls.key (private key), and ca.crt (CA bundle).

The V1 Trust Establishment Problem ("Don't Do This")

A significant shortcut taken in V1 was using the ca.crt from the issued secret as the trust anchor for the Kafka client. This ca.crt actually contained the entire certificate chain of the client's issuer, including the company's root CA. While it worked because all certificates were rooted in the same shared company CA, this approach is fundamentally flawed for two reasons:

1. Root CA Rotation Downtime: If the root CA needed to be rotated, replacing ca.crt in the secret would instantaneously switch the trust anchor. Any clients still using certificates from the old CA would immediately lose trust, leading to downtime. A proper CA rotation requires a transitional period where clients trust both the old and new CAs.
2. Incorrect Trust Verification: The ca.crt provided in the client's certificate secret is intended to verify the client's own chain. For MTLS, the client needs a trust anchor to verify the server's certificate. While in this specific setup, both client and server CAs were the same, this is not a robust or generalizable solution and can lead to subtle failures if the server's CA chain diverges.

V2: Robust Trust Management with Trust Manager

To address the root CA rotation problem and establish proper trust, Trust Manager was introduced. Trust Manager is a Kubernetes controller that manages trust bundles.

• It uses a custom resource called Bundle to define sources of trusted certificates (e.g., ConfigMaps containing root CAs) and targets (ConfigMaps in specific namespaces where these trust bundles should be made available).
• When a Bundle resource is created, Trust Manager ensures that ConfigMaps containing the specified trusted CAs are propagated to the designated target namespaces.
• Workload pods then mount these Trust Manager-generated ConfigMaps as their trust stores, containing only the necessary root CAs for verifying server identities.

This approach enables seamless root CA rotation:

1. The Bundle resource is updated to include both the old and new company CAs.
2. Trust Manager propagates this updated bundle to all target ConfigMaps.
3. Clients are restarted (using Reloader, as discussed later) to pick up the new trust bundle. At this point, clients trust both CAs.
4. Kafka servers can then be gradually migrated to certificates issued by the new CA.
5. Once all Kafka servers (and other MTLS endpoints) are using new CA certificates, the old CA can be removed from the Bundle.

This multi-phase approach ensures zero-downtime CA rotation, a critical requirement for a production system.

V2: Policy-Driven Cert-manager Access with Approval Policy and CEL

With Trust Manager handling trust, the next step was to unlock cert-manager for self-service while maintaining security and multi-tenancy constraints. This was achieved using Approval Policy, a cert-manager plugin that allows defining policies for CertificateRequest resources.

• By default, cert-manager includes an approve-all approver. This was disabled.
• Instead, CertificateRequestPolicy custom resources were created. These policies define conditions under which a CertificateRequest should be automatically approved or denied.
• Erik's key contribution was adding Common Expression Language (CEL) support to Approval Policy. CEL allows for powerful, dynamic policy expressions.

Example Policies using CEL:

• Client Certificates: A policy could enforce that the URI SAN in a CertificateRequest must begin with spiffe://trust-domain/ns/<request-namespace>/. This ensures that a tenant in namespace-A cannot request a certificate with a URI identifying it as belonging to namespace-B.
• Server Certificates: For server certificates, a policy could ensure that DNS SANs have a suffix matching the request-namespace (e.g., my-service.namespace-A.svc.cluster.local).

These policies, applied at the CertificateRequest level, allow tenants to directly create Certificate resources (which automatically generate CertificateRequests) or even CertificateRequest resources, while Approval Policy acts as a gatekeeper, ensuring adherence to organizational security policies and multi-tenancy boundaries. This enabled re-enabling the ingress-shim and considering CSI drivers, as policies would now govern their certificate requests.

External CA Backend: HashiCorp Vault

For enhanced security and robust CA management, the in-cluster CA Issuer was replaced with HashiCorp Vault's PKI engine. Cert-manager now acts as a client to Vault, sending CertificateSigningRequests to Vault for signing. This offloads the sensitive private key management and signing operations to a dedicated, hardened system. Vault itself can enforce additional policies on certificate requests, providing an extra layer of security. The integration ensures that cert-manager handles the automation of issuance and renewal, while Vault provides the secure, auditable signing authority.

Monitoring and Application Reloading

Two additional critical components for operational stability were highlighted:

• Monitoring: Using Prometheus and Grafana, the team monitors cert-manager metrics, certificate expiration dates (including custom monitoring for the earliest expiring CA in the chain), and policy adherence.
• Reloader: A community Kubernetes project called Reloader is used to watch ConfigMaps and Secrets. When a certificate secret or trust bundle ConfigMap changes, Reloader triggers a rolling update of associated workloads. This ensures applications pick up new certificates and trust bundles without manual intervention or downtime.

This comprehensive technical stack provides a resilient, secure, and automated solution for managing TLS across a vast, multi-tenant Kubernetes estate.

Demo / Proof of Concept

▶ Watch: Using cert-manager to mimic Spire X509 SVIDs (7:40)

The presentation details a comprehensive journey of implementing and refining a large-scale TLS solution rather than showcasing a live demonstration. Erik Godding Boye explicitly describes the evolution of the system from its initial state to the refined V2 architecture, effectively serving as an extended proof of concept for the methodologies and tools discussed.

While there wasn't a live demo of the system in action during the talk, the speakers recounted their experiences and challenges from the initial proof-of-concept phase with Spiffy/Spire, through the development and deployment of the V1 cert-manager setup, and finally to the robust V2 architecture incorporating trust-manager and approval-policy. The detailed technical explanations, including YAML examples for Bundle and CertificateRequestPolicy resources, provide a clear understanding of how the components work together in a real-world, critical infrastructure environment. The "How We Solved TLS at Scale" title itself suggests a retrospective account of a successful implementation, which is presented as the ultimate proof of concept.

Defensive Implications

▶ Watch: Multi-tenancy challenge: preventing unauthorized certificate issuance (8:50)

The detailed architecture and operational practices presented by Erik Godding Boye and Tim Ramlot offer several critical defensive implications for organizations managing TLS at scale, especially in multi-tenant or regulated environments.

1. Enforce Workload Identity for Authorization: The adoption of X.509 SVIDs with specific URI attributes (e.g., spiffe://trust-domain/ns/namespace-name/sa/service-account-name) for client certificates is a powerful defensive strategy. This enables fine-grained, identity-based authorization mechanisms, such as Kafka ACLs, to precisely control which workloads can access specific resources based on their verified identity and namespace context. This moves beyond network-level access control to a more secure, cryptographic identity-based model.

1. Automate the Entire TLS Lifecycle: Relying on cert-manager for automated certificate issuance, renewal, and distribution is paramount. This eliminates manual processes, which are prone to human error, missed renewals, and inconsistent configurations. Automation ensures certificates are always valid and rotated frequently, mitigating risks associated with long-lived credentials.

1. Implement Robust Trust Management for CA Rotation: The use of Trust Manager is a critical defensive measure against downtime during root CA rotation. By allowing a transitional period where applications trust both old and new CAs, organizations can perform necessary CA rotations (e.g., due to compromise, policy, or expiration) without impacting services. Directly mounting CA.CRT from a certificate secret, as done in V1, is explicitly identified as an anti-pattern that introduces significant operational risk.

1. Policy-Driven Multi-Tenancy Enforcement: Approval Policy with Common Expression Language (CEL) is a powerful defensive gatekeeper in multi-tenant Kubernetes clusters. It ensures that tenants can only request certificates that adhere to predefined organizational policies and are scoped to their authorized namespaces. This prevents privilege escalation, misconfigurations, and tenants from impersonating other workloads or accessing resources outside their boundaries. Re-enabling cert-manager's native CRs, ingress-shim, and CSI drivers becomes safe when robust policies are in place.

1. Centralize and Harden CA Operations: Migrating the certificate signing backend from an in-cluster CA Issuer to an external, hardened solution like HashiCorp Vault's PKI engine significantly enhances security. Vault provides advanced features like secret management, access control, auditing, and secure key storage, making it a more secure and auditable signing authority. It also allows for a separate layer of policy enforcement on certificate requests, independent of Kubernetes.

1. Comprehensive Monitoring and Alerting: Establishing end-to-end monitoring for the TLS automation pipeline is crucial. This includes tracking certificate expiration (especially for intermediate and root CAs), cert-manager controller health, and policy violations from Approval Policy. Prometheus and Grafana are excellent tools for this. Proactive alerting on impending expirations or policy breaches prevents outages and security incidents.

1. Ensure Application Certificate Reloading: The use of tools like Reloader to trigger rolling updates when certificate secrets or trust bundle ConfigMaps change is a vital operational defense. Without automatic reloading, applications might continue to use expired certificates or outdated trust bundles, leading to service disruption or security vulnerabilities.

By integrating these defensive strategies, organizations can build a highly secure, resilient, and manageable TLS infrastructure that scales effectively with the demands of modern cloud-native environments and critical systems.

Key Takeaways

• Automate TLS Lifecycle End-to-End: Leverage cert-manager for automated certificate issuance, renewal, and distribution, reducing manual errors and operational burden, especially for short-lived certificates required by systems like Kafka.
• Enable Self-Service with Strong Policy Enforcement: Empower tenants with self-service certificate provisioning through cert-manager, but couple it with Approval Policy and Common Expression Language (CEL) to enforce strict, dynamic policies that prevent unauthorized certificate issuance and maintain multi-tenancy boundaries.
• Master Trust Management for Zero-Downtime CA Rotation: Utilize Trust Manager to properly manage and distribute trusted CA bundles. This enables seamless, zero-downtime root CA rotation by allowing clients to trust multiple CAs concurrently during the migration period, avoiding the pitfalls of directly mounting CA.CRT.
• Adopt Workload Identities for Granular Authorization: Implement X.509 SVIDs within certificates to embed workload identity information (e.g., namespace, service account). This enables powerful, identity-based authorization mechanisms like Kafka ACLs, providing a robust security layer beyond network controls.
• Prioritize Comprehensive Monitoring and Alerting: Implement robust monitoring (e.g., Prometheus, Grafana) for all components of the TLS automation, including certificate expiration (especially for CAs), controller health, and policy adherence. Proactive alerts are crucial to prevent outages and security incidents.
• Secure Your CA Backend and Ensure Application Reloading: Integrate with hardened external CA solutions like HashiCorp Vault's PKI engine for secure certificate signing. Additionally, use tools like Reloader to ensure applications automatically pick up new certificates and trust bundles, preventing downtime from expired credentials.

About the Speaker(s)

Erik Godding Boye is a contractor and a dedicated maintainer of the cert-manager project. For many years, he has worked for Statnett, the Norwegian Transmission System Operator (TSO), which is responsible for the transmission of energy in the Norwegian power grid. Erik started his career as an application developer before transitioning to a platform team role, where he was instrumental in developing the advanced TLS solution discussed in this talk. His practical experience in securing critical infrastructure within an air-gapped, multi-tenant environment highlights his deep expertise in Kubernetes and certificate management.

Tim Ramlot is a software engineer at CyberArk and also a maintainer of the cert-manager project. As a key contributor to the cert-manager ecosystem, including projects like trust-manager and approval-policy, Tim brings extensive knowledge of X.509 certificate management on Kubernetes. His insights into the design and best practices of these tools are crucial for understanding scalable and secure TLS implementations.