How We Moved Spotify To a Proxyless gRPC Service Mesh - Erik Lindblad & Erica Manno, Spotify

How We Moved Spotify To a Proxyless gRPC Service Mesh

Speakers: Erik Lindblad, Staff Engineer, Spotify; Erica Manno, Senior Engineer, Spotify

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=2_ECK6v_yXc

Overview

In this KubeCon EU talk, Erik Lindblad and Erica Manno from Spotify delve into the monumental task of evolving Spotify's service discovery infrastructure. They recount how the company outgrew its decade-old DNS-based system, affectionately dubbed "Nameless," and embarked on a journey to replace it with a sophisticated, in-house built, proxyless gRPC service mesh powered by XDS (X Discovery Service). The presentation meticulously outlines the challenges faced with traditional DNS, the rigorous evaluation process for selecting a new architectural direction, and the intricate technical details of designing, implementing, and rolling out a bespoke XDS control plane across Spotify's vast microservice ecosystem.

This talk is particularly significant for organizations grappling with the limitations of conventional service discovery in large-scale, heterogeneous microservice environments. Spotify's unique approach—opting for a proxyless service mesh and implementing a custom XDS control plane—offers valuable insights into maintaining control, optimizing performance, and enabling advanced traffic management features without the overhead of sidecar proxies. The speakers provide a candid account of the engineering trade-offs, the meticulous rollout strategy, and the lessons learned, making it a compelling case study for infrastructure engineers and architects aiming to build highly scalable, resilient, and flexible service networks.

Background

▶ Watch: Introduction and talk agenda (0:00)

Spotify operates a massive microservice architecture running on GKE (Google Kubernetes Engine), having migrated from an on-premise custom orchestration layer to GCP around 2017. The majority of their services, approximately 75%, communicate via gRPC, utilizing an in-house Java framework named Apollo. A smaller, remaining portion uses a proprietary protocol that is currently being migrated off. Crucially, Spotify employs proxyless gRPC, where gRPC clients connect directly to servers, diverging from traditional service meshes that often rely on forwarding proxies and sidecars. The infrastructure spans five GCP regions, designed with a regional failure domain, allowing regions to operate independently while supporting cross-region calls when necessary.

The existing service discovery system, "Nameless," was initially developed in 2013 to replace a collection of bash scripts managing DNS zone files. Over the years, Nameless evolved to support heartbeat-driven service registration for VM-based workloads and subscribed to EndpointSlice changes in Kubernetes clusters through a component called "Shameless." Service discovery data was primarily exposed as DNS SRV records via CoreDNS, queried by services through Cloud DNS and custom resolvers. This regionalized architecture meant Nameless had no inter-regional dependencies.

Despite its robustness and simplicity, the DNS-based system began to hit significant limitations as Spotify scaled. Key challenges included:

• DNS response size limits: For the largest deployments, DNS responses frequently exceeded the 65-kilobyte maximum, leading to operational issues.
• Uncertainty with caching: The multi-layered DNS caching made it difficult to reason about the time it took for a service to unregister, impacting reliability and incident response.
• Inflexibility and one-way communication: DNS's inherent limitations hindered the development of advanced routing features, which Spotify's evolving needs increasingly demanded.

Recognizing these constraints, Spotify initiated a structured three-step evaluation process to find a successor. The team first defined a set of core requirements focusing on reliability, scalability, and extensibility, including specific needs like proxyless gRPC support, zone-aware load balancing, and compatibility with their heterogeneous service architecture. They then evaluated several candidates:

1. Traffic Director: Google's managed control plane. Spotify had prior experience but found it incompatible with their network needs due to its managed, locked-down nature and limited extensibility. A variant involving a custom proxy layer was also considered but dismissed.
2. Istio: Evaluated both in self-hosted and managed modes (e.g., Solo, Anthos). Concerns arose regarding Istio's scalability for Spotify's immense scale and the lack of critical features for their proxyless gRPC mode.
3. Extending the existing system: This option involved building upon the proven foundation of Nameless, tailored to Spotify's specific network requirements.

After a thorough assessment, extending their existing system emerged as the most attractive option. While seemingly an "engineer's hunch," the rigorous evaluation process was crucial for validating this conclusion, gaining management buy-in, and clearly defining the problems they needed to solve. The speakers also noted that this evaluation was conducted over a year prior to the talk, acknowledging that the landscape of service mesh products is rapidly evolving.

Key Findings

▶ Watch: Spotify's legacy DNS service discovery: Nameless (2:20)

The core finding of Spotify's journey was the successful transition to an XDS-based service mesh by extending their existing "Nameless" system. This strategic decision allowed them to overcome the limitations of DNS and build a highly customized and scalable service discovery solution tailored to their proxyless gRPC architecture.

A fundamental deliverable was the introduction of XDS as the new protocol for managing the data plane. XDS is a universal data plane API, originally spun off from the Envoy proxy project, and is natively supported in both gRPC and Envoy. Spotify had prior experience using XDS for their Envoy-based perimeter, giving them confidence in its capabilities. The decision to adopt XDS was driven by its robust feature set and the flexibility it offered to build out prioritized functionalities without significant hassle.

The adoption of XDS immediately unlocked several critical capabilities and features:

1. Traffic Splitting: XDS enabled the implementation of an imperative API for diverting traffic based on custom rules. A pilot use case involved separating synthetic ML training traffic from real production traffic at multiple points in the call chain. This was solved at the network layer, preventing code duplication and cumbersome application-level changes across numerous services.
2. Zone-Aware Routing: Spotify's deployment across multiple availability zones within each GCP region was not fully optimized, as their service network was an "artifact from the on-prem days." Sending traffic across zonal boundaries incurs costs. The new XDS-based system allowed for the development of a zone-aware load balancer. By leveraging XDS's persistent connections, the control plane could understand client distribution across zones and combine this with server locality data from Kubernetes EndpointSlices and cluster load reports (another XDS feature). This enabled routing traffic as much as possible within the same zone, using a modified version of Envoy's locality-aware load balancing algorithm.
3. Dependency Graph: With the new architecture providing persistent connections to every client and having a complete view of all servers, Spotify gained the ability to easily generate a global, real-time dependency graph. This dataset can show which services are connected to what, and with the addition of load report data, how traffic flows between them in real-time, significantly enhancing observability and operational insights.

A crucial decision in this process was the choice to implement XDS from scratch rather than relying on existing open-source libraries like Envoy's Java control plane library. Initial attempts with such libraries revealed they lacked the granular control necessary for a proxyless gRPC service mesh, where each node can have multiple gRPC channels to the control plane, each subscribing to different configuration resources. Implementing their own XDS control plane allowed Spotify to:

• Optimize for proxyless gRPC: They specifically implemented the Aggregated Discovery Service (ADS) variant of the XDS protocol, tailored for their client needs.
• Gain full control: This enabled future extensions, precise instrumentation for observability, and the ability to start simple and address bottlenecks only when they emerged.
• Manage complexity: Despite initial fears of creating overly complicated code, the XDS implementation turned out to be relatively concise and maintainable.

Technical Deep Dive

▶ Watch: Why DNS-based service discovery failed (3:45)

The technical implementation of Spotify's proxyless gRPC service mesh involved a significant revamp of their existing service discovery system, "Nameless," and the integration of a custom XDS control plane.

Revamping Nameless into Shameless

The first step was to modernize the underlying infrastructure of Nameless. This involved:

• Migration to Kubernetes: Nameless components, previously running on VMs, were containerized and deployed as Kubernetes pods.
• Centralized Data Store: The original Nameless system scattered service discovery data across multiple locations: a Spanner database for VM workloads, in-memory stores in "Shameless" for Kubernetes workloads, and Cloud Data Store for regional redirection configurations. The revamped design consolidated this.
• VM Workloads: Workloads on VMs still send periodic heartbeats to a component called Nameless Registry. However, instead of writing directly to Spanner, Nameless Registry now calls out to Shameless. Shameless, leveraging the Kubernetes API server, writes Custom Resource Definitions (CRDs) into etcd to store this service discovery data.
• Kubernetes Workloads: Shameless connects to all Kubernetes clusters in a region, creating watches for EndpointSlice resources. It stores this data, along with the CRD data for VM workloads and regional redirect configurations, directly in its in-memory store.
• Simplified DNS Resolution: CoreDNS, still in use, now forwards DNS lookups directly to Shameless. Since Shameless holds the complete, up-to-date service discovery data in memory, it can answer DNS queries much faster, eliminating the need for the "Nameless Discovery" component and the external Spanner/Cloud Data Store lookups. This significantly reduced DNS query latency.

This consolidated architecture positions Shameless as the single source of truth for service discovery data within a region, making it the logical and most efficient place to implement the XDS interface.

Custom XDS Control Plane Implementation

The XDS protocol defines a set of APIs that a data plane (e.g., a gRPC client) uses to receive configuration from a control plane (e.g., Shameless). Resources of different types (e.g., Listener, Route, Cluster, Endpoint) are fetched by the client. While individual APIs can be implemented, Spotify opted for the Aggregated Discovery Service (ADS) variant, which allows the client to fetch configurations of all types over a single persistent gRPC stream.

Initially, Spotify explored using Envoy's open-source Java control plane library. However, this library proved insufficient for their specific needs. In a proxyless gRPC service mesh, each client node can establish multiple gRPC channels to the control plane, one for each service it needs to communicate with. Each channel, in turn, subscribes to a series of configuration resources. The existing library lacked the necessary granularity for the control plane to remember every individual gRPC channel, its subscriptions, and to push targeted updates when a resource changed.

This limitation led to the critical decision to implement XDS from scratch. This strategic choice provided several advantages:

• Optimization for Proxyless gRPC: The implementation was specifically tailored for proxyless gRPC clients, focusing on the ADS variant which was most suitable.
• Full Control: Spotify gained complete control over the XDS implementation, enabling future extensions to support other client types or protocols, and precise instrumentation to meet their observability requirements.
• Iterative Development: They could start with a simpler implementation and address performance bottlenecks or scaling challenges only as they emerged, avoiding premature optimization.

Challenges in implementing XDS from scratch included the lack of a formal protocol specification (relying on extensive Envoy documentation and examining other implementations like Istio's codebase) and initial concerns about the complexity of the resulting code, which ultimately proved manageable.

Gradual Rollout and Client Onboarding

Rolling out XDS to hundreds of thousands of services required a meticulously planned, gradual approach to minimize disruption. Key non-negotiable requirements for the rollout included:

• No "Big Bang" Deployment: A phased migration was essential due to the scale and criticality of the change.
• Configuration-Driven Control: The ability to control which services, and what percentage of traffic within those services, switched to XDS through configuration, based on metadata (e.g., region, reliability tier).
• Zero Service Owner Intervention: Service owners should not need to modify their code or configurations.
• Minimal Disruption: The rollout aimed for zero or minimal impact on production traffic.
• "Panic Button" Fallback: A rapid and reliable mechanism to revert to the known good DNS state was paramount.

The rollout was implemented within Apollo, Spotify's in-house managed Java framework, which provides gRPC-specific modules.

1. Custom Managed Channel: A custom managed channel was created within Apollo's gRPC client module. This channel maintains two open gRPC connections: one for DNS-based service discovery and one for XDS.
2. Global Flag for Switching: A background task periodically reads the value of a global flag, stored in a DNS TXT record, to determine whether XDS is enabled. This flag dictates which gRPC channel (DNS or XDS) is actively used. This mechanism also serves as the "panic button" for rapid fallback to DNS.
3. XDS Eligibility Checker: When a gRPC service boots up, an XDS eligibility checker evaluates the service's metadata against the current XDS rollout configuration (e.g., 10% of services in a specific region with a certain reliability tier). If deemed eligible, the service uses the new managed channel.
4. XDS Bootstrapping Logic: For eligible services, XDS bootstrapping logic dynamically generates the necessary configuration for the XDS client (within gRPC), instructing it to connect to Shameless and send identifying metadata. This configuration is currently injected as a system property at runtime, though a long-term plan involves using an init container for earlier, more robust initialization.

The rollout proceeded in phases:

• Initial Phase: Targeted less critical services, starting with a small percentage in one region, gradually increasing to 100% in that region, and then expanding to all other regions in increments. This phase, involving hundreds of distinct services, took approximately 1.5 months.
• Subsequent Phases: New, progressively more critical groups of services were onboarded, moving faster with larger global percentages. This iterative "rinse and repeat" approach continues until all eligible services are migrated.

Demo / Proof of Concept

▶ Watch: Evaluating Traffic Director, Istio, and custom solutions (5:58)

The talk primarily focused on the architectural design, detailed implementation, and gradual rollout process of Spotify's proxyless gRPC service mesh. While the speakers mentioned building a "pock" (Proof of Concept) early on to evaluate the open-source Java control plane library, this was part of their internal development process and not a live demonstration featured during the KubeCon presentation. The narrative emphasized the strategic decisions, engineering challenges, and operational considerations rather than showcasing a specific live demo or hands-on proof of concept of the final system.

Defensive Implications

▶ Watch: Assessment summary: why Spotify built custom XDS (6:40)

Spotify's migration to a proxyless gRPC service mesh with an in-house XDS control plane introduces several significant defensive implications, enhancing the security and resilience posture of their vast microservice ecosystem.

1. Enhanced Resilience and Fault Isolation: The implementation of zone-aware routing is a critical defensive improvement. By intelligently routing traffic within the same availability zone, the system significantly reduces the blast radius of zonal outages. If one zone experiences issues, traffic can be gracefully shifted or contained, preventing a regional failure. This granular control over traffic flow improves overall system resilience and reduces the potential for cascading failures.
2. Improved Observability for Incident Response: The ability to generate a real-time dependency graph showing service connections and traffic flow is invaluable for security operations. During an incident, this global view allows defenders to quickly identify affected services, understand the spread of an attack, and pinpoint potential weak points or unexpected communication paths. It aids in anomaly detection, enabling faster response times to unusual traffic patterns that might indicate compromise.
3. Advanced Traffic Management for Security: Features like traffic splitting offer powerful defensive capabilities. Security teams can use this to:

• Safely Test Security Patches: Route a small percentage of production traffic to services running new security patches or configurations before a full rollout.
• Isolate Suspicious Traffic: Divert traffic from potentially compromised clients or sources to honeypots or isolated analysis environments.
• Implement Fine-Grained Access Control: While not explicitly detailed in the talk, the XDS framework's capability to define routing rules based on various attributes creates a foundation for highly granular, network-layer access control policies, enforcing least privilege principles.

1. Rapid Fallback for Disaster Recovery: The DNS TXT record-based "panic button" is a crucial defensive mechanism. In the event of a critical issue with the XDS-based system, the ability to revert all services to the stable, DNS-based discovery within minutes ensures business continuity and minimizes downtime. This provides peace of mind for engineers and a vital safety net during high-pressure incidents.
2. Centralized Control and Policy Enforcement: By consolidating service discovery logic within Shameless, the XDS control plane becomes a central point for managing and enforcing network policies. This simplifies the application of security configurations, ensures consistency across services, and reduces the risk of misconfigurations in individual service deployments.
3. Addressing Pre-existing Issues: The rollout process itself, by sometimes "surfacing pre-existing issues" like services not having zero-downtime deployments, acts as a defensive audit. It forces teams to improve fundamental operational hygiene, leading to a more robust and secure overall architecture. This educational aspect is a long-term defensive gain.
4. Challenges in Troubleshooting Complexity: While offering many benefits, the increased complexity of a custom XDS implementation with many moving parts can present defensive challenges. Pinpointing issues in specific services amidst hundreds of others requires sophisticated monitoring, logging, and tracing capabilities to prevent obscure bugs from becoming security vulnerabilities or delaying incident resolution. Defenders must invest heavily in these areas to maintain visibility.

In essence, Spotify's move to XDS significantly strengthens their defensive posture by providing granular control, enhanced visibility, and robust resilience mechanisms, albeit with an increased need for sophisticated operational tooling and expertise.

Key Takeaways

• Spotify successfully transitioned its massive microservice architecture from a decade-old DNS-based service discovery system to a proxyless gRPC service mesh powered by an in-house XDS control plane.
• The decision to build a custom XDS implementation stemmed from a thorough evaluation that found existing market solutions like Traffic Director and Istio did not fully meet Spotify's specific scalability, extensibility, and proxyless gRPC requirements.
• The new XDS-based architecture, built on a revamped "Shameless" service, enabled critical advanced features such as fine-grained traffic splitting, cost-saving zone-aware routing, and real-time dependency graphs for enhanced observability.
• A meticulously planned, multi-phase rollout strategy, combined with a DNS TXT record-based "panic button" for rapid fallback, was crucial for onboarding hundreds of services with minimal disruption and building confidence in the new system.
• Implementing XDS from scratch provided full control and optimization for Spotify's unique environment but required navigating the lack of a formal protocol specification and addressing runtime configuration challenges, such as the flakiness of XDS bootstrapping logic.
• The project underscored the importance of robust observability, a pragmatic approach to managing complexity in large-scale infrastructure, and the necessity of educating service owners on improving operational hygiene like ensuring zero-downtime deployments.

About the Speaker(s)

Erik Lindblad is a Staff Engineer in Spotify's Core Infrastructure product area. His work focuses on building and maintaining the foundational systems that power Spotify's global services, playing a key role in evolving their massive microservice architecture.

Erica Manno is a Senior Engineer and Subject Matter Expert within Spotify's Service Networking team. Her expertise lies in the intricate details of service communication and discovery, making her instrumental in the design and implementation of Spotify's next-generation service mesh.