Secrets, Secrets Are No Fun... So We Got Rid of Them!

Secrets, Secrets Are No Fun... So We Got Rid of Them!

Speakers: Savanah Frisk (Security Engineer, Application Security, Snap Inc.); Winston Howes (Tech Lead, Application Security, Snap Inc.)

Conference: LocoMocoSec

YouTube: https://www.youtube.com/watch?v=mxHcGxcOfrY

Overview

In this insightful LocoMocoSec talk, Savanah Frisk and Winston Howes from Snap Inc. shared their journey to drastically reduce secret proliferation within their organization. Titled "Secrets, Secrets Are No Fun... So We Got Rid of Them!", the presentation details Snap's strategic move to abstract away secret management from developers, focusing primarily on authentication secrets. This initiative not only solved a pervasive industry problem but also significantly enhanced Snap's overall security posture and streamlined developer workflows.

The speakers illuminated the fundamental challenges associated with traditional secret management, including widespread leaks, misconfigurations, and the inherent complexity of diverse authentication methods. They presented a comprehensive solution centered around an internally developed Authentication Translation Service (ATS), which acts as a central proxy for all internal-to-external and external-to-internal service communications requiring authentication. This architecture allowed Snap to enforce standardized authentication, achieve granular access control, and ultimately remove a large class of secrets that developers previously had to handle directly.

The talk underscores the critical importance of shifting secret management to the infrastructure layer, enabling security teams to harden a single, well-controlled point rather than relying on distributed, often inconsistent, developer implementations. By addressing the root causes of secret-related vulnerabilities and offering tangible benefits to engineering teams, Snap's approach provides a compelling blueprint for other organizations seeking to bolster their security programs and reduce operational overhead associated with secrets.

Background

▶ Watch: Identifying secret categories and core problems (0:45)

The pervasive challenge of managing secrets stems from two primary use cases: authentication and encryption. While encryption secrets (for data at rest or payload signing) are generally well-served by Key Management Systems (KMS), the vast majority of developer-facing secret management arises from authentication requirements. This includes both internal service-to-service communication and internal services interacting with external third-party APIs.

The speakers highlighted that secret leakage, often found in source code repositories, is an industry-wide problem that consistently undermines fundamental security principles. A core tenet of robust security architecture dictates that security should not depend on the secrecy of source code; rather, it should rely on a few well-managed secrets separated from the codebase. When secrets are mismanaged or exposed, this foundational premise is violated. Furthermore, authentication itself is inherently complex. Developers often struggle with correct implementations, leading to common vulnerabilities like misconfigured or missing authentication. The sheer diversity of authentication methods—from simple API keys and basic authentication to more complex protocols like OAuth and HMAC—compounds this difficulty. The lack of an industry-wide standard means that integrating with third-party services often necessitates adopting multiple, disparate authentication mechanisms, adding significant complexity.

Faced with these challenges, Snap Inc. established several key goals:

1. Reduce secret proliferation: Minimize the number of secrets scattered across the development environment.
2. Enforce a standard internal authentication mechanism: Provide developers with a single, consistent way to authenticate internal services, reducing cognitive load and implementation errors.
3. Reduce one-off external authentication implementations: Centralize the handling of diverse third-party authentication methods, abstracting this complexity from individual service owners.
4. Achieve finer-grain access control: Move beyond broad permissions often granted by default tokens (e.g., GitHub tokens granting full repository write access for merely posting a comment) to more precisely scope API interactions.

These objectives laid the groundwork for a transformative approach to secret management, aiming to solve the problem not by better educating developers, but by fundamentally removing the need for them to manage secrets at all.

Key Findings

▶ Watch: Introducing the Auth Translation Service (ATS) proxy (8:20)

Snap's core innovation lies in the development and deployment of the Authentication Translation Service (ATS), a centralized proxy designed to abstract secret management and authentication complexity from individual development teams. The ATS effectively serves as a broker for all authentication-related traffic, ensuring that developers no longer directly handle sensitive credentials.

A foundational element enabling the ATS is a robust internal service authentication mechanism. Snap leverages cloud-based identity management, specifically Google's identity provider, to issue short-lived JWT tokens. These tokens, attached in custom headers, allow internal services to authenticate with each other without directly managing any secrets. This approach pushes secret management to the infrastructure layer, ensuring uniformity and reducing the risk of replay attacks by minimizing token lifespan.

For external service interactions, the ATS sits as an intermediary. Instead of each internal service implementing various authentication types (API keys, OAuth, HMAC) and managing their corresponding secrets, they authenticate with ATS using the standardized internal mechanism. ATS then handles the translation to the appropriate external authentication type, fetching secrets from a KMS as needed. This centralization allows the security team to own and harden the secret management process.

A significant finding from their implementation was the distribution of external service access needs: approximately 90% of internal services' external interactions were deemed "non-sensitive." This allowed Snap to cluster these common, low-risk operations (e.g., posting messages to Slack, sending emails via Sendgrid) under single, scoped tokens and provide automatic approval for their use. For the remaining, more sensitive 10%, the ATS still facilitates individual credentials and allows for enhanced auditing and ownership by service teams.

Crucially, the ATS enables finer-grain access control that often surpasses the capabilities of the external service's own credentials. As an example, a GitHub API token might grant broad write access to a repository. However, through ATS's internal configuration, Snap can restrict an integration to only specific API paths, methods, or even body content. This means an internal service can be confined to only post comments to GitHub, even if the underlying token technically permits broader write actions, significantly reducing potential blast radius in case of compromise.

Finally, the talk highlighted key operational findings for building such a reliable proxy. These included addressing "Noisy Neighbor" problems through sophisticated rate limiting and queue management (e.g., Cloud Tasks), ensuring scalability via over-provisioned instances and flexible, parameter-based integration definitions, and designing for safe rollbacks by separating configuration from the main service codebase (e.g., storing config in a GCS bucket).

Technical Deep Dive

▶ Watch: Scaling with parameter-based, generalizable integrations (22:50)

The architecture of Snap's secretless environment is predicated on two main components: a standardized internal authentication protocol and the Authentication Translation Service (ATS).

Internal Service Authentication:

At Snap, internal services authenticate using a cloud-based identity provider, specifically Google. When an internal service (Service A) needs to communicate with another internal service (Service B), Service A first requests a JWT (JSON Web Token) from the cloud identity provider. This token, which is short-lived, is then attached in a custom HTTP header to the request sent to Service B. Service B, upon receiving the request, validates the JWT with the same cloud identity provider. This mechanism ensures that:

• Secrets are never directly managed by developers or exchanged between services.
• Authentication is brokered by trusted infrastructure.
• The use of short-lived tokens mitigates the risk of replay attacks.
• The protocol is uniform across all internal services, simplifying development and security auditing.

This infrastructure-managed approach forms the bedrock upon which ATS operates.

Authentication Translation Service (ATS) Architecture:

The ATS functions as both a forward and reverse proxy, translating authentication methods between internal and external services. It supports two primary traffic flows:

1. Direct Proxy Flow (Internal to External):

• An internal service initiates a request, first obtaining its JWT from the cloud identity provider and attaching it in a custom header.
• This request is sent to ATS, which validates the internal service's JWT with the cloud identity provider.
• ATS then consults its internal configuration and a database lookup to determine:
• The identity of the sending internal service.
• The target external service and the specific API endpoint it intends to reach.
• The required external authentication type (e.g., API key, OAuth token, HMAC).
• Authorization rules based on URL path, HTTP method, and potentially request body.
• ATS fetches the necessary external secret or token from a KMS (Key Management System).
• Finally, ATS forwards the request to the external service, adding the correct external authentication to the outbound request.

1. Asynchronous Flow (External Webhooks to Internal Services):

• An external service sends a webhook notification to ATS.
• ATS performs an authentication lookup, validating the external service's authentication (if required) and pulling any necessary secrets from the KMS.
• ATS then consults its internal configuration to identify which internal services are subscribed to this webhook.
• For each subscribed internal service, ATS obtains a cloud identity JWT (representing ATS itself or a specific identity for the internal service) and adds it as a custom header.
• ATS then "fans out" the request to all subscribed internal services, ensuring they receive the webhook with the standardized internal authentication. This fan-out often leverages a queueing system like Cloud Tasks for reliability and retries.

Token Refresh Mechanism:

For external authentication types requiring periodic token refreshes (e.g., OAuth access tokens), ATS incorporates a dedicated flow. It periodically uses refresh tokens (stored in KMS) to obtain new access tokens from external services, updating the KMS with the freshest credentials to ensure continuous operation.

Configuration and Scalability:

Instead of hardcoding integration logic, ATS defines integrations as a set of parameters stored in a database. Key parameters include:

• hostname: Used for routing incoming requests, often leveraging wildcard certificates and specific subdomains (e.g., slack.ats.snap.com).
• API origin: The destination URL for the external service.
• receiving auth: The authentication type expected on inbound requests to ATS.
• sending auth: The authentication type(s) to add to outbound requests from ATS. Importantly, ATS supports adding multiple authentication types on outbound requests if dictated by infrastructure (e.g., internal service auth for on-prem hosted external services plus the external service's own auth).
• rate limit: Allows for per-integration rate limiting.

This parameter-based approach, combined with reusable authentication plugins (e.g., for shared secrets, basic auth, HMAC), allows Snap to onboard new external service integrations rapidly without modifying the core ATS codebase. An API is provided for updating these integration parameters in the database.

To manage the configuration safely, especially during rollbacks, Snap recommends separating the configuration from the main service codebase. Initially, a JSON config file was used, but the improved approach involves writing the config to a storage bucket like GCS (Google Cloud Storage), which ATS periodically pulls to update its operational configuration. This decouples config updates from service deployments, preventing unintended rollbacks of new integrations.

Addressing "Noisy Neighbor" problems is also critical for a reliable proxy. For external rate limits, ATS implements per-internal-service rate limits that are lower than the external service's limit, and shards traffic across multiple external integrations if needed. For webhook fan-out, to prevent a single service from monopolizing queue resources (e.g., Cloud Tasks bandwidth caps), a round-robin approach is used: requests are initially stored in service-specific GCS buckets, and a cron job pulls one request from each bucket to feed into the queue, ensuring fair processing.

Demo / Proof of Concept

▶ Watch: ATS as a hardened, single point of focus for security (30:00)

The talk describes the Authentication Translation Service (ATS) as a fully operational and integral part of Snap Inc.'s infrastructure, rather than presenting a standalone demo or a proof-of-concept. The speakers detail the architecture, implementation, and operational considerations of this deployed service, showcasing how it effectively manages authentication and secrets for a large-scale enterprise. The examples provided, such as the GitHub comment scenario and integration with Slack, illustrate the practical application and benefits of ATS in a production environment.

Defensive Implications

▶ Watch: Q&A: Overcoming SDK challenges for credential management (40:00)

The Authentication Translation Service (ATS) fundamentally reshapes Snap's security posture by centralizing and standardizing secret management and authentication. This has several profound defensive implications:

1. Reduced Attack Surface and Secret Proliferation: By abstracting secrets from developers and centralizing them within ATS and a KMS, the vast majority of secrets are removed from developer workstations, source code, and individual service configurations. This drastically reduces the potential points of compromise and limits the blast radius if a single service is breached.
2. Standardization and Reduced Complexity: Winston Howes aptly stated, "complexity is the devil of security." By enforcing a single, standardized internal authentication mechanism and centralizing external authentication handling, ATS eliminates a large class of vulnerabilities arising from inconsistent, custom, or improperly implemented authentication logic across numerous services. Developers no longer make assumptions about authentication that can lead to security flaws.
3. Finer-Grained Access Control: ATS allows security teams to enforce authorization policies that are more granular than what the underlying external service's credentials might permit. Restricting API access by URL path, HTTP method, or even request body significantly minimizes the potential damage if an external integration token is compromised, transforming a broad "write access" token into a highly specific, limited-scope capability.
4. Centralized Audit and Visibility: ATS provides a single, authoritative point for auditing all authenticated connections between Snap's internal services and external third parties. This centralized visibility is invaluable for security investigations, compliance, and maintaining an accurate inventory of third-party integrations, which is often challenging at scale.
5. Security by Design and Hardening Focus: Because ATS is managed by the security team, it is built from the ground up with security best practices in mind. Being a "single point of focus" (despite being a single point of failure), it allows for concentrated hardening efforts, rigorous security reviews, and dedicated resources to ensure its resilience against attacks.
6. Controlled Logging of Sensitive Data: The speakers acknowledged that some weak forms of authentication, particularly those with secrets in URLs, can result in sensitive data appearing in ATS logs. To mitigate this, Snap implements strict access controls on ATS logs, limiting access to only project owners and preventing widespread exportation, thereby containing the exposure of any inadvertently logged secrets.
7. Enhanced Operational Resilience: The architectural considerations for scalability, such as addressing "Noisy Neighbor" problems with intelligent rate limiting and queue management, separating configuration from code for safe rollbacks, and over-provisioning instances, contribute directly to the service's reliability and its ability to withstand operational stresses and potential denial-of-service attempts.

Key Takeaways

• Abstract Secret Management: Shift the burden of secret management from individual developers to a centralized, infrastructure-managed service like ATS, reducing proliferation and enhancing security.
• Standardize Internal Authentication: Implement a uniform, infrastructure-brokered authentication mechanism (e.g., cloud identity provider with short-lived JWTs) for all internal service-to-service communication.
• Build a Flexible Authentication Proxy: Create a central proxy (ATS) that handles all external service authentication, utilizing a parameter-based approach for defining integrations and reusable authentication plugins to ensure rapid onboarding and scalability.
• Enforce Granular Authorization: Leverage the proxy to implement fine-grained access control based on API paths, HTTP methods, and request bodies, even if the underlying external credentials offer broader permissions.
• Prioritize Operational Resilience: Design the proxy for high availability and reliability by addressing challenges like "Noisy Neighbor" problems (rate limiting, queue isolation), safe configuration rollbacks (decoupling config from code), and elastic scalability.
• Foster Engineering Partnerships: Position the security solution as an enabler for engineering teams, offering clear benefits such as reduced development time, simplified integration processes, and readily available integration patterns, turning security into a win-win for the entire organization.

About the Speaker(s)

Savanah Frisk is a Security Engineer on the Application Security team at Snap Inc. Her work focuses on enhancing the security of Snap's applications and infrastructure, and she played a key role in the development and implementation of the Authentication Translation Service (ATS).

Winston Howes is a Tech Lead on the Application Security team at Snap Inc. He leads initiatives to improve the overall security posture of Snap's applications, drawing on his expertise in secure system design and threat modeling. Winston was instrumental in architecting the secretless approach detailed in this talk.

Both speakers are integral members of Snap's application security team, dedicated to building robust and scalable security solutions that enable developers while protecting user data and company assets.