Securing Your Company's Crown Jewels While the World Watches The Crown

Securing Your Company's Crown Jewels While the World Watches The Crown

Speakers: Scott Behrens (Senior Application Security Engineer, Netflix); Lakshmi Sudheer (Senior Security Partner, Netflix)

Conference: LocoMocoSec

YouTube: https://www.youtube.com/watch?v=w5fiBezbz4Q

Overview

In this insightful talk from LocoMocoSec, Scott Behrens and Lakshmi Sudheer, both from Netflix's security teams, present Netflix's comprehensive strategy for securing its vast and complex data ecosystem. Titled "Securing Your Company's Crown Jewels While the World Watches The Crown," the presentation delves into the challenges of protecting valuable data in a large, distributed environment and outlines a risk-driven, data-centric approach designed to unify security and privacy initiatives. Their core objective is to enable data consumers and producers to access, use, and move data across all platforms with appropriate protection, all while fostering Netflix's culture of "freedom and responsibility."

The speakers emphasize the critical importance of data security, citing The Economist's 2017 declaration of data as the world's most valuable resource. They highlight the prevalent issue of data misconfiguration as a leading cause of breaches, referencing Verizon's 2019 CX survey which indicated that approximately 29% of customers would abandon a brand after a breach. Against a backdrop of ever-changing regulatory landscapes and the inherent difficulty for data platform owners to prioritize security and privacy work, Netflix has engineered a strategic program that leverages quantitative risk analysis and integrated security controls.

This article explores Netflix's innovative solutions, focusing on their "secure by default" philosophy, the development of "paved road solutions," and the critical role of the Data Gateway in centralizing authentication and authorization. It provides a technical deep dive into their methods for gaining data insight through cataloging, detection, and lineage, and details how these insights inform a pragmatic, risk-aligned security posture that minimizes friction for developers while safeguarding Netflix's most valuable assets.

Background

▶ Watch: Introduction: Data as the most valuable resource (0:00)

The pervasive nature of data in modern enterprises makes its security a paramount concern. Scott Behrens underscored this by referencing The Economist's 2017 assertion that data, not oil, is now the world's most valuable resource. For a content-driven company like Netflix, data is the primary business driver, enabling the creation and delivery of beloved shows. However, this immense value comes with significant security challenges. Data misconfiguration, particularly in cloud storage services like S3, consistently ranks among the top causes of data breaches, a problem so widespread it has generated "best of" lists for compromises.

The repercussions of such breaches are severe. Behrens cited Verizon's 2019 CX survey, which found that roughly 29% of consumers would cease using a brand after a breach – a staggering potential loss of customer trust and revenue. Compounding this challenge are the constantly evolving regulatory and compliance requirements that demand agile and adaptable data security strategies. Internally, data platform producers and consumers face the complex task of prioritizing work across security, privacy, efficiency, and productivity, often leading to difficult trade-offs.

To address these multifaceted problems, Netflix recognized the need for a quantifiable and unified approach. They adopted a risk-driven, data-centric methodology that integrates both security and privacy initiatives. Central to this approach is the FAIR (Factor Analysis of Information Risk) framework, a quantitative risk analysis model that defines the probable frequency and probable magnitude of future loss. By understanding how often a risk scenario might occur and its potential cost, Netflix can make informed, strategic decisions about where to invest its security resources, moving beyond qualitative assessments to a more objective and impactful security program.

Key Findings

▶ Watch: Gaining data insight: Cataloging, Detection, Lineage (6:38)

Netflix's data security program is built on several foundational principles and innovative solutions designed to protect sensitive information without impeding the company's culture of freedom and responsibility. The core findings and contributions of their approach include:

• Risk-Driven, Data-Centric Approach: The program unifies security and privacy initiatives for data platform owners, producers, and consumers. By focusing on quantifiable risk, Netflix can prioritize investments that deliver the most significant impact on reducing potential loss, moving beyond traditional compliance-driven security.
• Quantitative Risk Analysis (FAIR): Leveraging the FAIR framework, Netflix identifies risk scenarios, forecasts the probable frequency and magnitude of loss events, and performs analyses (e.g., using Monte Carlo simulations) to inform strategic decision-making. This allows for a clear understanding of the financial implications of various security risks.
• Three Strategic Initiatives for Data Insight: To effectively manage data risk, Netflix invests heavily in understanding its data landscape through:

1. Cataloging: Comprehensive inventorying of data assets, ownership, and classification.
2. Detection: Real-time identification and tagging of sensitive data, acting as a crucial supplemental control.
3. Lineage: Tracking how data moves through the environment to ensure consistent security safeguards and access control propagation.

• "Secure by Default" and "Paved Road Solutions" Philosophy: Netflix champions building security directly into central infrastructure and developer ecosystems. This means creating paved road solutions – security tools and services that are easy to adopt, cause less friction, and are integrated into the existing development workflow, making the secure path the easiest path.
• Data Gateway: A Centralized Security Proxy: Recognizing the challenges of securing a polyglot environment with diverse data stores, Netflix developed the Data Gateway. This security proxy (a combination of forward and reverse proxies) centralizes authentication and authorization for all data stores, providing a consistent and robust security layer that leverages existing mutual TLS (mTLS) infrastructure and a central authorization service.
• Risk-Aligned Encryption Strategy: Instead of blanket encryption, Netflix adopts a risk-based approach to encryption. Data at rest encryption at the database level is reserved for truly sensitive data (e.g., SSN) where the risk reduction justifies the overhead, acknowledging that the probability of certain high-impact threats (like physical data center theft) can be low.
• Holistic Security Controls: Beyond application-level controls, the program emphasizes control plane hardening (e.g., AWS admin console, security groups) and leveraging an asset inventory as a single source of truth for applications, data stores, and their connections, enabling informed investment in hardening efforts and continuous risk assessment.

Technical Deep Dive

▶ Watch: Data Gateway: Paved road authentication solution (15:30)

Netflix's data security program is underpinned by sophisticated technical mechanisms, starting with its risk assessment framework and extending into its innovative control implementations.

Risk Assessment: The FAIR Framework in Practice

The foundation of Netflix's data security strategy is the FAIR framework, which provides a quantitative lens for evaluating risk. The process involves a risk assessment life cycle:

1. Identify a Risk Scenario: This involves defining four key components:

• Asset: Anything of value that can be impacted, leading to loss (e.g., core PII, credit card numbers).
• Threat: Anything capable of harming the asset (e.g., activists, malicious adversaries, rogue insiders).
• Effect: The impact on the CIA triad (Confidentiality, Integrity, Availability).
• Method: The technique used to harm the asset (e.g., server-side request forgery, unsafe deserialization, a misconfigured S3 bucket).

An example scenario provided is "external malicious actors impacting the confidentiality of core pii via S3 misconfiguration."

1. Forecasting and Collecting Data: Security teams and forecasters gather internal incident data, external information, and expert opinions to estimate the probable frequency and magnitude of the scenario.
2. Perform Analysis: Using the collected data, Monte Carlo simulations are run (e.g., 20,000 times) to model the potential outcomes of the incident, providing a range of possible financial losses and frequencies.
3. Report on Findings: The results are presented to stakeholders to guide strategic investment decisions, ensuring resources are allocated where they can most effectively reduce quantifiable risk.

Data Insight: Cataloging, Detection, and Lineage

To apply risk assessments effectively, Netflix needs deep insight into its data. This is achieved through three strategic initiatives:

• Cataloging: The primary goal is to know "what data sets we have." This involves identifying ownership, tagging data with appropriate categories (e.g., customer data, studio data, employee data), and mapping data stores (e.g., Elasticsearch, RDS, S3). The aim is to build a comprehensive and continuously improving data classification process.
• Detection: As a supplemental control to cataloging, detection ensures sensitive data is appropriately tagged, even if it's misidentified or spills across locations. For relational databases like RDS, Netflix captures change data (similar to read replicas) and streams it into Keystone, their real-time data processing framework. In Keystone, the data is run against a sensitive data detection library to identify PII or other sensitive information. This process aggregates findings into another Keystone topic, enabling real-time reporting for privacy and security teams, alerting data owners to misclassifications, and updating the central data catalog for improved accuracy.
• Lineage: Understanding data lineage means knowing how data moves through the environment, including ownership information and which systems access the data. This context is crucial for propagating access controls and maintaining a consistent security posture as data flows across diverse platforms and services.

Data Security Control Implementation

Netflix's security philosophy for data translates into a set of core control areas:

• Authentication: The Data Gateway (Security Proxy)

Securing authentication across a "polyglot" environment (TCP, GPC, HTTP, AWS managed, off-the-shelf data stores like MongoDB) presents a significant challenge. Traditional username/password authentication lacks strong identity, making incident response difficult. To overcome this, Netflix developed the Data Gateway, which functions as a security proxy.

This proxy sits in front of all data stores, managing connections. Data store owners simply configure their stores to connect to this central proxy. Developers, in turn, spin up a local forward proxy that connects to the Data Gateway's reverse proxy. Critically, the Data Gateway leverages Netflix's existing mutual TLS (mTLS) paved road solution. This means that when an application (e.g., "Revenue app") tries to connect to a database, it first authenticates with the Data Gateway using its mTLS certificate, providing strong, verifiable identity.

• Authorization: Data Gateway + Gandalf

Authorization is complex, requiring a balance between strict least privilege and business agility ("right-sized security"). The Data Gateway, having already validated the identity via mTLS, then queries Gandalf, Netflix's central authorization paved road infrastructure. Gandalf evaluates policies to determine if the authenticated user or application is permitted to access the requested data store. If authorized, the connection proceeds. This architecture ensures that even if database passwords are compromised, an attacker cannot gain access without the correct mTLS certificate and a corresponding authorization policy in Gandalf.

• Encryption

Netflix takes a risk-based approach to encryption. Instead of universal database-level encryption at rest, they reserve this measure for "really really sensitive data" like SSNs. Their risk assessments indicated that the probable frequency of threats like physical data center theft, which encryption at rest primarily mitigates, is relatively low. For other data, existing "paved road solutions" for encryption within application-level tables are utilized.

• Other Controls
• Control Plane Hardening: This includes securing management consoles (e.g., AWS Admin Console, Google Console), tightening security groups, and implementing account isolation to limit blast radius.
• Application Security Controls: Standard measures against OWASP Top 10 vulnerabilities are also emphasized.
• Asset Inventory: Netflix views its asset inventory as the "source of truth" for all applications, data stores, connections, and flow logs. This inventory feeds into risk assessments, enabling the calculation of risk scores per application and data store, guiding where to invest in hardening efforts.

Demo / Proof of Concept

▶ Watch: Data Gateway: Authorization and right-sized security (18:00)

Lakshmi Sudheer provided a compelling demonstration of the Data Gateway in action, showcasing its effectiveness in centralizing authentication and authorization.

The demo began by establishing a direct connection to a PostgreSQL client to an Amazon RDS instance using a traditional username and password. This connection was successful, simulating a typical, less-secure access method.

Next, Lakshmi illustrated the Data Gateway workflow from a developer's perspective. She executed a command to spin up a forward proxy on her local machine, configured to connect to the Data Gateway's reverse proxy, which fronts the target RDS instance. This forward proxy, leveraging Netflix's existing mTLS infrastructure, ensured her laptop's certificate was used for authentication. After verifying the forward proxy was running, she attempted to connect to the database via localhost (routed through her forward proxy to the Data Gateway). Initially, this connection was successful, demonstrating that her user identity was recognized and authorized by the system.

The critical phase of the demo involved demonstrating the authorization capabilities. Lakshmi updated the central authorization system, Gandalf, to remove her user's access policy for that specific database. Upon re-attempting the connection (both by refreshing the existing client and opening a new one), the connection was denied. This clearly illustrated that even with the correct username and password, and even having successfully authenticated via mTLS, access was blocked because the Gandalf policy explicitly disallowed it.

This demonstration effectively highlighted the Data Gateway's ability to:

• Centralize Authentication: By leveraging mTLS through a proxy.
• Centralize Authorization: By integrating with Gandalf.
• Enhance Security: By making compromised credentials useless without proper authorization policies and mTLS certificates.
• Simplify Developer Experience: By providing a "paved road solution" for secure data access.
• Enable Scale: Allowing a single team to configure new off-the-shelf databases to work with the security proxy, facilitating easier monitoring and fixes across the entire data ecosystem.

Defensive Implications

▶ Watch: Q&A: How to plan risk assessment coverage (32:30)

Netflix's approach offers several crucial defensive implications for organizations grappling with complex data security challenges:

• Embrace a Risk-Based, Quantifiable Security Strategy: Moving beyond compliance checklists, organizations should adopt frameworks like FAIR to quantitatively assess risk. This enables data-driven prioritization of security investments, ensuring resources are allocated to address the most impactful threats to crown jewel data.
• Leverage Proxy/Sidecar Architectures for Polyglot Environments: For diverse data platforms (databases, caches, logs, various protocols), implementing a security proxy like Netflix's Data Gateway is highly effective. This centralizes security controls (authentication, authorization) outside of the application or data store itself, making it easier to secure a wide array of technologies without burdening individual developer teams. This aligns with the concept of Envoy or other service mesh sidecars for consistent policy enforcement.
• Prioritize Strong Identity through Centralized Authentication: Implement mutual TLS (mTLS) as a "paved road solution" for all inter-service and user-to-data-store communication. Centralizing this via a proxy ensures strong, verifiable identity, which is foundational for granular authorization and effective incident response.
• Implement Centralized, Context-Aware Authorization: Develop or adopt a central authorization service (like Gandalf) that integrates with the authentication layer. This allows for fine-grained, policy-driven access control based on user/application identity, data classification, and operational context, enforcing the principle of least privilege.
• Invest in Data Insight (Cataloging, Detection, Lineage): You cannot protect what you don't understand. Organizations must build robust capabilities for data cataloging (knowing what data exists, its ownership, and classification), sensitive data detection (real-time identification and tagging), and data lineage (tracking data movement). These are foundational for accurate risk assessment and effective control placement.
• Adopt a Risk-Aligned Encryption Strategy: Rather than blanket encryption, conduct risk assessments to determine where data encryption at rest provides the most significant risk reduction. Focus database-level encryption on truly sensitive data, while leveraging existing application-level or platform-provided encryption for other datasets.
• Harden the Control Plane and Maintain a Robust Asset Inventory: Don't overlook the security of administrative interfaces and underlying infrastructure (e.g., AWS admin consoles, security groups, account isolation). Furthermore, maintain a comprehensive and up-to-date asset inventory as a single source of truth for all applications and data stores. This inventory can feed into continuous risk assessments and guide proactive hardening efforts.
• Foster Strategic Partnerships with Data Platform Teams: Security should not be an afterthought or an impediment. By embedding security engineers into data platform teams and building "paved road solutions," organizations can proactively integrate security into development workflows, ensuring security is seen as an enabler rather than a blocker.
• Plan for Continuous Improvement and Automation: Netflix's future plans include "ever-scanning" for control adoption, risk analysis at scale (via a program called SAGE), and right-sizing access based on usage patterns. Defenders should similarly plan for continuous monitoring, automated security scanning, and leveraging machine learning for large-scale risk assessment and policy recommendations. The Data Mesh project for propagating tags across data lifecycle is another advanced concept for maintaining security context.

Key Takeaways

• Adopt a Quantifiable, Risk-Based Approach: For complex, polyglot infrastructures, a risk-driven strategy using frameworks like FAIR is essential to prioritize security investments and ensure resources are allocated where they can most effectively reduce probable financial loss.
• Leverage Proxy-Based Solutions for Centralized Controls: Implementing a security proxy like Netflix's Data Gateway simplifies securing diverse data platforms by centralizing authentication and authorization, providing a consistent "paved road" for developers and reducing friction.
• Build Strong, Strategic Partnerships: Effective data security relies heavily on deep collaboration and integration with data platform teams, embedding security into their strategies rather than bolting it on as an afterthought.
• Invest in Foundational Data Insight: Comprehensive data cataloging, real-time sensitive data detection, and clear data lineage are critical for understanding the data landscape, assessing risks accurately, and applying appropriate security controls.
• Implement Robust, Centralized Identity and Access Management: Utilize strong identity mechanisms like mTLS for authentication and integrate with a central authorization service (e.g., Gandalf) to enforce granular, context-aware access policies and achieve least privilege.
• Strategically Apply Security Controls: Prioritize controls based on their impact on reducing the highest identified risks, such as focusing on control plane hardening and targeted encryption for highly sensitive data, rather than a one-size-fits-all approach.

About the Speaker(s)

Scott Behrens is a Senior Application Security Engineer at Netflix, working within the Product and Application Security Organization. His work focuses on integrating security into the product development lifecycle and ensuring the security of Netflix's applications. He is deeply involved in developing and implementing Netflix's risk-driven data security program, including the use of quantitative risk analysis frameworks.

Lakshmi Sudheer is a Senior Security Partner at Netflix, part of the Application Security team. She plays a crucial role in enabling data producers and consumers to securely manage data by building "paved road solutions" that streamline security processes. Her expertise lies in designing and implementing practical security controls, such as the Data Gateway, to address complex authentication and authorization challenges across Netflix's diverse data landscape.