Product Security Lessons from Incident Response

Product Security Lessons from Incident Response

Speakers: Astha Singhal, Director of Application Security, Netflix

Conference: LocoMocoSec

YouTube: https://www.youtube.com/watch?v=MOo-wvNUt_Q

Overview

In this insightful talk from LocoMocoSec, Astha Singhal, Director of Application Security at Netflix, challenges the traditional dichotomy of proactive versus reactive security functions. Often viewed as diametrically opposed, this mindset, embedded in many security frameworks, can inadvertently create organizational silos and inefficiencies. Singhal argues for a more hybrid incident response model where product security teams are deeply integrated into the incident response lifecycle, particularly for vulnerabilities within their domain. Drawing from Netflix's experience, she illustrates how this integrated approach not only streamlines incident resolution but also significantly enhances the overall maturity and effectiveness of the product security program, fostering stronger engineering partnerships and building critical skills within the team.

The presentation delves into the inherent structural issues of separating security functions strictly into "protect" and "respond" categories, as often interpreted from frameworks like the NIST Cybersecurity Framework (NCSF). Singhal explains how Netflix’s model empowers product security engineers to act as Incident Commanders (ICs) for issues originating from their ecosystem, such as bug bounty findings or vulnerability scanning alerts. This approach leverages their deep domain expertise, leading to faster, more context-aware resolutions and providing a continuous feedback loop that informs future proactive security investments.

The talk is crucial for security leaders and practitioners grappling with the challenges of scaling security programs, bridging the gap between security and engineering, and maximizing the impact of their security investments. By sharing Netflix's practical learnings, Singhal makes a compelling case for breaking down traditional barriers and embracing a more collaborative, skills-diverse approach to information security that ultimately makes security teams more resilient and empathetic partners to their engineering counterparts.

Background

▶ Watch: Traditional proactive vs. reactive security concepts (0:00)

The prevailing approach to organizing security functions often categorizes efforts into proactive and reactive strategies. This perspective is deeply ingrained in widely adopted frameworks like the NIST Cybersecurity Framework (NCSF), which outlines five core functions: Identify, Protect, Detect, Respond, and Recover. Within this framework, Protect is largely seen as the proactive pillar, focusing on implementing safeguards and preventing incidents, while Respond embodies the reactive element, dealing with events after they have occurred.

The Protect function, often synonymous with Product Security (ProdSec) or application security, involves activities like identity and access management, data security, security reviews, and threat modeling. The core goal of a ProdSec team is to ensure secure product development by providing foundational security services, conducting reviews on new products, and identifying/remediating vulnerabilities. Success in this role typically demands deep technical domain expertise (e.g., AppSec, system security), strong security software engineering skills, and the ability to manage stakeholder relationships for cross-functional collaboration.

Conversely, the Respond function, primarily handled by Incident Response (IR) or Security Incident Response Teams (SIRT/CSIRT), focuses on developing and implementing actions to address detected security incidents. This includes response planning, coordination, communication, mitigation, and timely resolution. Skills vital for an effective IR professional include broad security domain expertise, the ability to navigate ambiguity, flexible risk assessment, strong judgment, high integrity, and clear communication, especially when determining the criticality of an event.

While this functional separation seems logical given the distinct skill sets required, in practice, it often leads to significant challenges. The speaker highlights issues such as context silos between ProdSec and IR teams, where critical domain knowledge held by one team is not readily available to the other during an incident. This can result in increased response times due to handoffs between separate teams for triage, verification, and remediation. Imagine a P0 issue discovered in a widely used Java library within an engineering ecosystem. In a traditional model, if a proactive scanning tool found the issue, the ProdSec team might identify it, but the SIRT team would then be responsible for managing the incident. This often necessitates the SIRT team looping ProdSec back in as Subject Matter Experts (SMEs), leading to inefficient communication loops and potentially waking up the wrong teams in the middle of the night, prolonging resolution. This structural separation, while seemingly rational, can leave significant opportunities for efficiency and effectiveness on the table.

Key Findings

▶ Watch: Netflix's hybrid Product Security Incident Response (PSER) model (9:00)

Netflix's security model directly addresses the limitations of traditional proactive/reactive separation by embracing a hybrid incident response model. The core finding is that integrating Product Security (ProdSec) teams directly into the incident response function, particularly for product and engineering-related vulnerabilities, yields superior outcomes.

At Netflix, the central Security Incident Response Team (SIRT) maintains its primary role in managing incidents impacting users, systems, and infrastructure, as well as developing core Incident Management processes and tooling, such as their platform Dispatch. However, the Product Security team takes on a crucial function called Product Security Incident Response (PSER). This means ProdSec identifies, evaluates, and coordinates responses to security vulnerabilities specifically found in the products Netflix builds and its broader engineering ecosystem. This responsibility is handled as part of their team's on-call rotation.

In this model, ProdSec acts as a "little outpost of First Responders" for specific incident types. For example, if a bug bounty researcher discovers a P0 vulnerability, or an alert from vulnerability scanning tools or threat intelligence indicates in-the-wild exploitation of a product used by engineering, the ProdSec team would typically serve as the Incident Commander (IC). They leverage all the robust tooling and processes built by the central SIRT, such as Dispatch, but SIRT does not necessarily need to be involved in every incident. This allows SIRT to reserve its bandwidth for incidents that genuinely require its broader expertise, such as employee investigations or infrastructure-level zero-days (e.g., a VPN zero-day).

The key benefits of this distributed model are manifold:

1. Leveraging Domain Expertise: ProdSec engineers, with their deep understanding of the engineering ecosystem, product use cases, and business logic, can guide resolution more effectively, prioritize remediation, and accurately determine severity and impact.
2. Faster Resolution Times: By handling incidents end-to-end when they originate from sources closest to them (e.g., bug bounty, language dependency scanning), ProdSec eliminates the time overhead of handoffs between triage, verification, and remediation across separate teams.
3. Improved Tooling and Processes: ProdSec's direct experience as incident responders provides firsthand feedback on both SIRT-built Incident Management tools (like Dispatch) and their own proactive security tools (like their asset inventory product), leading to continuous improvement.
4. Enhanced Stakeholder Experience: For external parties like bug bounty researchers, they maintain a consistent point of contact. For internal stakeholders like Legal or Engineering, interactions with the Incident Management tooling remain consistent, regardless of which security team is leading the response.

This hybrid approach makes a compelling case for modern security teams to move beyond rigid functional separation, demonstrating how a more integrated strategy can significantly improve efficiency, collaboration, and the overall security posture.

Technical Deep Dive

▶ Watch: PSER benefits: deeper context, faster incident resolution (17:00)

The success of Netflix's hybrid incident response model hinges on a robust framework of shared processes, tooling, and clear role definitions, allowing Product Security (ProdSec) teams to effectively act as Incident Commanders (ICs).

At the heart of incident management at Netflix is the concept of a security incident, defined as "a potential, developing, or current security problem that needs some level of coordinated effort to investigate and resolve with some level of urgency." Incidents are assigned priorities based on potential impact to systems, business, or customers. For instance, a major vulnerability like Heartbleed would trigger an incident even for its potential impact, with the investigation for indicators of compromise being part of the incident process itself.

For every incident, an Incident Commander (IC) is assigned. The IC's role is critical: they maintain overall incident context, direct discussions, solicit ideas from Subject Matter Experts (SMEs), and ultimately choose the course of action. This requires core incident responder skills like judgment, integrity, communication, and situational awareness. The IC relies heavily on SMEs but is solely responsible for driving the incident to closure.

The distributed nature of the model means the IC role can be filled by different teams depending on the incident's origin and scope:

• SIRT as IC: For broader, infrastructure-level incidents (e.g., a zero-day vulnerability in a core VPN service), the central Security Incident Response Team (SIRT) would be the IC, handling the incident end-to-end.
• ProdSec as IC: For product-specific vulnerabilities (e.g., a P0 found via bug bounty, a newly discovered Apache Struts vulnerability, or a severe issue from vulnerability scanning tools or threat intelligence affecting a product in the engineering ecosystem), the ProdSec team member on call would step in as the IC. They leverage the same SIRT-built tooling and processes, ensuring consistency.

Several factors contribute to the success of this distributed IC model:

1. Common Processes and Training: All teams utilize shared incident response processes, runbooks, and participate in standardized Incident Commander training. This ensures that regardless of who is acting as IC, there's a consistent approach and understanding of responsibilities, significantly reducing the learning curve for ProdSec engineers new to incident response.
2. Shared Tooling: Netflix uses a unified Incident Management platform called Dispatch. This centralized tool manages all incident activities, from creating resources and assembling participants to sending notifications and scheduling Post-Incident Reviews (PIRs). Using Dispatch for all incident types drastically reduces cognitive load during a high-stress situation and ensures consistency across all incidents.
3. Clear Roles and Transparency: A clear understanding of roles, responsibilities, and transparent incident context facilitates seamless collaboration between various teams and stakeholders (e.g., Legal, Communications, Engineering).

One of the most significant advantages is the end-to-end ownership given to the IC. This empowerment allows them to make decisions efficiently, considering all inputs from SMEs, and drive the incident to resolution without unnecessary handoffs or delays. For instance, a bug bounty researcher interacts consistently with a ProdSec engineer, regardless of incident severity. Internal stakeholders experience uniform interactions with Dispatch, reducing confusion about which security team is leading.

ProdSec's deep engineering ecosystem context is invaluable during incident response. They can quickly determine what to prioritize when remediating an issue affecting multiple internet-facing services, understand the usage context of specific technology stacks, identify the correct engineering teams to engage for platform-level issues, and propose the most effective fix methods. Their understanding of business logic and application use cases allows for accurate severity and impact assessments. This contextual awareness directly translates to faster, more efficient resolution times, avoiding the need to "wake up additional folks in the middle of the night" for issues ProdSec can handle autonomously.

Furthermore, ProdSec's involvement provides crucial feedback for tooling. By acting as ICs, they offer firsthand insights into the usability of Dispatch and other Incident Management tools. This feedback loop is so powerful that Netflix realized the critical need for its asset inventory product directly because of ProdSec's role as PSER responders. They found themselves needing to consult multiple data sources during incidents, leading to the development of a centralized asset inventory that now serves broader proactive security use cases, but whose genesis was rooted in reactive incident response needs.

Demo / Proof of Concept

▶ Watch: Incident trends inform and strengthen proactive security strategy (24:00)

This talk focuses on a strategic and organizational model for integrating product security into incident response, rather than demonstrating a specific technical tool or exploit. While the speaker mentions Netflix's internal Incident Management platform, Dispatch, and their asset inventory product, these are described as integral components supporting their operational model, not as subjects of a live demonstration or proof of concept in the context of the conference talk. The presentation's value lies in its detailed explanation of processes, organizational structure, and the resulting benefits.

Defensive Implications

▶ Watch: Q&A: Quantifying incident impact with cost analysis (33:50)

Netflix's hybrid incident response model offers profound defensive implications for any organization seeking to enhance its security posture and operational efficiency. The integration of Product Security (ProdSec) into incident response creates a multifaceted defensive advantage:

1. Cultivating Crisis Management Skills in ProdSec: ProdSec engineers, traditionally focused on proactive measures, gain invaluable crisis management skills by acting as Incident Commanders (ICs). This includes developing good judgment under pressure, de-escalating situations, accurately gauging severity and impact, and communicating incident information with high integrity to stakeholders and leadership. These skills not only make them more effective responders but also elevate their overall capabilities as security engineers, fostering a more resilient and adaptable team environment built on trust and teamwork.

1. Establishing a Continuous Feedback Loop for Security Controls: Post-Incident Reviews (PIRs) become powerful mechanisms for continuous learning. By directly responding to incidents, ProdSec teams gain firsthand insight into the effectiveness of existing proactive security controls. PIRs help identify process and capability gaps, allowing teams to continually test and evolve their strategies. This feedback loop ensures that security investments are data-driven and directly address real-world vulnerabilities and attack vectors. For example, an incident might reveal that a workload isolation strategy didn't limit impact as expected, prompting adjustments to future designs or response strategies.

1. Building Empathy and Trust with Engineering Partners: When ProdSec engineers are "in the trenches" with their engineering counterparts during an incident, collaborating on investigations using shared observability and reliability tools, it fosters a deeper understanding of engineering pain points, processes, and tools. This direct collaboration dismantles the stereotype of security teams "throwing requirements over the fence" and instead establishes ProdSec as empathetic, invested partners in high-stakes situations, significantly strengthening relationships and trust.

1. Strengthening the Product Security Program with Data-Driven Insights: Incident trends (frequency, severity, and potential cost analysis) become a vital input for assessing the overall security posture and driving strategic investments. Netflix reports on these trends in quarterly updates to engineering leaders, providing a comprehensive picture of current security initiatives, risks, and future investment needs.

• Quantifying Risk and Investment: If multiple incidents consistently point to sensitive information ending up in logs, the long-term solution isn't just cleaning up spills but investing in log wrapper libraries or continuous data detection. Similarly, if access-related incidents for contractors highlight offboarding issues, the missing investment is in identity lifecycle management, not just authorization controls.
• Making a Stronger Case for Proactive Controls: By performing incident cost analysis (e.g., five incidents related to an authorization issue costing $40,000 each), ProdSec can attach a dollar value to the impact of prioritizing proactive controls. This data, combined with insights from attack tree exercises or red team assessments, provides a compelling business case for security investments, moving beyond theoretical risk to tangible financial impact.
• Driving Secure by Default Initiatives: Learnings from incidents can directly inform and prioritize secure by default initiatives. For instance, an incident revealing misconfigured internet-facing applications led to standardizing baseline security controls, ensuring default HTTP to HTTPS redirects, and proper logging/retention windows. ProdSec's close relationships with engineering teams allow them to effectively drive these long-term initiatives.

1. Influencing Product Roadmaps with Incident Response Needs: ProdSec engineers who partner with teams building internal or customer-facing products (e.g., a secure authentication proxy) can now represent incident response needs directly in product roadmaps. This ensures that new products are designed with both proactive and reactive security in mind, incorporating features like emergency blocklists or default request logging for common incident use cases.

1. Enhancing Self-Service Security Products: Incident context can be integrated into self-service security products for engineers. Beyond open vulnerabilities or outdated libraries, these platforms can now display critical incident-related information like "insufficient logging," missing on-call details, or long-term post-incident action items. This holistic view empowers engineering teams to proactively address security gaps identified through reactive events.

In essence, by doing incident response, ProdSec teams gain a direct, unfiltered view into the real-world impact of security weaknesses, enabling them to build a more effective, empathetic, and data-driven security program that continually assesses and improves its investments.

Key Takeaways

• Hybrid Model Enhances Product Security: Integrating Product Security (ProdSec) engineers into incident response (PSER) makes them better security professionals by providing firsthand experience with real-world vulnerabilities and their impact.
• Faster, Context-Aware Resolution: ProdSec acting as Incident Commanders for product-related issues leverages their deep domain expertise and eliminates handoffs, leading to significantly faster and more accurate incident resolution.
• Builds Crucial Crisis Management Skills: Participation in incident response develops essential skills in ProdSec engineers, including judgment under pressure, de-escalation, effective communication, and integrity, fostering a more resilient and capable team.
• Drives Data-Driven Security Investments: Incident trends, severity, and cost analysis provide invaluable data to engineering leaders, enabling them to make informed decisions about proactive security investments and prioritize initiatives like secure-by-default controls or identity lifecycle management.
• Fosters Empathy and Trust with Engineering: Working alongside engineering teams during high-stakes incidents builds stronger relationships, dismantles stereotypes, and establishes security as an empathetic and invested partner.
• Continuous Feedback Loop for Security Controls: Post-Incident Reviews (PIRs) and direct incident experience create a powerful feedback loop that continually assesses the effectiveness of existing security controls and informs the evolution of security strategies and tooling.

About the Speaker(s)

Astha Singhal is the Director of Application Security at Netflix. With a background that initially leaned towards a more traditional, separated view of proactive and reactive security functions, Singhal has experienced firsthand the transformative power of Netflix's hybrid model. Her expertise lies in leading and evolving product security programs, and through her talk, she shares practical insights and lessons learned from integrating incident response directly into the product security mandate at a large-scale, dynamic organization like Netflix. Her presentation reflects a deep understanding of organizational security challenges and a pragmatic approach to building more effective, collaborative, and resilient security teams.