When One Vulnerability Cascades Across Cloud Infrastructure

When One Vulnerability Cascades Across Cloud Infrastructure

Speakers: Albin Vattakattu (Vulnerability Disclosure Program Lead, AWS); Ryan Nolette

Conference: fwd:cloudsec

YouTube: https://www.youtube.com/watch?v=lyr6gsnewpk

Overview

This talk, "When One Vulnerability Cascades Across Cloud Infrastructure," by Albin Vattakattu and Ryan Nolette from AWS, provides an unparalleled behind-the-scenes look into how a major cloud provider handles zero-day vulnerabilities, particularly those stemming from third-party dependencies. It delves into the intricate, multi-team choreography required to identify, assess, remediate, and disclose such critical issues, not just within AWS's vast infrastructure but also across a global ecosystem of impacted organizations and their customers. The speakers highlight the escalating challenges posed by the exponential growth of vulnerability reports, driven in part by AI discovery tools, and propose a modern framework for Vulnerability Disclosure Programs (VDPs) designed for scale and efficiency.

Albin Vattakattu, AWS's Vulnerability Disclosure Program Lead, initiates the discussion by illustrating the difference between reacting to a crisis without communication and proactively informing stakeholders, drawing a parallel to the critical need for coordinated disclosure in cybersecurity. Ryan Nolette, a technical lead in the program, complements this by presenting concrete data and insights into the program's operational challenges and successes, particularly regarding automation and report quality. The talk is crucial for anyone involved in cloud security, vulnerability management, or incident response, offering a candid perspective on the complexities of securing hyperscale environments and the evolving role of VDPs.

The presentation underscores that a single vulnerability report can trigger a massive, synchronized response involving dozens of internal teams and external partners, culminating in a carefully orchestrated public disclosure. It moves beyond theoretical discussions to provide practical principles and data-driven observations for building resilient and scalable VDPs in an era of unprecedented threat velocity and volume. Ultimately, the talk serves as a call to action for both vendors and security researchers to collaborate more effectively, emphasizing transparency, context, and a focus on eliminating entire classes of vulnerabilities.

Background

▶ Watch: Introduction and zero-day analogy (0:00)

The landscape of cybersecurity is continuously evolving, with cloud infrastructure becoming the backbone of modern digital operations. This reliance on vast, interconnected systems inherently introduces complex security challenges, particularly concerning zero-day vulnerabilities. These are previously unknown flaws that can be exploited before developers have a chance to patch them, posing significant risks. The problem is exacerbated when these vulnerabilities reside in third-party dependencies, which are ubiquitous in modern software development. A single vulnerable component can ripple through an entire ecosystem, impacting numerous services and organizations that integrate it.

Traditionally, vulnerability disclosure has followed a more linear path: a researcher discovers a flaw, reports it to a vendor, the vendor fixes it, and then a public disclosure occurs. However, this model is proving increasingly unsustainable in the face of several contemporary pressures. Albin Vattakattu points out that in 2026 (the conference year), the number of vulnerability reports is growing exponentially, partly due to the proliferation of AI-powered discovery tools. While many of these are invalid, a significant portion represents legitimate, high-impact findings. Many organizations are struggling to keep pace, with some scaling back or even shutting down their disclosure programs.

This environment necessitates a fundamental shift in how cloud providers and other large organizations manage vulnerabilities. The sheer scale of cloud infrastructure, with potentially thousands of services and millions of customers, means that a "one-report, one-person" approach is no longer viable. The talk highlights the need for robust, automated, and strategically coordinated processes that can handle immense volume, provide rapid context, and address not just individual bugs but entire classes of vulnerabilities. The underlying problem is the "velocity of a whisper" – how quickly a single vulnerability, once reported, can cascade across an entire infrastructure and beyond, demanding an equally rapid and coordinated response to prevent widespread compromise.

Key Findings

▶ Watch: Vulnerability in a third-party dependency discovered (3:15)

The talk presents several critical findings and principles that define modern vulnerability disclosure and response in a hyperscale cloud environment:

1. The Cascading Effect of Third-Party Dependencies: A single vulnerability in a third-party dependency can impact a staggering number of services. In the specific case study presented, an initial report led to the discovery that the vulnerable dependency was used in 50 different AWS services, both customer-facing and internal. This highlights the immense blast radius and remediation challenge posed by shared components.
2. Exponential Growth in Vulnerability Reports: Ryan Nolette's data reveals a 200% increase in total report volume between 2024 and 2025 alone. This surge, partly attributed to AI-generated reports, strains traditional VDP models and necessitates significant automation.
3. Low Signal-to-Noise Ratio: Despite the massive increase in volume, only a fraction of reports are valid. Ryan states that only 31% of reports were valid security issues requiring a code change. Furthermore, 50-80% of reports were completely out of scope, including reports for other Cloud Service Providers (CSPs) or unrelated user account issues. This low signal-to-noise ratio consumes significant human resources.
4. Three Principles for the Modern VDP: Albin Vattakattu introduced three core principles implemented at AWS for building scalable and effective VDPs:

• Time to Context: The goal is to provide all necessary information for the next person in the remediation chain to act immediately, bridging the "delta" of information exchange. Systems should automatically provide vulnerability assessments, risk/impact analysis, historical trends, and even suggested code fixes before human intervention.
• Scale: VDPs cannot scale through people alone. Automation is key, with systems triaging a significant portion of reports, identifying and routing to necessary teams, and managing follow-ups. Administrative work performed by humans indicates a lack of scalability.
• Classes: Instead of playing "whack-a-mole" by fixing individual vulnerabilities, great programs aim to eliminate entire classes of vulnerabilities across the infrastructure. This involves identifying underlying patterns and systemic weaknesses.

1. The Importance of Coordinated Disclosure and Embargoes: For vulnerabilities with broad impact, AWS coordinates with the third-party vendor and other critical organizations under embargo. This allows impacted entities to implement temporary solutions while the vendor develops a permanent fix, minimizing public exposure to the vulnerability before a patch is widely available. The eventual public disclosure is a precisely choreographed event, releasing the CVE ID, vendor's permanent solution, and security bulletins simultaneously.

These findings collectively paint a picture of a cybersecurity landscape where traditional approaches are failing, and innovative, data-driven, and highly automated strategies are essential for protecting cloud infrastructure and its vast user base.

Technical Deep Dive

▶ Watch: Global impact and embargoed disclosure strategy (5:30)

The core of the talk's technical deep dive revolves around the intricate, multi-stage process AWS employs to handle a zero-day vulnerability report, especially when it implicates a third-party dependency. This process is a testament to the operational complexity of securing a hyperscale cloud environment.

The journey begins with the triage team receiving an initial, often vague, report. Without a working Proof of Concept (PoC) or mitigation, the first critical step is to engage in back-and-forth communication to obtain minimal actionable information. Once a PoC is secured, the incident response team takes over as the orchestrator. Their immediate tasks include performing an impact assessment and calculating the potential blast radius. This involves rapidly identifying affected internal services and potentially external customers.

Simultaneously, engineering teams responsible for the product are pulled into virtual war rooms. This is where the deep technical analysis begins:

1. Code Analysis: Engineers analyze the service's code to pinpoint the vulnerability.
2. Forensic Investigation: A critical step is to determine if the vulnerability has ever been exploited in the past.
3. Third-Party Dependency Identification: A crucial realization often occurs here: the vulnerability might not be in AWS's proprietary code but in a third-party dependency. This triggers a fundamental question: "Where else are we using this dependency?"

To answer this, the vulnerability assessment team performs comprehensive scans across all AWS services and their entire code repository. In the presented case, this scan revealed the dependency was used in 50 different services, both customer-facing and internal, each with its own roadmap and technical constraints. This discovery escalates the incident from a single service fix to a large-scale security campaign.

The campaign team is then engaged to drive these massive remediation efforts. The solution typically emerges in two forms:

• Dependency Removal: Departments that do not actively need the dependency remove it entirely.
• Temporary Solution Development: For departments that require the dependency, a temporary solution (a patch or workaround) is developed and rapidly implemented.

Following implementation, the application security team conducts rigorous testing to ensure the vulnerability is effectively remediated without introducing new flaws.

The next phase is outbound disclosure. Albin's team reaches out to the original third-party vendor, providing them with the PoC and the temporary mitigation strategy. If the third party requires an extended period for a permanent fix, AWS's responsibility extends beyond its borders. The question "Who else is using this dependency?" is asked again, leading to the identification of other critical organizations worldwide.

A measured approach is then adopted. Under embargo, AWS, in close coordination with the third party, reaches out to these identified critical organizations. They are provided with the temporary solution and PoC, allowing them to implement protections while the vendor develops a permanent fix. All parties agree to maintain strict confidentiality during this period.

The final stage is public disclosure. This is a precisely choreographed event where the embargo lifts, and three key actions occur simultaneously:

1. A CVE ID is published, providing a standardized identifier for the vulnerability.
2. The vendor's permanent solution is publicly released.
3. Security bulletins are issued by all impacted organizations.

Beyond this specific incident workflow, the talk also touches upon the technical implementation of Albin's "Three Principles for the Modern VDP." For Time to Context, the vision is for AI-powered systems to automatically provide comprehensive context (vulnerability assessment, risk, impact, historical trends, suggested code fixes) before a human even touches a report. For Scale, automation is key for triage, routing reports to the correct teams, and managing follow-ups, reducing human administrative burden. Albin emphasizes a preference for deterministic AI workflows and "spec-based vibe coding" over less predictable chat-based approaches for critical security functions, utilizing everything from sub-agents to large-scale orchestrators. This technical infrastructure is designed to handle the "thousands per day" volume of reports, weeding out the 50-80% out-of-scope submissions and streamlining the processing of valid ones.

Demo / Proof of Concept

▶ Watch: Choreographed public disclosure and CVE release (6:30)

While the talk does not feature a live demonstration or a specific Proof of Concept (PoC) of a vulnerability being exploited, the narrative heavily relies on the concept and utility of PoCs throughout the vulnerability disclosure and remediation process. The speakers describe how an initial vulnerability report, often lacking a PoC, requires active engagement to obtain one. This minimal PoC is then crucial for the internal AWS incident response team to validate the claim, perform impact assessments, and analyze code.

Furthermore, when AWS reaches out to the third-party vendor and other critical organizations during the outbound disclosure phase, they provide them with the PoC alongside their temporary mitigation strategy. This ensures that the external parties can independently verify the vulnerability and test the proposed solutions. The PoC, therefore, acts as a vital technical artifact that facilitates effective communication, validation, and remediation across multiple internal and external stakeholders, even if it wasn't demonstrated live during the conference talk itself.

Defensive Implications

▶ Watch: Principle 1: Time to context for efficient remediation (9:00)

The insights shared by Albin Vattakattu and Ryan Nolette offer critical defensive implications for organizations of all sizes, from small businesses to other hyperscale cloud providers. The escalating volume and complexity of vulnerabilities demand a proactive and systematic approach to security.

1. Adopt Modern VDP Principles: Organizations must move beyond traditional, manual vulnerability management. Implementing the "Time to Context," "Scale," and "Classes" principles is paramount. This means investing in automation to enrich reports with context (risk, impact, historical data, suggested fixes) before human intervention. It also requires building systems that can triage, route, and manage follow-ups for a high volume of reports, freeing security teams from administrative tasks. Most importantly, focus should shift from fixing individual bugs to identifying and eliminating entire classes of vulnerabilities to achieve systemic security improvements.
2. Understand Third-Party Dependency Risk: The cascading effect highlighted in the talk underscores the critical need for robust Software Bill of Materials (SBOM) management and continuous monitoring of third-party components. Organizations must have a clear inventory of all dependencies, understand their usage across their entire infrastructure, and be prepared for rapid remediation when a vulnerability in a shared component is discovered.
3. Prepare for Coordinated Disclosure: For critical vulnerabilities, especially those in widely used dependencies, organizations should be prepared to participate in coordinated vulnerability disclosure (CVD) processes under embargo. This requires establishing clear communication channels with vendors and other stakeholders, having incident response plans ready for rapid deployment of temporary mitigations, and synchronizing public announcements.
4. Optimize Researcher Engagement: For security researchers, the speakers offer clear guidance:

• Focus on Vendors/Cloud Providers: Direct research efforts towards vendors and cloud providers, rather than individual customers, as vendors are better equipped to handle and disseminate fixes broadly.
• Report Accidental Data Access Immediately: If customer data is accidentally accessed during research, stop immediately and report it.
• Coordinate Disclosure: Work closely with vendors for coordinated disclosure, respecting embargoes and ensuring that patches are available before publicizing vulnerabilities. Well-written, actionable reports with clear PoCs significantly aid vendor response time.

1. Leverage Automation for Triage and Data Analysis: Given the 200% increase in report volume and the high percentage of invalid or out-of-scope submissions (50-80%), organizations must implement intelligent filtering and triage automation. This not only reduces human workload but also allows security teams to focus on the truly valid and impactful 31% of reports. AI can play a role in this, but Albin advocates for deterministic AI workflows where outputs are predictable and reliable.
2. Embrace Transparency: As highlighted by Ryan Nolette, transparency builds trust. Vendors should publish clear VDP scope information (what's in scope for rewards, what's in scope for CVEs, safe harbor guidance) and consider allowing public disclosure of resolved reports (with PII sanitization) to foster greater trust and accountability with the research community.

By adopting these defensive strategies, organizations can better prepare for the inevitable discovery of vulnerabilities, enhance their ability to respond effectively at scale, and foster a more collaborative and secure cybersecurity ecosystem.

Key Takeaways

• Third-party dependencies pose a massive, cascading risk: A single vulnerability can impact dozens of services across an organization's infrastructure and affect numerous other companies globally.
• Vulnerability report volume is skyrocketing, driven by AI: VDPs are seeing exponential increases in reports (e.g., 200% year-over-year at AWS), with a significant portion being invalid or out-of-scope, necessitating advanced automation for triage.
• Modern VDPs must prioritize Time to Context, Scale, and Eliminating Classes: Moving beyond "whack-a-mole," effective programs automate context delivery, leverage systems over people for scalability, and focus on fixing systemic weaknesses rather than individual bugs.
• Coordinated disclosure with embargoes is critical for widespread vulnerabilities: For high-impact flaws, a precisely choreographed disclosure process involving the vendor, affected organizations, CVE publication, and simultaneous security bulletins is essential to protect customers.
• Security researchers are vital partners and should focus on vendors: Researchers are urged to report to vendors, stop immediately if customer data is accessed, and coordinate disclosure to ensure maximum impact and protection.
• Transparency and well-written reports improve response times: Vendors should provide clear VDP guidelines and scope, while researchers should submit actionable, well-structured reports to facilitate rapid and effective remediation.

About the Speaker(s)

Albin Vattakattu is the Vulnerability Disclosure Program Lead for AWS. In this role, he is at the forefront of designing and implementing strategies for managing and responding to zero-day vulnerabilities across AWS's vast cloud infrastructure. He is a recognized thought leader in the industry, having developed the three core principles for modern VDPs (Time to Context, Scale, and Classes) that are implemented at AWS and have gained external recognition, including a blog feature by HackerOne. Albin co-authored the inaugural AI security white paper jointly published by AWS and the SANS Institute, highlighting his expertise in advanced security methodologies, including the strategic application of AI in security workflows.

Ryan Nolette serves as a technical lead within AWS's Vulnerability Disclosure Program. With a background in incident response, Ryan brings a practical, data-driven perspective to the challenges of vulnerability management. He is instrumental in building the automation and tooling that enables AWS to handle the massive and exponentially growing volume of vulnerability reports. Ryan's work focuses on data analysis, improving report quality, and enhancing the efficiency of triage and categorization processes. He is deeply involved in ensuring consistency in report handling and is a strong advocate for transparency and accountability in the VDP space.