Doing bad things for the right reasons: Vulnerability Disclosure at Amazon and AWS

Doing bad things for the right reasons: Vulnerability Disclosure at Amazon and AWS

Speakers: Alvin Batakato, Casmir Scholes, Justin Knight, Wesley, Jonathan Bowman, Ryan

Conference: Cloud Village

YouTube: https://www.youtube.com/watch?v=kV6Bxanefrw

Overview

This comprehensive talk, "Doing bad things for the right reasons: Vulnerability Disclosure at Amazon and AWS," delves into the intricate world of vulnerability disclosure from both the vendor and researcher perspectives. Presented by a diverse panel of Amazon and AWS security experts, including Alvin Batakato, Casmir Scholes, Justin Knight, Wesley, and top researcher Jonathan Bowman, the session offers a unique 360-degree view of the processes, challenges, and best practices involved in securing large-scale cloud and consumer services. The speakers illuminate the critical collaboration required between security researchers and organizations to identify, report, and remediate vulnerabilities, ultimately protecting millions of customers.

The discussion is structured to provide actionable insights for aspiring and seasoned security researchers, detailing how to craft impactful vulnerability reports that expedite triage and remediation. It also offers transparency into Amazon's and AWS's Vulnerability Disclosure Programs (VDP) and Bug Bounty Programs (VRP), outlining their scope, reward structures, and the benefits of engaging with their security teams. By sharing real-world examples, including a detailed technical deep dive into a specific AWS vulnerability, the talk underscores the importance of precise technical analysis, responsible disclosure, and the ongoing commitment from both sides to enhance the global security posture.

The talk is particularly significant because it bridges the gap between theoretical security research and practical, enterprise-level vulnerability management. It emphasizes that effective disclosure is a "choreographed" process involving multiple teams and that researchers are indispensable in the continuous battle against zero-days. The speakers collectively advocate for a culture of clear communication, mutual respect, and a shared goal of improving security, making it an essential listen for anyone involved in offensive or defensive security operations.

Background

▶ Watch: AWS Vulnerability Disclosure Program overview and researcher rules (0:00)

The landscape of cybersecurity is constantly evolving, with new vulnerabilities emerging daily across an ever-expanding digital attack surface. For large organizations like Amazon and AWS, which operate at immense scale, managing security requires a proactive and collaborative approach. This necessitates robust vulnerability disclosure programs (VDPs) and bug bounty programs (VRPs) that incentivize external security researchers to identify and report weaknesses. Historically, vulnerability disclosure has been a contentious area, often characterized by miscommunication, disagreements over severity, and uncoordinated public releases.

The need for formalized processes became evident as the complexity of software systems grew. Early disclosure practices were often ad-hoc, leading to "full disclosure" debates where vulnerabilities were sometimes publicly released without vendor patches available, potentially exposing users to risk. Over time, the industry has largely converged on coordinated vulnerability disclosure, where researchers work with vendors under embargo to ensure a patch is ready before public release. This talk builds on this foundation, showcasing Amazon and AWS's refined approach, which includes publishing a CVE ID, releasing official solutions, and issuing security bulletins simultaneously.

Challenges persist for researchers, including accurately assessing the root cause of vulnerabilities, correctly assigning Common Weakness Enumeration (CWE) categories, and precisely calculating Common Vulnerability Scoring System (CVSS) scores. Misinterpretations in these areas can lead to reports being undervalued, misprioritized, or even dismissed. For vendors, the sheer volume of reports, including "slop" or low-quality submissions, demands efficient triage and validation mechanisms. This talk provides context by addressing these common pain points, offering guidance from both sides of the disclosure equation to foster more effective and mutually beneficial collaborations.

Key Findings

▶ Watch: Casmir's unique perspective as a CNA and researcher (3:20)

The talk presents several key findings and insights, primarily centered on effective vulnerability reporting and the operational dynamics of large-scale VDPs and VRPs. From the researcher's perspective, Casmir Scholes outlined four cardinal rules for crafting high-quality reports that ensure proper handling and faster resolution:

1. Detailed Root Cause Explanation: Researchers should go beyond surface-level observations to identify and articulate the fundamental flaw causing the vulnerability. This prevents misinterpretations, avoids duplicate findings, and aids triagers in understanding the true impact.
2. Clean Environment Documentation: All reproduction steps must be documented from a clean, reproducible environment. This eliminates pre-existing conditions that could hinder a triager's ability to validate the report, ensuring consistency (e.g., specifying US East 1 region).
3. Accurate Severity Assessment: Researchers should avoid overinflating CVSS scores. Casmir noted that exaggerating severity (e.g., claiming a CVSS 10 when unwarranted) can lead to reports being taken less seriously or downgraded more aggressively than deserved. He emphasized that the vulnerability he discussed was "not a critical" and reporting it as such earned credibility.
4. Proposed Solutions/Remediations: Including potential solutions or remediation instructions can significantly accelerate resolution times and foster goodwill with program organizers, sometimes leading to additional rewards.

From the vendor's side, Amazon and AWS shared key metrics and program insights:

• Amazon VRP Scope: The program boasts an extensive scope, including Amazon Wildcard, Audible Wildcard, Ring, Twitch, and Amazon devices like Alexa, Echo, Eero, Kindle, and Fire OS.
• Triage and Bounty Timelines: The Amazon VRP aims for an average of 23 hours for initial triage and two weeks to bounty payout, demonstrating a commitment to valuing researchers' time.
• Reward Structure: Bounties range from $200 for low severity to $12,000-$25,000 for critical vulnerabilities, with additional bonuses for private program participants.
• Private Program Benefits: Trusted researchers gain priority triage, invitations to live security researcher events (LSREs), beta program access, and opportunities for direct collaboration with security engineers and partner teams.
• Handling AI-Generated Reports: Both AWS and Amazon VRP acknowledge the increase in AI-generated "slop" reports. While they have automated filtering for volume and reputation, every report, even if AI-generated, is still manually validated by a human to ensure no legitimate issue is missed, highlighting the human element's irreplaceable role.

Jonathan Bowman, a top Amazon researcher, shared personal "takeaways" for aspiring bug bounty hunters: getting a buddy for collaboration and impact assessment, going deep into specific rabbit holes within Amazon's vast scope, and maintaining a part-time, hobbyist approach to avoid burnout and enable riskier, more profound research.

Technical Deep Dive

▶ Watch: Four essential features for writing effective vulnerability reports (3:50)

The technical highlight of the talk was Casmir Scholes' detailed breakdown of a novel vulnerability he discovered in the AWS Titan watermark image generator. This was the first valid report for the AWS VDP program. The core of the vulnerability exploited the interaction between the watermark creation and validation systems, allowing an attacker to arbitrarily add or remove watermarks from images.

Casmir identified four key components of the watermarking system that, when chained, enabled the exploit:

1. Easy Watermarked Image Obtainment: The system allowed users to easily obtain a watermarked version of an image after any changes, providing a starting point for analysis.
2. Full Image Watermarking from Partial Modification: Modifying even a small portion of an image would result in the entire image being watermarked, not just the changed part. This was crucial for creating a usable watermark mask.
3. Watermark Detection on Similar Images: The watermark detection mechanism was robust enough to identify a watermark even if the applied watermark mask (derived from the difference between the watermarked and original image) was applied to an image similar to the original, rather than an exact one-to-one match. This allowed for some flexibility and artifacting in the process.
4. Small Portion Detection: Critically, a watermark would still be detected even if only a small part of the image, down to 16x16 pixels, contained the watermark. This meant an attacker didn't need to perfectly apply the watermark across the entire image for it to register.

The exploit leveraged these features by:

• Taking an original, unwatermarked image.
• Using a tool (like the remove object tool mentioned by Casmir) to make a small, controlled modification to the image (e.g., removing a "man in green").
• This process would watermark the entire modified image.
• By subtracting the original image from the watermarked image, a watermark mask could be extracted. This mask, due to the full-image watermarking feature, effectively contained the "signature" of the watermark.
• This extracted watermark mask could then be applied to any other image, effectively watermarking it. Conversely, subtracting the mask from an already watermarked AI-generated image would remove the watermark. Casmir specifically noted that artifacting occurred if there was a "great change" in the modified section, which is why he boxed out that component in his report.

Casmir also provided invaluable advice on reporting technical details:

• CWE Selection: He recommended starting with miter.org's weaknesses for simplified mapping, drilling down from broad categories (like "improper input validation") to find the most fitting CWE.
• CVSS Scoring Nuances (CVSS 3.1):
• Attack Complexity: This field refers to how easy it is for an attacker to reuse the vulnerability, not just the initial exploitation difficulty. A low attack complexity means it works "out of the box" every time, while high complexity requires specific, rare conditions.
• Scope: A scope change occurs only when an attacker gains a different level of control (e.g., from user-level process to kernel privileges). Remote Code Execution (RCE) alone does not constitute a scope change; this is why almost no CVEs truly deserve a CVSS 10 score. Misunderstanding scope is a common reason for overinflating CVSS.
• Code in Reports: He advised keeping proof-of-concept scripts as short as possible, embedding them directly in markdown for easy review, and explicitly explaining any "magic numbers" or coordinates so triagers can reproduce and test with different values.

This deep dive illustrates how a combination of seemingly minor design choices in a system can be exploited when understood thoroughly, emphasizing the importance of a root cause analysis in vulnerability research.

Demo / Proof of Concept

▶ Watch: Demonstrating the Titan watermark vulnerability's impact (5:50)

Casmir Scholes' presentation of the Titan watermark image generator vulnerability served as the primary demonstration of a real-world exploit and effective reporting. His slides included direct screenshots and snippets from his original report, adhering to his principle of using only "pieces of my original report."

The demonstration visually showcased two key aspects of the exploit:

1. Adding a Watermark to a Real Image: Casmir presented an image he obtained from the internet (a "real image"). He then showed a "watermarked" version of this image where, visually, there was "no real difference at all" from the original. This highlighted the stealthy nature of the added watermark. Crucially, he then displayed the "final output from the watermark," which was the watermark extracted from the image, explicitly demonstrating that the invisible watermark had indeed been successfully embedded. He also noted a "big missing black box" in the extracted watermark, hinting at the artifacting from the "remove object tool" used during the mask creation.
2. Removing a Watermark from an AI-Generated Image: For the inverse attack, Casmir used an AI-generated image of "a dog with a bee" (chosen for its non-sensitive nature for public sharing). He showed that after the watermark was removed using his technique, the image's confidence score in the watermark detection service "went from a high confidence score to not detect it at all." This clearly illustrated the arbitrary removal capability.

The talk elaborated on the methodology behind these demonstrations, detailing how Casmir utilized the remove object tool and specific prompting to create a modified image. This modified image, when processed by the Titan generator, resulted in a fully watermarked version. By deriving the watermark mask from this process, he could then apply or subtract it from other images. He emphasized that his PoC scripts were kept concise, often just "two functions and the main function," and were embedded directly into the markdown report. Furthermore, he meticulously explained the origin of any numerical values, such as bounding box coordinates, to ensure triagers could easily reproduce and adapt the exploit to their own test cases. This adherence to clear, reproducible steps and direct visual evidence made the demonstration highly effective and reinforced his best practices for reporting.

Defensive Implications

▶ Watch: Why detailed root cause analysis is crucial in reports (6:20)

The insights shared in this talk offer several critical defensive implications for organizations, particularly those operating large-scale services or managing their own VDPs/VRPs.

Firstly, Casmir Scholes' detailed remediation suggestion for the Titan watermark vulnerability provides a direct lesson: the system's flaw was that it watermarked the entire image even for small changes, and the watermark could be extracted. The proposed fix was to "only mask the portion that actually changed" and to prevent the extraction of the watermark entirely. This highlights that defenders must deeply understand the root cause of a vulnerability, not just apply a superficial patch. Security features, like watermarking, need thorough analysis to ensure their underlying mechanisms cannot be subverted through unexpected interactions. The fact that a 16x16 pixel portion could register a watermark also implies a need for stricter validation on watermark integrity and distribution.

More broadly, the talk emphasizes the importance of a well-structured and transparent vulnerability management program. Organizations should:

• Establish Clear VDP/VRP Guidelines: Amazon and AWS demonstrate the value of clear scope definitions, expected triage times (e.g., 23 hours for triage, 2 weeks to bounty), and explicit reward structures. This transparency builds trust with researchers and attracts high-quality submissions.
• Invest in Triage and Communication: Dedicated triage teams (Tier 1 and Tier 2 as described by AWS) are crucial for efficiently processing reports, determining severity, and escalating critical issues. The emphasis on human review, even for AI-generated reports, underscores that automated systems are assistants, not replacements, for human judgment.
• Provide Remediation Guidance: As Casmir suggested, researchers providing remediation instructions can significantly speed up the fix. This implies that organizations should be open to and capable of integrating external suggestions into their development lifecycle.
• Educate on CWE and CVSS: Organizations should internally educate their teams (and potentially external researchers through their VDP documentation) on the accurate application of CWE categories and CVSS 3.1 scoring, particularly the nuances of attack complexity and scope change. Misinterpretations can lead to misprioritization of risks.
• Leverage Tools like CNA Guru: Ryan mentioned the CNA Guru open-source application, developed at Bonecon, which assists in calculating CVSS scores, determining CWEs, and drafting CVEs. This tool is a valuable asset for organizations to streamline their internal vulnerability management processes and ensure compliance with CNA rules.
• Foster Collaboration: The Amazon VRP's private programs, live events, and direct engagement with researchers (e.g., pin test and device partners) demonstrate a proactive approach to collaboration. Building relationships with trusted researchers can lead to early detection of vulnerabilities in new products and services.
• Distinguish Remediation from Mitigation: As highlighted by the AWS team, defenders must understand the difference: mitigation is a temporary fix to "stop the bleeding," while remediation is the long-term, permanent solution. Both are necessary, but the ultimate goal is full remediation.

By adopting these practices, organizations can enhance their defensive posture, accelerate vulnerability resolution, and build stronger, more effective relationships with the security research community.

Key Takeaways

• Prioritize Root Cause Analysis: Detailed explanation of the underlying vulnerability is crucial for accurate triage, preventing duplicates, and speeding up remediation.
• Accurate Severity and CWE: Avoid overinflating CVSS scores; understand nuances like attack complexity and scope change (as per CVSS 3.1). Utilize resources like miter.org for precise CWE mapping.
• Structured Disclosure Programs: Amazon and AWS operate comprehensive VDPs and VRPs with clear guidelines, rapid triage (e.g., 23 hours), competitive bounties (up to $25K), and benefits for trusted researchers.
• Researcher Best Practices: Submit reports from a clean environment, provide clear reproduction steps, sanitize payloads, offer potential remediations, and consider collaborating with a "buddy" for impact assessment.
• The Power of Collaboration: Engaging with programs through private events and direct communication fosters goodwill, provides deeper scope access, and accelerates the protection of customers.
• Deep Research Pays Off: Focus on specific, complex areas within a large scope; even subtle flaws, like those in the Titan watermark generator, can be chained for significant impact.

About the Speaker(s)

The talk featured a panel of experts and a top researcher, offering diverse perspectives on vulnerability disclosure:

• Alvin Batakato: Representing AWS, Alvin discussed the public disclosure phase, emphasizing the choreographed process involving CVE ID publication, official solution releases, and simultaneous security bulletins. He highlighted the importance of researchers and collaboration within AWS's VDP.
• Casmir Scholes: A Director at Hidden Layer and an experienced security researcher, Casmir holds a unique position as both a CNA and a researcher. He shared his personal methodology for writing effective vulnerability reports, including his detailed technical deep dive into the AWS Titan watermark image generator vulnerability. He is reachable on LinkedIn and Twitter.
• Justin Knight: A Security Engineer on Amazon's Bug Bounty Program (VRP), Justin shared his journey from packing boxes in Amazon warehouses to hacking them as a security engineer. He emphasized personal discipline and provided insights into the Amazon VRP's scope, triage times, and reward structures.
• Wesley: Also a Security Engineer on Amazon's Bug Bounty Program (VRP), Wesley has five years of experience in bug bounties, starting at Yahoo. He detailed the benefits of Amazon's private program, including priority triage, live security researcher events (LSREs), and beta program access.
• Jonathan Bowman: Described as a "top researcher" and "the first security doctor" by the host, Jonathan shared his seven-year experience hacking Amazon. He revealed that 90% of his income comes from web bounties, primarily on Amazon. He advocates for a "buddy system" for research, "going deep" into targets, and treating bug bounties as a part-time hobby to avoid burnout. He noted he rarely uses AI for research and has a running joke about eating carrots.
• Ryan: While not explicitly listed in the speaker metadata, Ryan acted as a host/moderator and contributed to the discussion, notably mentioning the creation of CNA Guru, an open-source application to help with CVSS scoring and CWE determination. He also confirmed his presence upfront with Sarah.