Yield the Mallet: Getting Out of Security Whack-a-Mole

Yield the Mallet: Getting Out of Security Whack-a-Mole

Speakers: Jorge Lopez, Principal Security PM Manager, Microsoft

Conference: LocoMocoSec

YouTube: https://www.youtube.com/watch?v=FNO_gF8C7Ls

Overview

In his LocoMocoSec talk, "Yield the Mallet: Getting Out of Security Whack-a-Mole," Jorge Lopez, then Principal Security PM Manager at Microsoft and former lead of the Vulnerability Response Team, addresses a pervasive challenge in the cybersecurity industry: the seemingly endless cycle of patching individual vulnerabilities without tackling their underlying causes. Lopez posits that this "whack-a-mole" approach, while sometimes satisfying, ultimately leads to exhaustion, inefficiency, and a failure to significantly improve an organization's security posture. The talk advocates for a strategic shift from reactive bug-fixing to proactive, class-based problem-solving.

Lopez draws upon Microsoft's extensive experience in vulnerability response, sharing insights and success stories from one of the world's largest software vendors. He illustrates how a dedicated focus on root cause analysis, coupled with significant engineering investment in policy, tooling, and fundamental platform changes, can effectively eliminate entire categories of security issues. The core message is a call to action for security professionals across all roles—from red teamers to blue teamers, security researchers, and enterprise architects—to break the "law of inertia" and embark on long-term initiatives that promise compounding returns, ultimately allowing organizations to "yield the mallet" and move beyond the perpetual cycle of reactive defense.

The presentation emphasizes that true security advancement comes not just from fixing more bugs, but from strategically identifying and mitigating the systemic weaknesses that generate them. By showcasing Microsoft's journey in tackling major vulnerability classes like cryptography issues, stack corruptions, use-after-free bugs, and uninitialized memory, Lopez provides a compelling framework for how any organization can move towards a more sustainable and impactful security strategy. This shift is crucial for combating the ever-increasing volume of security reports and ensuring customer safety in a complex threat landscape.

Background

▶ Watch: Introduction: Understanding security whack-a-mole (0:00)

The "security whack-a-mole" metaphor aptly describes the common state of many security operations: constantly reacting to new threats, alerts, or vulnerabilities as they emerge, only to see more surface shortly after. Lopez explains that while fixing an individual bug or responding to an alert can provide immediate satisfaction, these actions often only move the needle slightly, leading to an exhausting and ultimately unsustainable cycle. This inertia, he suggests, is often easier to maintain than taking decisive action to solve a class of issues once and for all.

To contextualize the challenge, Lopez shares a glimpse into Microsoft's product security ecosystem and its annual vulnerability response metrics. The Microsoft Security Response Center (MSRC), Microsoft's equivalent of a Product Security Incident Response Team (PSIRT), stands at the forefront of this effort. Supporting MSRC are distributed Security Assurance organizations, Incident Response teams (tailored to specific products like Xbox Live, Azure, or Office 365), Threat Intelligence teams, and the core Product Groups where engineering and maintenance occur.

In 2019, Microsoft received approximately 9,600 valid vulnerability submissions, leading to 5,500 cases opened for investigation. Of these, 4,060 were unique issues, with 1,750 deemed "critical or important" severity, warranting a security update. The company issued 899 unique CVEs (Common Vulnerabilities and Exposures) that year. Furthermore, Microsoft's bug bounty program is incredibly active, paying a record $13.7 million in rewards in 2019. This volume highlights the sheer scale of vulnerabilities being reported and processed.

A critical observation is that over a four-year period, all these metrics—total cases, unique cases, critical/important issues, and CVEs—showed an upward trend. This might initially suggest a worsening "whack-a-mole" problem. However, Lopez clarifies that an increase in reported vulnerabilities does not necessarily equate to a decline in security posture. Microsoft analyzes the percentage of CVEs exploited within 30 days of a fix being available, and this metric has consistently decreased over time, indicating improved customer protection.

Moreover, widespread attacks, once common in the early 2000s (server-side, productivity app, and browser exploits), are now becoming the exception rather than the norm. This shift is attributed to significant industry-wide advancements in mitigations and security features. Lopez cites examples like Windows XP Service Pack 2 (firewall by default), Windows Vista/7 (ASLR), Office 2010 (DEP, ASLR, Protected View), and Microsoft Edge (sandbox, ASLR, DEP, Control Flow Guard). These mitigations have not only fixed individual vulnerabilities but have also eliminated entire classes of issues or significantly reduced their severity, demonstrating the power of a proactive, systemic approach.

Key Findings

▶ Watch: Reducing customer risk despite increasing vulnerability reports (13:00)

The central finding of this talk is that it is possible to escape the "security whack-a-mole" by shifting focus from individual vulnerability fixes to identifying and resolving root causes that affect entire classes of issues. Microsoft's experience demonstrates that strategic investment in policy, tooling, compiler-level changes, and even fundamental programming language shifts can lead to a dramatic reduction in risk and operational overhead.

Key findings and contributions include:

• Root Cause Analysis is Foundational: By categorizing and analyzing the root causes of reported vulnerabilities, organizations can pinpoint systemic weaknesses. Microsoft's data revealed cross-site scripting and memory safety issues as the two largest categories, consistent with industry trends.
• Elimination of Vulnerability Classes: Proactive measures can completely or significantly reduce the occurrence of specific vulnerability types. Examples from Microsoft include:
• Cryptography Issues: A 10% share of cases in 2016 was virtually eliminated by 2018 through policy enforcement (e.g., mandatory HTTPS for new websites) and automated tooling for identification and remediation.
• Stack Corruptions: Once a significant portion of memory corruption bugs, these have been "essentially dead" due to platform-level mitigations.
• Use-After-Free (UAF) Bugs in Internet Explorer: A surge in UAF reports between 2013-2015 was addressed by the development and deployment of Memory Garbage Collector (MGC), which effectively eliminated this class of issues in IE.
• Compiler-Based Solutions for Systemic Problems: Significant classes of issues, like uninitialized memory vulnerabilities, can be addressed at the compiler level. The InitAll mitigation, which automatically initializes stack variables to zero, effectively eliminated new reports of this bug class on supported platforms.
• Long-Term Strategic Shifts: For pervasive issues like memory safety, a long-term strategy involving the adoption of memory-safe programming languages like Rust is being actively pursued. This represents a multi-year journey to fundamentally alter the attack surface.
• Reduced Customer Risk Despite Increased Reports: While the volume of reported vulnerabilities has increased, the risk to customers has decreased, evidenced by fewer widespread attacks and a lower percentage of CVEs exploited post-patch. This indicates the effectiveness of the strategic, class-based approach.

These findings underscore that a commitment to deep analysis, strategic planning, and significant engineering effort can transform a reactive security posture into a proactive, risk-reducing one, ultimately yielding greater security dividends.

Technical Deep Dive

▶ Watch: Root cause analysis: Eliminating cryptography issues (20:00)

Microsoft's journey to escape the security whack-a-mole involved a multi-pronged technical approach, focusing on deep root cause analysis and implementing systemic solutions.

Root Cause Analysis and Categorization:

Lopez highlights that analyzing vulnerabilities by their root cause is paramount. Microsoft categorizes issues into six main types, with cross-site scripting (XSS) and memory safety issues consistently being the "big two" across the industry. This analysis revealed a specific success story: cryptography issues. In 2016, these constituted about 10% of reported vulnerabilities. The problem stemmed from a proliferation of Microsoft websites, many of which were collecting sensitive information over insecure HTTP connections. The solution was not technical in the sense of a new code-level mitigation, but rather a combination of policy enforcement (requiring HTTPS for all new websites collecting data, even at the contracting phase) and tooling (proactively identifying and flagging existing insecure websites for remediation). This proactive, policy-driven approach, implemented throughout 2017, effectively eliminated this class of issues for the MSRC response function.

Tackling Memory Safety Issues:

Memory safety remains a pervasive problem, particularly in system-level programming languages like C and C++. Within this category, Lopez details several specific successes:

1. Stack Corruptions: Once accounting for over 20% of memory corruption vulnerabilities in 2006, these are now "essentially dead" due to various platform-level mitigations developed over time.
2. Use-After-Free (UAF): Between 2013 and 2015, a significant increase in UAF reports was observed, driven by security researchers becoming highly adept at finding and exploiting them. In response, a dedicated team developed MGC (Memory Garbage Collector) for Internet Explorer. This mitigation effectively eliminated the UAF class of issues in IE, demonstrating how a targeted, data-driven engineering effort can address a specific, prevalent bug class.

The Uninitialized Memory Initiative:

By 2018, uninitialized use emerged as a growing concern within memory safety, with 82 reports in 2017 and 61 in 2018. This problem presented two key challenges:

• Security Risk: Uninitialized memory can lead to critical security issues like data leaks or code execution.
• Assessment Difficulty: It was "extremely difficult to prove whether one of these bugs actually had or did not have a security impact." As a conservative software vendor, Microsoft had to fix many such bugs even without clear security impact, leading to churn for internal teams and patch fatigue for customers.

Led by MSRC engineer Joe BC, a coalition involving Windows product groups and the Visual Studio team (recognizing a compiler-based solution was likely) was formed. The solution, called InitAll, involved automatically initializing stack variables to zero. This required meeting stringent engineering criteria from the Windows team:

• Security: Must genuinely make things safer.
• Performance: No unacceptable performance degradation.
• Compatibility: Must not break existing Windows programs.
• Flexibility: Must not "paint ourselves into a corner" for future development, given Windows' long support lifecycle.

The development of InitAll was an iterative process. Early attempts saw Windows fail to boot. After refinement and careful trade-offs, the first phase of InitAll for uninitialized stack memory went live in November 2018 and was incorporated into Windows in March/April 2019. Since its deployment, Microsoft has not seen a single uninitialized memory vulnerability reported that reproduces on a platform with the InitAll mitigation. While not all older platforms could receive this mitigation, it represented a significant win for new Windows versions.

A separate but related problem was uninitialized Kernel pool memory, which accounted for roughly half of the reported uninitialized memory issues. This required a different solution: two new APIs that are zero by default. Unlike InitAll, this requires explicit code changes by developers to adopt. The APIs were released earlier in the year, and Microsoft is now on a long journey to drive adoption within Windows teams and among third-party driver vendors.

The Future: Safer Systems Programming Languages (Rust):

Lopez highlights that despite these successes, the fundamental problem of memory safety in system-level languages persists. Microsoft's long-term vision is to migrate, where possible, to memory-safe programming languages. Rust has emerged as the preferred choice for several reasons:

• Performance and Control: Rust offers performance comparable to C/C++, essential for low-level systems.
• Strong Safety Guarantee: Rust is memory-safe by design, with unsafe operations explicitly marked by the unsafe keyword.
• Correctness: "If it compiles, it works" often holds true due to Rust's strong type system and borrow checker.
• Community: A vibrant community provides robust tooling, training, documentation, and libraries, offering confidence for a long-term commitment.

This transition is acknowledged as a "10-year journey," a massive undertaking for a codebase as vast as Windows. The strategy involves:

1. Finding Internal Enthusiasts: Building momentum from engineers already experimenting with Rust.
2. Joining the International Community: Participating and contributing to the Rust ecosystem.
3. Starting Small: Embarking on experimental projects to understand integration challenges and benefits. Examples include:

• Intern projects designing COM libraries in Rust.
• Building the security daemon for Azure IoT Edge in Rust.
• An experimental rewrite of a low-level component of Windows in Rust.
• Using Rust in other cloud components.
• Making platform security technologies like Control Flow Guard available in Rust.

This phased, experimental approach allows Microsoft to gather learnings, address technical challenges (like build system integration and interoperation with C/C++), and gradually build confidence for broader adoption. The ultimate goal is to eliminate the largest class of security issues—memory safety—at its source.

Demo / Proof of Concept

▶ Watch: Embracing Rust for future memory-safe systems programming (42:00)

While the talk did not feature a live, interactive demonstration in the traditional sense, Jorge Lopez effectively presented several "proofs of concept" through real-world examples of Microsoft's implemented mitigations and ongoing strategic initiatives. These examples serve as concrete evidence of how the "yield the mallet" philosophy translates into tangible security improvements.

The elimination of cryptography issues, with a 10% reduction in cases, serves as a proof of concept for policy and tooling-driven remediation. The deployment of Memory Garbage Collector (MGC) for Internet Explorer, which directly addressed the surge in use-after-free vulnerabilities, stands as a clear technical demonstration of how a targeted mitigation can nullify an entire class of bugs. Most prominently, the InitAll compiler-based solution for uninitialized stack memory is a powerful proof of concept; its successful deployment and subsequent absence of new reports on supported platforms unequivocally demonstrate its effectiveness.

Similarly, the ongoing efforts to adopt Rust, though a multi-year journey, are presented with specific project examples (e.g., Azure IoT Edge security daemon, experimental Windows component rewrites). These are "proofs of concept in progress," showcasing the early, deliberate steps towards a future where memory-safe languages fundamentally eliminate a vast category of vulnerabilities. These practical examples, backed by real-world data and engineering challenges, serve to illustrate the viability and impact of the strategies discussed.

Defensive Implications

▶ Watch: Q&A: Bug bounties and class-level vulnerability fixes (55:00)

The insights shared by Jorge Lopez offer profound defensive implications for security professionals aiming to move beyond reactive "whack-a-mole" tactics:

• Prioritize Root Cause Analysis (RCA): Defenders must invest time and resources into systematically analyzing the root causes of vulnerabilities, security incidents, and recurring alerts. This is crucial for identifying patterns and systemic weaknesses rather than just fixing individual symptoms.
• Leverage Data for Strategic Investment: Use collected data to identify high-return-on-investment opportunities. Instead of patching an endless stream of similar bugs, identify the underlying class of issues and seek a solution that addresses them broadly (e.g., policy, tooling, platform-level changes).
• Embrace Incremental Solutions: Don't let the pursuit of a perfect, 100% solution paralyze action. An 80% solution, like InitAll for stack memory, can have a massive immediate impact, allowing resources to then tackle the remaining challenges (e.g., kernel pool memory). Break down large problems into manageable parts.
• "Have Skin in the Game": Security professionals should move beyond simply identifying problems. They should actively participate in designing and implementing solutions, collaborating closely with engineering and product teams. This means rolling up sleeves and contributing directly to the resolution, rather than just dictating requirements.
• Carve Out Time for Long-Term Projects: Managers and individual contributors alike must allocate dedicated time for strategic, long-term initiatives that promise to eliminate entire classes of issues. These projects, though potentially spanning years (like the Rust adoption), offer exponential returns on investment by fundamentally altering the security landscape.
• Consider Policy and Tooling for Broad Impact: Simple yet effective solutions can stem from robust security policies (e.g., mandatory HTTPS) enforced through automated tooling. This can eliminate common, lower-severity issues that consume significant response resources.
• Invest in Compiler and Platform-Level Mitigations: For deeply entrenched issues like memory safety, collaborate with platform and compiler teams to implement systemic mitigations (e.g., InitAll). These changes provide broad protection across many applications built on the platform.
• Explore Memory-Safe Languages: For new development or rewrites of critical components, seriously evaluate and gradually adopt memory-safe languages like Rust. This is a long-term, fundamental shift that can drastically reduce the attack surface for an entire category of severe vulnerabilities.
• Foster Cross-Organizational Partnerships: Successful initiatives, like InitAll or Rust adoption, require strong partnerships between security response, product engineering, and even external communities. Breaking down organizational silos is essential for implementing large-scale changes.

Key Takeaways

• Shift from Reactive to Proactive: Move beyond the "security whack-a-mole" by identifying and addressing systemic root causes of vulnerabilities rather than just fixing individual instances.
• Data-Driven Decisions: Utilize root cause analysis of vulnerabilities, incidents, and alerts to identify patterns and prioritize long-term solutions with high return on investment.
• Policy, Tooling, and Compiler Mitigations are Powerful: Entire classes of issues (e.g., insecure HTTP, stack corruptions, use-after-free, uninitialized memory) can be significantly reduced or eliminated through strategic policies, automated tooling, and platform-level compiler changes.
• Embrace Incremental and Long-Term Strategies: Don't let the scope or duration of a problem deter action. Start small, iterate, and commit to multi-year initiatives (like Rust adoption) that promise fundamental security improvements.
• Active Participation is Key: Security professionals should "have skin in the game," actively collaborating with engineering teams to design and implement solutions, rather than just pointing out problems.
• Memory-Safe Languages are the Future: For system-level programming, migrating to languages like Rust offers a long-term, fundamental solution to the pervasive and costly problem of memory safety vulnerabilities.

About the Speaker(s)

Jorge Lopez was a Principal Security PM Manager at Microsoft at the time of this talk. He previously led Microsoft's highly critical Vulnerability Response Team (MSRC), which is responsible for receiving and responding to security vulnerabilities across Microsoft's vast product ecosystem. He has since transitioned to a role as Threat Intelligence Lead, reflecting his deep expertise in understanding and addressing complex security challenges. Lopez is passionate about moving the security industry beyond reactive "whack-a-mole" approaches towards more strategic, impactful solutions.