The Myths Of Software Security

The Myths Of Software Security

Speakers: Mark Curphey (Crash Override)

Conference: HITB Amsterdam

YouTube: https://www.youtube.com/watch?v=a9e6NFGwHvk

Overview

In his characteristically frank and uncompromising style, Mark Curphey, founder of Crash Override, delivered a provocative talk at HITB Amsterdam titled "The Myths Of Software Security." Curphey’s presentation served as a stark call to action for security professionals to critically re-evaluate long-held industry dogmas, marketing-driven narratives, and self-serving benchmarks that often distract from genuine security challenges. He argued that much of what is considered "best practice" in software security is built on shaky foundations, often propagated by vendors, analysts, or committees with vested interests, leading practitioners to waste valuable time and resources on initiatives that offer little real-world benefit.

Curphey, drawing from decades of experience in the field, systematically dismantled several prevalent myths, from the efficacy of "shift left" to the reliability of CVEs and SBOMs, and the integrity of industry awards and benchmarks. His core message was an impassioned plea for security professionals to "not give a f\\k" about the noise and instead focus their efforts on verifiable, impactful security measures that address actual risks. The talk resonated with its blunt honesty, challenging attendees to think independently and apply critical scrutiny to the information they consume and the strategies they implement.

This article delves into Curphey’s arguments, dissecting the myths he exposed and exploring the technical and practical implications for software security. It aims to provide a comprehensive overview of his findings, offering actionable insights for defenders seeking to navigate the complex and often misleading landscape of modern cybersecurity.

Background

▶ Watch: Introduction: The 'don't give a f***' philosophy (0:00)

The cybersecurity industry, particularly software security, is characterized by rapid evolution, complex threats, and a constant influx of new tools and methodologies. This dynamic environment, however, has also fostered a culture susceptible to marketing hype, unverified claims, and self-serving narratives. Mark Curphey highlighted this pervasive issue, drawing parallels to historical examples like the tobacco industry's misleading health claims or the unproven assertions of caffeine shampoos. He recounted an early career experience at Charles Schwab, where a purported "vulnerability" turned out to be a marketing ploy by a web application firewall vendor, illustrating how easily security teams can be sidetracked by what he termed "village idiots" and commercial rackets.

Curphey emphasized that this susceptibility extends beyond individual incidents to foundational aspects of the industry. Many widely accepted "truths" or "best practices" originate from sources that lack scientific rigor, independent validation, or are directly influenced by commercial interests. Analyst firms like Forrester and Gartner, he noted, often operate on a "pay-to-play" model, where their research can be influenced by vendor payments. Similarly, industry awards are frequently exposed as opaque schemes where nominations can be purchased, and winning odds increased through financial contributions. This background sets the stage for Curphey's deep dive into specific myths, revealing how deeply entrenched these misleading practices have become in shaping security strategies and tool adoption across organizations.

Key Findings

▶ Watch: Debunking the 'Shift Left' security principle (10:13)

Mark Curphey’s talk centered on debunking several core myths that he believes permeate the software security industry, causing misdirection and inefficiency. His key findings can be summarized as follows:

1. The "Shift Left" Myth is Unsubstantiated: The widely cited claim that fixing bugs earlier in the development lifecycle is 100x or 200x cheaper lacks credible evidence. This notion largely stems from an untraceable 1967 IBM study, which is irrelevant to modern CI/CD development practices.
2. Software Bill of Materials (SBOMs) are Currently Flawed: While conceptually sound, current SBOM implementations suffer from a lack of build provenance and code provenance. Factors like local package caches (e.g., npm) can lead to non-deterministic SBOMs for the same source code. Furthermore, SBOMs can be easily faked or manipulated, rendering them untrustworthy without robust attestation mechanisms.
3. Common Vulnerabilities and Exposures (CVEs) are a "Smoke Screen": The CVE system, including the National Vulnerability Database (NVD), does not validate vulnerability submissions; it primarily checks for grammar. The CVE Numbering Authorities (CNAs) model allows vendors to hide or delay public disclosure of issues. More critically, the number of publicly reported CVEs is a trivial fraction of actual vulnerabilities, and they often don't represent the most exploited issues in the wild.
4. Cybersecurity Market Size Reports are Clickbait: Reports from firms like McKinsey or industry journalists claiming multi-trillion-dollar market opportunities often lack rigorous validation and are used by vendors (e.g., Cisco) to promote their products. These are frequently generated by individuals with sales backgrounds, not independent researchers.
5. Community Benchmarks are Self-Serving and Outdated: Benchmarks like the OWASP Benchmark are often created by individuals or groups with vested interests. They can become outdated (e.g., 9 years without significant updates) and are then used to discredit existing security tools (like SAST/DAST) while promoting new approaches that align with the creators' own commercial offerings.
6. The OWASP Top 10 Suffers from Circular Causation and Commercial Bias: While transparent, the OWASP Top 10’s data collection relies heavily on submissions from tool vendors and penetration testing companies. This leads to circular causation, where training based on the Top 10 results in more findings related to the Top 10, perpetuating its relevance even if the actual threat landscape shifts. Furthermore, specialized "Top 10s" (e.g., for API security or Kubernetes) are often created by security companies whose products magically address those specific issues, serving as marketing data sheets.

Curphey’s overarching finding is that a significant portion of the software security industry's advice and metrics are either based on myths, lack scientific validation, or are driven by commercial motivations, demanding a more skeptical and data-driven approach from practitioners.

Technical Deep Dive

▶ Watch: Why CVEs are a 'smoke screen' (20:01)

Curphey’s talk wasn't just about general skepticism; it delved into the technical underpinnings (or lack thereof) of these myths.

The Myth of Shift Left's 100x Cost Savings

The "shift left" paradigm, advocating for security activities earlier in the Software Development Life Cycle (SDLC), is pervasive. Its primary justification is the notion that fixing bugs earlier is dramatically cheaper, often cited as 100x or 200x. Curphey meticulously traced this claim back to a 1997 book by Capers Jones, which referenced an alleged IBM study from 1967. The critical technical flaw, as Curphey pointed out, is that this IBM study likely never existed, and if it did, its context (software development in 1967) bears no resemblance to modern practices. The advent of CI/CD (Continuous Integration/Continuous Delivery) pipelines, automated testing, and rapid deployment fundamentally alters the economics of bug fixing. In a CI/CD world, where code changes are frequent and automated tests run continuously, the cost difference between fixing a bug pre-commit versus post-merge in a development branch is vastly different from fixing a bug found months later in a monolithic mainframe application. The supposed "hockey stick growth curve" of bug fix costs promoted by vendors like GrammaTech (claiming 640% savings) is, therefore, built on a foundation of sand.

The Fragility of SBOMs

The push for Software Bill of Materials (SBOMs) gained significant traction after incidents like SolarWinds and Log4Shell, leading to executive orders like the one from President Biden. However, Curphey highlighted fundamental technical challenges that undermine their current utility. The core problem lies in the non-deterministic nature of software builds and the lack of build provenance.

He used the example of npm, a popular package manager. Building the exact same source code on different machines or at different times can yield different SBOMs. This is because npm might pull from a local cache, leading to variations in library versions that meet specified criteria (e.g., ^1.1.0). Without knowing the precise build environment and conditions, an SBOM for a given binary cannot be definitively linked back to its source or prove its integrity. This means you cannot reliably bind the source code to the binary and guarantee that "this was the input, these were the conditions, therefore this is what this thing represents."

Furthermore, Curphey elaborated on the ease of faking SBOMs. Malicious actors could generate a clean SBOM, then use post-build hooks or other mechanisms within the build process to replace legitimate libraries with vulnerable ones. This "Swiss cheese" problem means that simply possessing an SBOM offers little assurance of software integrity. He advocated for efforts in code provenance and build provenance, referencing projects like Salsa (from Google) and ongoing work by big tech companies to digitally sign artifacts and establish verifiable links between source, build, and binary. He also humorously noted the absence of PHP in these discussions, stating, "not that I hate PHP but I deserve what you get if you use it."

The Illusion of CVEs

The Common Vulnerabilities and Exposures (CVE) system is a cornerstone of vulnerability management. However, Curphey exposed its critical technical shortcomings. He stated unequivocally that when a CVE is submitted, "they don't validate anything." The National Vulnerability Database (NVD), funded by the US government, merely checks for grammar, not the actual existence or severity of the vulnerability.

The distributed nature of CVE Numbering Authorities (CNAs), intended to scale the system, paradoxically creates avenues for abuse. Vendors (e.g., Atlassian, Pivotal for Spring) can register as CNAs for their products. This allows them to "reserve" CVEs, effectively hiding issues from public view for extended periods until they can release a fix, thus controlling the narrative around responsible disclosure.

More profoundly, Curphey cited work from his previous company, SourceClear, which built a database of known vulnerability patterns. This research found that while approximately 8,000 CVEs were public at the time, their analysis predicted around 245,000 actual vulnerabilities. This stark disparity means that the public CVE database represents only a "trivial" fraction of real-world issues. Scanners relying solely on CVEs are therefore missing the vast majority of vulnerabilities, and exploited vulnerabilities are "probably not the ones the scanners find and the CVEs."

He concluded with an absurd but genuine example: CVE-2022-29219, which described how playing Janet Jackson's "Rhythm Nation" music video on an old hard drive could cause random hard drives from ~2005 to stop working. This illustrates the lack of practical filtering and validation in the system, rendering many CVEs meaningless for real-world defense. He pointed to Google's efforts with osv.dev (a distributed vulnerability database with validated issues) and deps.dev (providing library information) as attempts to address these systemic flaws.

Flawed Benchmarks and Top 10 Lists

Curphey extended his critique to industry benchmarks and popular lists. The OWASP Benchmark, for instance, created by individuals like Dave Wickers (whom Curphey respected for their earlier OWASP contributions), had not been significantly updated for nine years. Yet, it was used to claim that traditional SAST (Static Application Security Testing) and DAST (Dynamic Application Security Testing) tools were "rubbish" and that a "new approach" (often aligning with the creators' own products) was needed. This illustrates how even community-driven efforts can be subverted by commercial interests and become technically outdated, leading to misleading conclusions.

Similarly, the OWASP Top 10, while transparent in its methodology, suffers from circular causation. The data feeding the Top 10 comes from tool vendors and pen-testers who are often already looking for Top 10 issues. When organizations then train their 140,000 developers for two days on the OWASP Top 10, they are perpetuating a cycle where the Top 10 remains relevant because everyone is focused on it, rather than necessarily reflecting the true, evolving threat landscape. Curphey cited instances of specialized "Top 10s" for API security or Kubernetes being explicitly created by sales-driven individuals or groups of Israeli startups to generate "sales data sheets," where their own products conveniently address the listed vulnerabilities. The Kubernetes Top 10 was even described as being created by "one guy who actually has a magic Kubernetes tool," with no external scrutiny.

These examples highlight a critical technical challenge: the methodologies for collecting and validating security data, whether for vulnerabilities, benchmarks, or top lists, are often deeply flawed, leading to a distorted view of the actual risks and effective mitigation strategies.

Demo / Proof of Concept

▶ Watch: Flaws in OWASP Top 10 and community benchmarks (29:30)

Mark Curphey’s talk did not include a traditional technical demonstration or proof of concept in the sense of live code or exploit execution. Instead, in line with his philosophy of challenging norms and making security conferences more engaging, he announced a unique, non-technical "Capture the Flag" style treasure hunt.

This event involved a graffiti artist painting a mural somewhere in Amsterdam. Attendees were invited to a reception where they would receive the first clue to a checkpoint. From there, they would race on bikes through Amsterdam, taking selfies at checkpoints and receiving further clues via direct message on Twitter, to locate the final mural. The first team to find the mural and take a selfie would win a framed, signed print of the artwork. This unconventional "demo" served to underscore Curphey's broader message about injecting fun and critical thinking into the security community, moving beyond the often-boring and commercially driven aspects of traditional conferences.

Defensive Implications

▶ Watch: Unique conference treasure hunt for custom art (36:00)

Curphey's debunking of software security myths carries significant defensive implications, urging practitioners to adopt a more critical, data-driven, and pragmatic approach:

1. Prioritize Real Risk over Perceived Risk: Defenders should shift focus from chasing every reported CVE or attempting to "shift left" every bug based on unproven cost savings. Instead, identify and secure important applications and assets that pose the highest risk to the organization. This involves understanding your own unique threat model and asset criticality, rather than relying on generic industry lists.
2. Embrace CI/CD Automation and Production Observability: While the "shift left" mantra is flawed, the principle of integrating security into the development pipeline is sound. Focus on maximizing automation within CI/CD pipelines for testing. Crucially, leverage observability in production environments to understand what vulnerabilities are actually being exploited or causing issues, enabling a data-driven approach to prioritize remediation efforts. Many repos never reach production, so scanning everything indiscriminately is inefficient.
3. Demand Build and Code Provenance for SBOMs: Do not blindly trust SBOMs without verifiable build provenance and code provenance. Defenders should push for tools and processes that can cryptographically sign artifacts and guarantee that a given binary was built from a specific source under known, tamper-proof conditions. Until such mechanisms (like Salsa) are widely adopted and robust, current SBOMs should be treated with extreme skepticism.
4. De-emphasize CVEs as a Primary Metric: CVEs should not be the sole or primary driver of vulnerability management programs. Given their lack of validation, vendor manipulation, and limited scope, relying heavily on CVE counts or "CVE-free" software claims is misleading. Instead, focus on vulnerability intelligence that provides real-world exploitability data and contextual risk. Actively monitor for patterns of vulnerabilities that affect your specific technology stack, even if they don't have a public CVE.
5. Critically Evaluate Security Advice and Tools: Adopt a skeptical mindset towards all security claims, reports, and benchmarks. Before adopting a new tool or strategy, ask: "Who created this? Why did they create it? What is their motivation?" Do not rely on vendor-sponsored reports, awards, or self-serving community benchmarks.
6. Develop Internal Security Tool Validation: Instead of trusting external benchmarks, organizations should develop their own criteria and test cases for evaluating security tools. Utilize your company's historical bug data and known vulnerabilities to assess how effectively a tool identifies issues relevant to your codebase and environment. This ensures that security tools are genuinely effective and provide value in your specific context.
7. Challenge the Status Quo: Curphey's talk is an invitation to challenge long-standing assumptions. Defenders should foster a culture of critical inquiry within their teams, questioning why certain practices are followed and demanding evidence-based justifications for security investments.

By adopting these defensive implications, organizations can move away from chasing marketing fads and towards building more resilient and effective software security programs based on real risks and verifiable data.

Key Takeaways

• Skepticism is Paramount: Do not blindly trust industry claims, market reports, awards, or benchmarks. Always question the source, motivation, and methodology behind security advice and tools.
• Shift Left's Foundation is Flawed: The widely cited cost savings associated with "shift left" are based on an unverified and outdated study, largely irrelevant to modern CI/CD development. Focus instead on efficient CI/CD security automation and production observability.
• SBOMs and CVEs are Deeply Problematic: Current SBOMs lack reliable build provenance and are easily faked, while CVEs suffer from a lack of validation, vendor manipulation, and represent only a fraction of real-world vulnerabilities. They should not be the sole basis for security decisions.
• **Focus on What Matters to Your Organization:** Prioritize securing critical applications and addressing vulnerabilities that pose genuine, data-backed risks to your specific environment, rather than generic industry lists or unproven metrics.
• Validate Tools Internally: Evaluate security tools using your own code, historical bug data, and specific requirements, rather than relying on vendor-created or self-serving community benchmarks like the OWASP Benchmark.
• Demand Provenance and Real Data: Advocate for and adopt technologies that provide verifiable code and build provenance (e.g., Salsa, Google's osv.dev) to establish trust in software components and vulnerability information.

About the Speaker(s)

Mark Curphey is a seasoned veteran in the software security industry and the founder of Crash Override. With a career spanning over two decades, he brings a wealth of practical experience and a refreshingly candid perspective to the field. Early in his career, he ran software security at Charles Schwab, a massive financial services company, where he gained firsthand experience with the challenges and misinformation prevalent in the industry.

Curphey is recognized for his significant contributions to the open-source security community, notably as a co-founder of the OWASP (Open Web Application Security Project), which he started in 2001. He also has entrepreneurial experience, having created a Software Composition Analysis (SCA) company that was later sold to Veracode (SourceClear). His work has extended to advising major initiatives, including contributing to the Linux Foundation's open-source security moonshot strategy focused on SBOMs. Mark Curphey is known for his no-nonsense approach and his commitment to helping security professionals focus on what truly matters.