Making the Case for DevSecOps

Mark Miller (Core Organizer of DevSecOps Days)

OFFZONE 2019

Overview

In his compelling OFFZONE presentation, "Making the Case for DevSecOps," Mark Miller, a core organizer of DevSecOps Days and co-founder of All-Day DevOps, meticulously traces the historical imperative for integrating security into the software development lifecycle. Miller argues that in today's complex, open-source driven application landscape, traditional, siloed security approaches are not merely inefficient but fundamentally obsolete. The talk serves as a critical call to action, emphasizing that security must be an intrinsic part of development from inception, a concept encapsulated by the term DevSecOps.

Watch on YouTube

Visual summary for Making the Case for DevSecOps by Mark Miller
Visual summary for Making the Case for DevSecOps by Mark Miller

Key moments

  1. 0:00 Introduction: DevSecOps roots in quality principles
  2. 7:00 Eli Goldratt's Theory of Constraints: Optimize bottlenecks
  3. 13:00 The Phoenix Project: Applying constraints to IT
  4. 20:00 Exploit time drastically reduced: Urgency for speed
  5. 22:00 Shannon Lietz: Security is everyone's responsibility
  6. 27:00 Equifax breach: A case study in exploit speed
  7. 38:00 Build safe: You cannot inspect quality into a product
  8. 42:00 Key takeaways: Responsibility, optimization, automation

Making the Case for DevSecOps

Speakers: Mark Miller, Core Organizer of DevSecOps Days

Conference: OFFZONE

YouTube: https://www.youtube.com/watch?v=ZndetW4BR_s

Overview

In his compelling OFFZONE presentation, "Making the Case for DevSecOps," Mark Miller, a core organizer of DevSecOps Days and co-founder of All-Day DevOps, meticulously traces the historical imperative for integrating security into the software development lifecycle. Miller argues that in today's complex, open-source driven application landscape, traditional, siloed security approaches are not merely inefficient but fundamentally obsolete. The talk serves as a critical call to action, emphasizing that security must be an intrinsic part of development from inception, a concept encapsulated by the term DevSecOps.

Miller's address is particularly pertinent given the accelerating pace of software delivery and the pervasive reliance on open-source components, which now constitute an estimated 80% of modern applications. He highlights the alarming rate of known vulnerabilities propagating through the software supply chain and the drastically shrinking window for remediation before exploitation. The core message resonates with anyone involved in software delivery: security is no longer the sole responsibility of a dedicated team but a collective endeavor, demanding automation, cultural shifts, and a holistic understanding of the entire system.

The presentation provides a robust framework for understanding the evolution from quality assurance to DevOps, and finally to DevSecOps, underpinned by foundational management theories. Miller leverages real-world data and infamous breaches like Equifax to underscore the catastrophic consequences of neglecting integrated security. For developers, operations teams, and security professionals alike, this talk offers not just a historical perspective but actionable insights into building more secure, resilient software systems in an era of relentless cyber threats.

Background

▶ Watch: Introduction: DevSecOps roots in quality principles (0:00)

The journey towards DevSecOps, as articulated by Mark Miller, is a logical progression rooted in decades of management and quality theory, adapted for the unique challenges of software development. The conceptual groundwork was laid by W. Edwards Deming in post-World War II Japan, where he introduced principles of quality control and process flow to industries like Toyota. Deming's core tenet was that quality must be built into a product from the very beginning, rather than inspected at the end. Miller cleverly reframes "quality" as "security" for the audience, demonstrating the timeless relevance of Deming's ideas: addressing issues early significantly reduces cost and risk downstream. This foundational idea directly challenges the outdated model of manual, late-stage security inspection, which is demonstrably ineffective in today's rapidly evolving, complex systems.

Building on Deming's principles, Miller introduces Eli Goldratt's Theory of Constraints (ToC), detailed in his 1984 book The Goal. Goldratt posited that every complex system has a bottleneck that dictates its overall speed and output. Critically, he argued against local optimization – improving every part of a system independently – as a waste of resources. Instead, resources should be focused solely on alleviating the primary constraint. Miller translates this directly to software development, identifying common bottlenecks like an overwhelmed "go-to guy" or a manual security review process. The ToC provides a powerful analytical lens for identifying where security efforts should be concentrated for maximum impact, rather than spreading resources thinly across the entire system.

The direct lineage from Goldratt's ToC to modern software practices is most evident in the DevOps movement. Miller highlights pivotal moments: John Allspaw and Paul Hammond's presentation on "10+ Deploys Per Day" at Velocity 2009, which challenged conventional wisdom; Patrick Debois's coining of "DevOpsDays"; and the publication of seminal works like Continuous Delivery by Jez Humble and Dave Farley, and The Phoenix Project by Gene Kim. Kim's book, in particular, is noted as a direct application of Goldratt's ToC to the software supply chain within an enterprise setting, illustrating how development and operations teams can overcome bottlenecks. The DevOps Enterprise Summit, founded by Gene Kim, further solidified these practices, bringing together major enterprises to share their experiences.

However, the rapid adoption of DevOps, while improving speed and agility, did not inherently solve the growing security crisis. As Miller explains, the sheer volume of open-source components—making up 80% of modern applications—introduced a vast and often unmanaged attack surface. The annual State of the Software Supply Chain Report, launched in 2016 by Derek Weeks, began to quantify this problem, revealing that 1 in every 16 open-source downloads contained a known vulnerability. This burgeoning threat landscape, coupled with the accelerating speed of exploits, necessitated a further evolution: DevSecOps. Shannon Lietz, dubbed the "godmother of DevSecOps," articulated the core principle in the DevSecOps Manifesto: security is no longer the sole responsibility of a dedicated team, but a shared accountability across the entire software development lifecycle. This cultural shift, Miller emphasizes, is paramount, as tools alone are insufficient without collective buy-in and a proactive approach to security from the very beginning.

Key Findings

▶ Watch: The Phoenix Project: Applying constraints to IT (13:00)

Mark Miller's presentation unveils several critical findings that underscore the urgency and necessity of adopting DevSecOps practices in modern software development. These insights are not merely theoretical but are backed by alarming statistics and real-world examples.

Firstly, the ubiquity and vulnerability of open-source software form a central theme. Miller highlights that approximately 80% of a modern application is built from open-source components, with proprietary code often serving as the "glue" that binds them together. This reliance translates into staggering download figures: 131 billion downloads of open-source components annually from the Maven Central repository in 2016, escalating to 146 billion by 2019. The critical finding is that a significant portion of these downloads carries known risks; in 2016, 1 in every 16 downloads had a known vulnerability, and by 2019, 10.3% of 130 billion downloads contained known vulnerabilities. This indicates a massive influx of insecure components directly into production systems, often without developers' or security teams' awareness.

Secondly, Miller dramatically illustrates the shrinking exploit window. He states that the time adversaries take to exploit a newly discovered vulnerability has decreased by a staggering 93%. Where organizations once had approximately 45 days to fix a vulnerability, this window has condensed to as little as three days, or even hours, before active exploitation begins. This rapid exploitation rate renders traditional, manual security processes utterly ineffective. The Equifax breach serves as a stark example: the Apache Struts 2 vulnerability was announced, and within three days, adversaries exploited it to breach Equifax's systems. Equifax, however, remained unaware of the breach for five months, highlighting a critical failure in tracking, monitoring, and notification. Even more concerning, Miller notes that after the Equifax breach was publicly announced, there was an increase in downloads of the vulnerable Struts 2 component, demonstrating a systemic lack of awareness and automated safeguards.

Thirdly, the talk emphasizes that manual security processes are no longer viable. With the exponential growth of software complexity, the sheer volume of open-source components, and the speed of modern exploits, a security team cannot manually inspect or remediate issues. Miller points out the daunting ratio of security professionals to developers (e.g., 75 security people trying to serve 7,500 developers or items), making individual oversight impossible. This necessitates the integration of automation and security gates directly into the software development process.

Finally, Miller underscores the critical importance of a cultural shift over a mere tool adoption. While tools are necessary, he stresses that true DevSecOps success hinges on buy-in from the entire organization, from developers to the C-suite. Without this shared responsibility and collective commitment, any DevSecOps initiative is destined to fail. This cultural transformation means everyone in the software supply chain takes ownership of security at their point of contact, rather than deferring it to a separate security team at the end of the cycle.

Technical Deep Dive

▶ Watch: Shannon Lietz: Security is everyone's responsibility (22:00)

The technical core of Mark Miller's DevSecOps advocacy revolves around proactive, integrated strategies for managing the software supply chain and optimizing security processes. These strategies are directly derived from the foundational theories discussed in the background.

A primary technical focus is supply chain management for open-source components. Miller emphasizes three critical aspects:

  1. Minimum Number of Suppliers: While Deming advocated for a single supplier, in the open-source world, this translates to choosing the "best source" for components. This means not indiscriminately grabbing packages but identifying reputable providers of high-quality, well-maintained open-source projects.
  2. Best Parts from Those Suppliers: Even from trusted suppliers, not all versions or components are equally secure. Organizations must actively select the most secure versions.
  3. Track and Monitor: This is presented as the most crucial and often overlooked step. Once components are integrated into applications and deployed, organizations must know where they are. Miller recounts the widespread ignorance during Heartbleed and Struts 2 vulnerabilities, where many enterprises spent weeks manually searching Excel spreadsheets to identify affected applications, often missing a significant number. Automated tracking and monitoring systems are essential to identify vulnerable components in production and receive immediate notifications when new vulnerabilities are announced. Without automated notification, tracking alone is insufficient.

Miller then delves into the practical application of Eli Goldratt's Theory of Constraints (ToC) within software development. The ToC methodology involves:

  1. Identify the Constraint: Pinpoint the single slowest part or bottleneck in the entire software supply chain (e.g., a specific team, a manual security review process, a legacy system).
  2. Exploit the Constraint: Maximize the output of the bottleneck using existing resources.
  3. Subordinate Everything Else: Align all other activities and resources to support the bottleneck, ensuring it's never starved for work. This means taking resources from faster, non-bottleneck processes and directing them to the constraint.
  4. Elevate the Constraint: If the bottleneck still limits overall system output, invest additional resources (e.g., new tools, training, personnel) to improve its capacity.
  5. Go Back to Step 1: Once a constraint is alleviated, a new one will emerge, and the cycle repeats.

Miller illustrates this with the analogy of a Boy Scout troop marching, where placing the slowest boy at the front ensures the entire troop moves together efficiently. In software, this means addressing the slowest security process or the most critical vulnerability first, rather than optimizing minor local inefficiencies.

The concept of a Software Bill of Materials (SBOM) is introduced as a critical technical component for future security. Just as food products list ingredients, software applications should carry a comprehensive list of all their constituent components. This is especially relevant for containerized applications, where the contents of a container should be transparently declared rather than requiring introspection to determine. An SBOM provides tangible, immediate information when a vulnerability is announced, enabling rapid identification of affected applications. Miller notes that mature DevOps practices increasingly incorporate SBOMs, with 50% of mature DevOps organizations using them.

Finally, Miller addresses the principle of "never pass a defect downstream," offering a nuanced perspective from DJ Chalene. While ideal, completely halting the pipeline for every defect can slow down the business. The refined approach suggests that a defect (e.g., a vulnerable component) can move through the system, but it must be explicitly tagged as a known issue. This ensures that while the development flow continues, the component is unequivocally marked as unsuitable for production and must be remediated before release. This approach maintains flow while preserving security integrity, providing time for fixes without completely grinding the system to a halt. This intricate balance highlights the need for sophisticated automated security gates that can enforce policies and provide visibility into tagged defects without becoming an insurmountable bottleneck.

Demo / Proof of Concept

▶ Watch: Equifax breach: A case study in exploit speed (27:00)

No explicit live demo or proof of concept was presented during Mark Miller's talk. The presentation primarily focused on a historical overview, theoretical frameworks, statistical data, and strategic arguments to make the case for DevSecOps. The speaker leveraged real-world examples and industry reports to illustrate the points, rather than demonstrating specific tools or code.

Defensive Implications

▶ Watch: Key takeaways: Responsibility, optimization, automation (42:00)

The insights shared by Mark Miller provide a clear roadmap for organizations to bolster their security posture and mitigate the escalating risks in the modern software landscape. The defensive implications are profound and necessitate a fundamental shift in strategy and culture.

Firstly, automation of security processes is non-negotiable. Miller repeatedly stresses that manual security is "literally impossible today" due to the volume of open-source components (146 billion Java downloads projected for 2019), the complexity of systems, and the drastically shrinking exploit window (from 45 days to 3 days). Defenders must integrate automated tools for vulnerability scanning, component analysis, and dependency tracking directly into their CI/CD pipelines. This means having systems that can automatically identify known vulnerabilities in open-source components, track their deployment across the enterprise, and provide immediate notification upon the announcement of new exploits. Without this, organizations will inevitably fall victim to the speed of adversaries, as exemplified by the Equifax breach.

Secondly, organizations must "shift left" security. This means embedding security considerations and practices into the earliest stages of the software development lifecycle, rather than treating security as a final gate before production. Deming's principle of building quality in from the start directly applies here. Developers need to be empowered with security knowledge, guidelines, and feedback loops that allow them to address vulnerabilities as they write code, where remediation is exponentially cheaper and more efficient. This proactive approach prevents defects from propagating downstream, avoiding costly rework and critical breaches.

Thirdly, fostering a culture of shared security responsibility is paramount. The traditional model of a small security team policing an entire organization is unsustainable. Miller advocates for a model where "everyone is responsible" for security. This requires educating developers, operations teams, and even business leaders about their role in maintaining security. Security teams should transition from being gatekeepers to enablers, providing guidelines, tools, and expertise to empower development teams to secure their own code. This cultural buy-in, from the C-suite down, is crucial for any DevSecOps initiative to succeed.

Fourthly, robust open-source software supply chain security must be implemented. Given that 80% of modern applications rely on open source, organizations need to:

  • Vet Suppliers: Choose trusted sources for open-source components.
  • Select Secure Versions: Actively identify and use the most secure available versions of components.
  • Track and Monitor: Implement automated systems to know exactly which open-source components are in use, where they are deployed, and their current vulnerability status.
  • Maintain a Software Bill of Materials (SBOM): Generate and maintain a comprehensive list of all components within each application and container. This provides critical transparency and enables rapid response when vulnerabilities are disclosed.

Finally, adopting the Theory of Constraints (ToC) provides a strategic framework for prioritizing defensive efforts. Security teams should analyze their entire software delivery pipeline to identify the primary bottleneck — whether it's a specific manual review, a legacy system, or a particular vulnerability class. By focusing resources on alleviating this single constraint, organizations can achieve significant overall improvements in their security posture, rather than diluting efforts through local optimizations that don't impact the slowest part of the system. This also implies that increasing deployment frequency, as noted by Miller, can lead to more secure systems, as smaller, more frequent changes are easier to secure and remediate.

Key Takeaways

  • Everyone is Responsible for Security: Security is no longer the sole domain of a dedicated team but a shared responsibility across the entire software development lifecycle, from developers to operations and leadership.
  • Automate Security from Inception: Manual security processes are obsolete and ineffective against the speed and scale of modern threats. Automation of vulnerability scanning, component tracking, and policy enforcement must be integrated into the CI/CD pipeline from the very beginning.
  • Manage Your Open-Source Supply Chain: With 80% of modern applications built from open source, organizations must actively vet component sources, choose secure versions, and implement automated tracking, monitoring, and notification systems for all third-party dependencies.
  • Identify and Alleviate Bottlenecks: Apply Eli Goldratt's Theory of Constraints to security. Focus resources on the slowest or most critical security constraint in your system to achieve the greatest overall improvement in security posture, rather than optimizing locally.
  • The Exploit Window is Shrinking Rapidly: The time from vulnerability disclosure to active exploitation has condensed from 45 days to as little as 3 days. This demands continuous, integrated security practices and rapid remediation capabilities.
  • Adopt a Software Bill of Materials (SBOM): Implement SBOMs for applications and containers to provide transparent, machine-readable lists of all included components, enabling faster identification and response to newly discovered vulnerabilities.

About the Speaker(s)

Mark Miller is a prominent figure in the DevSecOps community, known for his extensive contributions to fostering and promoting the integration of security into the DevOps methodology. He serves as a Core Organizer of DevSecOps Days, a globally recognized series of events dedicated to advancing DevSecOps practices. Miller is also a co-founder of All-Day DevOps, an innovative online conference that brought together 13,400 people globally in its first year and grew to 30,000 the following year, demonstrating his commitment to making DevSecOps knowledge accessible worldwide. He opened the talk by noting he had flown in from New York City for OFFZONE.

Reviews

Dr. Zero (Offensive Security Researcher) — WEAK

A DevSecOps 101 talk that could have been a blog post in 2017. Miller traces the Deming-to-Goldratt-to-DevOps lineage competently, but there's nothing here that wasn't in Gene Kim's books a decade ago. The 'everyone is responsible for security' conclusion is not a finding — it's the conference's default assumption.

Heather Calloway (CISO) — WEAK

A well-intentioned overview of DevSecOps history and principles, but this is 2019 material delivered without the specificity or rigor a CISO audience needs. No new data, no operational playbooks, no hard decisions examined. If your leadership team doesn't already understand why security belongs in the pipeline, this is fine remedial content. Otherwise, it's a conference filler.

→ Top-rated talks at OFFZONE 2019

All talks from OFFZONE 2019