Pushing the SOC Left to Achieve Nash Equilibrium

Pushing the SOC Left to Achieve Nash Equilibrium

Speakers: O'Shea Bowens (CEO and Founder, Null Hat Security)

Conference: LocoMocoSec

YouTube: https://www.youtube.com/watch?v=8d6HTzT5hO4

Overview

In this insightful LocoMocoSec talk, O'Shea Bowens, CEO and Founder of Null Hat Security, introduces a compelling framework for integrating Security Operations Center (SOC) functions earlier into the software development lifecycle, a concept he terms "Pushing the SOC Left to Achieve Nash Equilibrium." Drawing inspiration from Game Theory and the work of mathematician John Nash, Bowens posits that the traditional disconnect between defensive security (SOC) and application development (DevOps/AppSec) often leads to suboptimal outcomes for both parties. His central thesis is that by aligning their strategies and working synchronously, both the SOC and development teams can achieve a state where neither gains by unilaterally changing their approach, leading to a mutually beneficial security posture.

Bowens, leveraging his extensive background in defensive security, highlights the critical need for SOC analysts and incident responders to understand and engage with the application security and DevOps domains. He argues that this proactive integration, or "pushing left," is no longer optional but a strategic imperative to combat the rising tide of breaches originating from code-related vulnerabilities and misconfigurations. The talk serves as a practical guide for security practitioners looking to bridge this gap, offering actionable steps, tool recommendations, and a mindset shift necessary to build more resilient applications and environments.

The analogy of Nash Equilibrium underscores the idea that in a competitive environment, if one player (e.g., the SOC) makes a strategic adjustment, the other player (e.g., DevOps) might suffer a loss if they don't adapt in kind. However, if both teams adjust synchronously and cooperatively, they can both achieve a "win" state where security is inherently built into the process, rather than bolted on as an afterthought. This talk is crucial for any organization grappling with the complexities of securing modern, rapidly evolving software delivery pipelines.

Background

▶ Watch: Introduction: Pushing SOC Left to Achieve Nash Equilibrium (0:00)

The genesis of this problem lies in the historical separation of concerns within IT and security. Traditionally, Security Operations Centers (SOCs) are tasked with monitoring, detection, and response—the reactive pillars of defense. Bowens simplifies the SOC's role into these three core buckets, emphasizing the paramount importance of robust logging, tagging, and understanding data sources for effective incident identification and recovery. However, he notes that SOCs are often brought into the security process too late, typically after an incident has already occurred, or when vulnerabilities are discovered in production.

On the other hand, DevOps emerged to break down the barriers between software development and IT operations, enabling rapid, continuous delivery of applications. This involves Continuous Integration (CI) and Continuous Delivery/Deployment (CD) pipelines, utilizing tools for source control, build automation, and code testing. While DevOps revolutionized deployment speed (e.g., Amazon deploying 23,000 times a day), it introduced new security challenges. The traditional security controls (like vulnerability management tools, EDR, IDS, SIEM) designed for slower, less agile environments often fail to keep pace with daily or even hourly deployments.

This rapid development cycle gave birth to Application Security (AppSec), which aims to integrate security into the development process itself. However, Bowens points out a persistent "disalignment" where AppSec programs might even operate outside the main security apparatus, leading to a critical lack of communication between AppSec specialists and SOC personnel. The consequence is a "woe is me" scenario for the SOC, where they are left to respond to incidents stemming from "bad code" or "horrible config" without prior involvement or understanding of the development landscape. This disconnect is evident in industry reports, with Bowens citing Verizon's Data Breach Investigations Report (DBIR) indicating that breaches related to "crappy code or crappy controls around code-related automation tools or cloud providers" jumped from 10-20% in 2017-2018 to approximately 40%. This alarming trend underscores why the SOC must proactively engage with AppSec and DevOps.

Key Findings

▶ Watch: AppSec challenges with rapid DevOps deployments. (7:00)

The central finding of O'Shea Bowens' talk is that the traditional, siloed approach to security, where the Security Operations Center (SOC) operates distinctly from Application Security (AppSec) and DevOps, is no longer sustainable. This operational separation creates a "disalignment" that leaves organizations vulnerable, as evidenced by the increasing number of breaches attributed to application-layer weaknesses and misconfigurations in development pipelines.

Bowens argues that to achieve a state of Nash Equilibrium—where both security and development teams operate optimally without incentive for unilateral change—the SOC must fundamentally shift its engagement model. This shift involves "pushing left," meaning the SOC needs to proactively integrate its expertise and processes earlier into the Software Development Life Cycle (SDLC). Instead of merely reacting to incidents in production, SOC personnel must gain a deep understanding of how applications are built, deployed, and managed.

The key contributions and findings from his perspective for the SOC are:

1. Expanded Scope of Knowledge: SOC analysts must learn the tools, frameworks, and methodologies prevalent in AppSec and DevOps, such as Static Application Security Testing (SAST), Dynamic Application Security Testing (DAST), CI/CD pipelines, containerization (Docker), and various deployment mechanisms.
2. Proactive Engagement: Instead of being solely reactive, the SOC needs to apply its defensive mindset (monitoring, detection, response) to the development and pre-production phases. This means identifying vulnerabilities and misconfigurations before they reach production.
3. Collaborative Defense: The emphasis is on breaking down organizational silos. SOC teams should actively engage with development and AppSec teams, fostering a "purple team" mentality where offensive and defensive insights are shared to build more robust security controls.
4. Risk-Driven Approach: Security efforts, even those "pushed left," must be tied back to organizational risk. This involves understanding how vulnerabilities in code or pipelines translate into business impact and communicating this effectively to management.

In essence, Bowens' key finding is that the SOC's mission to defend the organization necessitates an evolution beyond traditional network and endpoint security. It requires an embrace of the application and infrastructure-as-code paradigms, transforming the SOC into a more comprehensive, proactive, and integrated security function.

Technical Deep Dive

▶ Watch: Demo: Exploiting a vulnerable PHP application. (16:00)

Pushing the SOC left demands a significant technical expansion for security operations personnel. Bowens delineates several key areas where SOC teams must develop expertise, moving beyond traditional network and endpoint security.

Firstly, understanding Application Security (AppSec) tools is paramount. He categorizes these into two main types:

• Static Application Security Testing (SAST): Tools that analyze source code, bytecode, or binary code to identify potential vulnerabilities without executing the application. These run "line by line" within the code itself.
• Dynamic Application Security Testing (DAST): Tools that test a running application, typically from the outside, to find vulnerabilities that might not be apparent from static code analysis.

While Bowens remains vendor-agnostic, he encourages SOC teams to explore available SAST/DAST tools, emphasizing that the initial step is simply to "grab it and try it." Complementing these tools, Threat Modeling and Penetration Testing are highlighted as foundational practices that SOC teams are already familiar with in other contexts and should apply to applications. Threat modeling helps identify "what are we concerned about" and "what are we going to do about it," while pen testing provides the offensive perspective crucial for understanding defensive needs.

The core of modern application delivery is the DevOps pipeline, specifically Continuous Integration/Continuous Delivery (CI/CD). SOC analysts need to comprehend the components of these pipelines: source control management, build automation tools, web application servers, and code testing coverage. Bowens references various deployment tools like Jenkins, Ansible, Packer, and cloud-native options such as Azure tools and AWS Lambda. He expresses a particular interest in Lambda for its flexibility, especially when paired with Python, and the potential to inject security agents (like a Bro/Zeek agent) as dependencies within these builds.

A critical concept for securing these pipelines is the implementation of Golden Images and Golden Pipelines. A Golden Image refers to a hardened, standardized configuration of an operating system or application environment, including specific dependencies, that doesn't change frequently. This mirrors the traditional practice of hardening servers but applied to the ephemeral nature of cloud and container environments. Similarly, Golden Pipelines establish controlled processes for deployments, limiting who can make changes via pull requests and ensuring robust auditing and access controls. This reduces the attack surface by standardizing and securing the build artifacts and deployment mechanisms.

Bowens also stresses the importance of understanding Microcontainers and Docker. He warns that threat actors are increasingly targeting vulnerable Docker instances, often found via tools like Shodan, due to default configurations or credentials. He cites the example of the Maze ransomware leveraging headless VMs within containers, highlighting how an incident responder might not "innately go look" in these areas without prior knowledge. The ability to detect malicious images or container escapes becomes a new frontier for SOCs.

Finally, integrating traditional SOC pillars—monitoring, detection, and response—into this new landscape requires a fresh approach. Bowens suggests leveraging tools like Osquery and Sysmon (especially with configurations like those from Swifton Security) for endpoint visibility within the development ecosystem. He also emphasizes aligning defensive efforts with frameworks like MITRE ATT&CK. With the release of sub-techniques and tools like Roberto Rodriguez's HK, ATT&CK provides a structured way to build countermeasures, link them to risk registers, and communicate their value to management, converting technical findings into a language of business risk. This comprehensive technical understanding empowers the SOC to be truly proactive.

Demo / Proof of Concept

▶ Watch: Leveraging "golden images" and "golden pipelines" for security. (27:00)

O'Shea Bowens illustrates the practical application of "pushing left" through a simulated attack scenario against a vulnerable web application. This demonstration serves as a proof of concept for how SOC analysts can adopt an offensive mindset to better inform their defensive strategies.

The target for the demonstration is a "super small, super vulnerable" restaurant review application written in PHP. This choice reflects common legacy or poorly developed applications that might still exist in enterprise environments. The scenario unfolds as follows:

1. Initial Reconnaissance and Scanning: The first step, familiar to any security professional, involves scanning the application. Instead of traditional network scanners like Nmap, Bowens emphasizes the use of SAST and DAST tools to identify potential vulnerabilities within the code and the running application. The output from these scans provides the initial "report" of what's wrong.
2. Threat Model Re-evaluation: With a list of identified vulnerabilities, the attacker (and by extension, the SOC adopting a purple team approach) revisits the threat model. Questions are asked: What vulnerabilities do we care about? How can they be fixed? How long might these vulnerabilities persist in the environment?
3. Exploitation Phase: Putting on the "red team hat," the demonstration proceeds to exploit the identified weaknesses. In this specific case, Bowens mentions exploiting PHP vulnerabilities, including a Local File Inclusion (LFI) vulnerability. The goal is to inject simple commands via the URL bar to verify command execution on the target machine. This confirms the vulnerability is exploitable and provides a "sign number one" of success.
4. Post-Exploitation and Data Exfiltration: Once initial access is confirmed, the attacker's objective shifts. A secondary payload is dropped, designed to "beacon out" to a command-and-control (C2) server within the attacker's environment. Upon establishing this connection, further actions are taken:

• Dropping another payload, such as a keylogger.
• Navigating to the application's database.
• Zipping up the sensitive data from the database.
• Preparing for data exfiltration—the ultimate goal for many professional attackers, who aim to collect and sell data rather than simply destroy the target.

This simulated attack flow demonstrates how a SOC analyst, by understanding the attacker's perspective and the lifecycle of an exploit in an application context, can identify critical points for monitoring, detection, and response. It highlights that the core principles of vulnerability discovery and exploitation remain consistent, even if the tools and environment (application code vs. network device) differ. The exercise underscores the value of hands-on experience with vulnerable applications (many of which are available online) to build a robust defensive posture.

Defensive Implications

▶ Watch: Q&A: Quickest win for pushing left in AppSec. (37:00)

The core message for defenders is clear: the SOC must proactively engage with the application and development lifecycle. This means a fundamental shift in mindset and skill development.

First, SOC personnel must learn the tools and processes of DevOps and AppSec. This includes familiarizing themselves with GitHub for source control, Docker for containerization, and various deployment tools like Lambda, Jenkins, and Ansible. Bowens suggests that if direct hands-on experience is difficult, leveraging resources like CIS Benchmarks for Docker and Jenkins can provide valuable insights into hardening these critical components. The goal is to understand "what the heck is going on with the devops folks" and "how things are deployed within your environment."

Second, organizations should adjust their SOC Charter to incorporate AppSec practices. This formalizes the "pushing left" approach, ensuring that security operations are involved earlier. This involves asking critical questions about pipelines and images during security assessments, focusing on concepts like Golden Images and Golden Pipelines to standardize and secure deployment artifacts and processes. Implementing auditing and strict access controls around pipeline changes is crucial.

Third, overcommunication and collaboration are non-negotiable. Bowens emphasizes the importance of interpersonal communication, sitting down with development and AppSec teams, and understanding their day-to-day challenges. He advocates for a "purple team" approach, where offensive and defensive insights are shared to build more effective countermeasures. This collaborative spirit helps bridge the traditional "that's not my problem" mentality.

Fourth, Bowens identifies vulnerability and dependency management as the "quickest win" for organizations starting their "push left" journey. Integrating tools (like those from Palo Alto or other container security vendors) that scan build dependencies for known CVEs (e.g., CVE 2020-1472) allows teams to identify and prioritize critical risks associated with specific application components, especially those involved in sensitive functions like user registration.

Fifth, the concept of "breaking the build" is introduced as a powerful defensive control. If a golden image or pipeline introduces a vulnerability or falls out of compliance, the deployment should be halted. However, Bowens cautions that this is a "hard thing to sell" to development teams and requires careful communication and a collaborative approach to avoid disrupting productivity.

Finally, all defensive efforts must be framed in terms of risk for management. It's not enough to find and fix vulnerabilities; the SOC must articulate how these efforts reduce organizational risk, align with MITRE ATT&CK for structured countermeasure development, and contribute to the company's overall security posture. This ensures that security initiatives receive the necessary support and resources.

Key Takeaways

• Embrace "Pushing Left": The SOC must proactively integrate into the AppSec and DevOps lifecycle, moving beyond reactive monitoring to understand and secure applications from their inception.
• Learn the DevOps & AppSec Ecosystem: SOC analysts need to acquire knowledge of tools like GitHub, Docker, Jenkins, Lambda, SAST, and DAST to effectively monitor and defend modern application pipelines.
• Prioritize Vulnerability & Dependency Management: A quick win for "pushing left" is to implement scanning for known CVEs in application dependencies, allowing for early detection and prioritization of critical risks.
• Foster Overcommunication and Collaboration: Break down silos between SOC, AppSec, and DevOps teams through regular, open communication and purple teaming exercises to build a unified defensive strategy.
• Implement Golden Images and Pipelines: Standardize and harden deployment artifacts and processes through "golden images" and "golden pipelines" with robust auditing and access controls to reduce the attack surface.
• Align Security with Business Risk: Translate technical security findings and defensive measures into terms of business risk for management, demonstrating the value of proactive security investments.

About the Speaker(s)

O'Shea Bowens is the CEO and Founder of Null Hat Security. Based in Boston, he is a prominent figure in the local security community, regularly participating in the Boston Security Meetup and DC617. Bowens also serves on the board of directors for Blacks in Cyber. His extensive professional background is primarily rooted in the defensive security space, where he has built and managed security operations from the ground up, holding positions from analyst to director. This deep experience in blue teaming and incident response forms the foundation of his perspective on the critical need for SOCs to "push left" and engage more proactively with application security and DevOps. He is passionate about sharing his learnings and helping others navigate the evolving landscape of cyber defense.