Going Beyond Box Checking with Annual Pentests

Going Beyond Box Checking with Annual Pentests

Speakers: Sean ‘4dw@r3’ Juroviesky, Senior Security Engineer, SoundCloud

Conference: Blue Team Con Online

YouTube: https://www.youtube.com/watch?v=6i2nxAYdAwA

Overview

In his Blue Team Con Online talk, "Going Beyond Box Checking with Annual Pentests," Sean Juroviesky, a Senior Security Engineer at SoundCloud and an organizer of the BBSE community, challenges the conventional approach to annual penetration tests. Juroviesky argues that for many organizations, these mandated exercises have devolved into a predictable, uninspired routine, often yielding the same findings year after year without leading to meaningful resolution or budget allocation. The core problem lies in a disconnect: security teams know what needs fixing, but struggle to articulate the business impact to executives who control the purse strings.

Juroviesky proposes a paradigm shift, transforming the annual pentest from a mere compliance checkbox into a powerful tool for budget justification and executive buy-in. Instead of broadly searching for new vulnerabilities, he advocates for a targeted, collaborative approach where security teams work directly with pentesters to focus on known, sticky issues. The goal is to leverage the pentester's expertise to thoroughly enumerate the business impact of these specific vulnerabilities, translating technical risks into concrete financial and operational consequences that resonate with executive leadership.

This talk is crucial for any organization grappling with recurring security findings, budget constraints, or a perceived lack of executive understanding regarding cybersecurity investments. Juroviesky's methodology provides a practical framework for security professionals to re-engage with their pentest partners, turning a potentially mundane compliance activity into a strategic initiative that drives real security improvements and secures necessary resources. By reframing the conversation around revenue impact and business continuity, security teams can effectively advocate for the changes they know are essential.

Background

▶ Watch: Introduction and the core problem with pentests (4:00)

The pervasive requirement for annual penetration tests stems from various mandates, including insurance policies, regulatory compliance (such as PCI DSS, HIPAA, GDPR), and contractual obligations with customers. While these requirements are designed to ensure a baseline level of security diligence, their implementation often falls short of their intended purpose. As Juroviesky highlights, a common scenario sees security teams receiving pentest reports that merely reiterate findings from previous years. These persistent issues often remain unaddressed due to a lack of budget, resources, or executive prioritization.

This stagnation leads to a cycle of frustration. Security professionals are aware of critical vulnerabilities but lack the "oomph" to convince leadership to invest in their remediation. Simultaneously, the pentesters themselves become disengaged. They entered the field to discover novel threats and exploit complex systems, but are instead tasked with repeatedly uncovering the "lowest level, lowest effort things" that haven't been resolved. The predictable nature of these engagements diminishes their value for both parties, transforming a potentially insightful exercise into a perfunctory "box-checking" activity.

The fundamental problem lies in the communication gap between technical security teams and business-oriented executives. Technical jargon like "ransomware" or "database breach" often fails to convey the true gravity of the situation to non-technical leaders. Executives are primarily concerned with revenue generation, cost reduction, and market reputation. Without a clear translation of technical vulnerabilities into these business-centric terms, security requests are often perceived as abstract costs rather than essential investments to prevent tangible losses. Juroviesky's approach seeks to bridge this gap by providing a structured method for security teams to leverage their pentests to speak the language of business, thereby securing the necessary attention and resources for critical security initiatives.

Key Findings

▶ Watch: Why annual pentests are often predictable and boring (5:00)

The talk's primary "key finding" is not a specific vulnerability or exploit, but a reimagined methodology for conducting and leveraging annual penetration tests. Juroviesky asserts that the true value of a pentest, especially when mandated, lies in its potential to serve as a powerful tool for internal advocacy, specifically for justifying budget and resources to address known, persistent security issues. This shifts the focus from merely identifying new problems to strategically demonstrating the business impact of existing ones.

The core contributions of this methodology include:

1. Targeted Scoping for Justification: Instead of a broad, unfocused assessment, the pentest should be narrowly scoped to focus on specific "sticky issues" that the security team knows require attention but has struggled to get funded. Pentesters are then tasked with thoroughly enumerating every possible weakness within that targeted system or process.
2. Translation of Technical Risk to Business Impact: The crucial step is converting technical findings into clear, quantifiable business impacts. This means moving beyond terms like "vulnerability" or "exploit" to articulate consequences such as revenue loss, reputational damage, operational disruption, or even potential legal repercussions for executives.
3. Collaboration with Financial Planning & Analysis (FP&A): Juroviesky identifies FP&A departments as invaluable allies. These teams already possess data on service disruptions, hourly revenue generation, and projected losses. They have often "done 90% of the work" required to translate potential security incidents into concrete monetary figures, providing the factual basis for executive-level budget requests.
4. Strategic Reporting for Executive Buy-in: The final report should be tailored for an executive audience, featuring:

• Executive Summaries that focus on business impacts and cost-benefit analyses (cost to fix vs. cost of inaction).
• Diagrams and Chains of Effects that visually represent how a technical vulnerability cascades into widespread business disruption.
• Targeted Language using internal company terminology to resonate with executives familiar with specific teams and functions.
• Inclusion of Strengths: Counterintuitively, Juroviesky advises including positive feedback on the security team or existing controls. This builds credibility, avoids a "doom and gloom" perception, and makes the identified critical flaws stand out more prominently, making executives more receptive to proposed solutions.

1. Understanding Executive Risk Appetite: Acknowledging that executives are "gamblers" who weigh risks against potential revenue gains, the methodology emphasizes presenting a clear disparity between the cost of remediation and the much larger potential cost of a breach. This helps frame security spending as a preventative investment rather than an arbitrary expense.

By implementing these findings, organizations can transform their annual pentest from a compliance burden into a strategic asset, empowering security teams to drive meaningful change and secure the necessary resources for a more robust security posture.

Technical Deep Dive

▶ Watch: Collaborating with pentesters for impactful results (6:45)

While "Going Beyond Box Checking" doesn't delve into specific code exploits or network protocols, it offers a deep dive into the process and communication architecture required to transform a conventional penetration test into a strategic tool for executive advocacy. This technical deep dive focuses on the structured engagement methodology, from pre-engagement planning to post-exploit reporting.

Pre-Engagement Planning: Defining the Strategic Objectives

The foundational element is the pre-engagement phase, where security teams must clearly define their strategic goals. This goes beyond simply "finding vulnerabilities." Examples of specific goals include:

• "Close the risk associated with unpatched legacy system X."
• "Increase staffing levels in the SOC because we have Y alerts and Z staff."
• "Justify budget for implementing MFA across all internal applications."

Once the goal is set, meticulous scoping is paramount. This involves not only identifying which systems are in scope (e.g., a specific database, a new application, an internal network segment) but also explicitly defining what is out of scope and, critically, what systems are fragile and should be handled with extreme caution or avoided entirely to prevent accidental real-world outages. Causing an outage during a pentest can severely undermine the entire initiative.

The core of the deep dive is to target impact by focusing on creating potential revenue-impacting events. The pentesters' objective shifts from mere exploitation to demonstrating the chain of events that would lead to tangible business losses. This requires a deep understanding of the business functions tied to the targeted systems.

Supporting the Goal: The Financial Narrative

A critical aspect of this methodology is supporting the defined goal with a compelling financial narrative. Security teams must ensure that the cost to repair a vulnerability is presented as significantly less than the potential financial loss if the vulnerability is exploited. Juroviesky emphasizes that executives often view security spending as a direct cost, and if the cost to fix is perceived as too close to the potential loss, they may "take the 50% chance it happens over the 100% chance of spending that same amount of money." This requires a clear, quantified cost-benefit analysis.

Transparency and Realistic Emulation

Regarding information sharing with pentesters, a delicate balance is required. Security teams should avoid giving pentesters "complete admin access to everything," as this can provide executives with an excuse to dismiss findings ("they wouldn't have found that without excessive access"). However, pentesters need enough information to reasonably emulate an attacker's capabilities within the limited timeframe and budget of a pentest (e.g., achieving in 3 weeks what a real attacker might take 6 months to discover). This includes providing sufficient OSINT (Open Source Intelligence) and allowing for realistic attack vectors like phishing to obtain credentials, enabling them to pivot and demonstrate the full scope of an attack chain.

Engagement Readiness: Setting the Stage for Success

Before the actual testing begins, several checks ensure readiness:

1. Read-back of Goals and Objectives: The pentesters should articulate their understanding of the engagement's goals and objectives back to the security team, ensuring alignment.
2. Clear Boundaries and Limitations: Reconfirm what systems are strictly off-limits or require extreme caution (e.g., "don't DDoS us, that's not interesting unless it bypasses authentication like the UK Ministry of Defense back button example").
3. Communication Lines: Establish clear and immediate communication channels to ensure that security teams can be reached at all times in case of accidental real-world impact or discovery of new, critical services.
4. Scope Verification: A final double-check to confirm all intended systems are in scope and all excluded systems are out of scope.

Post-Exploit and Reporting: The Business Case

The post-exploit phase is where the technical findings are meticulously translated into business impact.

• Verification: Internal staff must verify all findings to ensure accuracy and prevent "hallucinated vulnerabilities" (a nod to AI-generated content issues).
• Targeted Language: Reports should use the company's internal terminology (e.g., "system traffic team" instead of generic "networking team") to make the impact more relatable and underscore which internal departments are affected.
• Comprehensive Impact Analysis: Beyond primary effects, the report must capture tertiary and secondary effects, diagramming how one compromised service can cascade to affect others (e.g., database down -> CRM down -> marketing emails not sent -> 5% revenue loss from coupon codes).
• Visual Communication: Diagrams are crucial. A "chain of effects" diagram, potentially with a "spectator view" or "customer view," can powerfully convey the impact to executives who might struggle with technical prose. Juroviesky even humorously references Google Gemini's ability to generate infrastructure diagrams.
• Highlighting Strengths: Counterintuitively, including positive aspects in the report ("Your team is fantastic, they are doing so well...") builds credibility and makes the identified critical flaws stand out more sharply, leading to greater executive buy-in.
• Executive Presentation: The final presentation requires practice, minimizing technical jargon, and being prepared to guide the conversation back on track when executives inevitably get "derailed." The goal is to present a clear action plan derived from the pentest findings, framed within the context of risk reduction to an "acceptable level" determined by executive leadership.

This detailed, structured approach to pentests leverages technical expertise not just for discovery, but for strategic communication, ensuring that security efforts are understood, valued, and adequately funded by the business.

Demo / Proof of Concept

▶ Watch: Strategy: Limit scope and enumerate specific issues (7:25)

This talk focuses on a strategic methodology and communication framework rather than a traditional technical demonstration of an exploit or a software tool. Sean Juroviesky does not present a live demo or a proof of concept in the conventional sense of exploiting a vulnerability or showcasing a new security product.

Instead, the "proof of concept" within this talk is the successful application of the described methodology. The efficacy is demonstrated through the potential outcome: securing executive buy-in and budget for critical security initiatives by effectively translating technical risks into quantifiable business impacts. While no specific technical exploit is shown, the talk provides a blueprint for how an organization could theoretically use a penetration test to demonstrate the cascading impact of a database breach on sales, marketing, and delivery teams, ultimately leading to a clear, revenue-impacting figure that compels executive action. The "demo" is the conceptual journey from a technical finding to a justified budget approval, underpinned by strategic communication and collaboration.

Defensive Implications

▶ Watch: Translate technical findings into business impact for executives (9:30)

Sean Juroviesky's talk offers profound defensive implications for Blue Teams and security practitioners seeking to move beyond reactive security measures and secure the necessary resources for proactive defense. The core message empowers defenders to leverage mandated penetration tests as strategic assets rather than mere compliance exercises.

1. Proactive Scoping and Goal Setting: Defenders should actively engage with their pentest providers before the engagement begins. Instead of passively accepting a broad scope, Blue Teams must clearly define specific, impactful goals for the pentest. This means identifying internal "sticky issues" – persistent vulnerabilities or under-resourced areas – and directing the pentesters to thoroughly enumerate the risk associated with these specific targets. This ensures the pentest serves the organization's unique needs, not just generic compliance.
2. Translate Technical to Business Risk: A critical defensive skill is the ability to translate technical findings (e.g., "unauthenticated API endpoint," "SQL injection") into tangible business impacts. Defenders must learn to articulate how these vulnerabilities lead to revenue loss, reputational damage, customer churn, operational downtime, or even legal liabilities. This requires understanding the business processes tied to technical assets.
3. Collaborate with Financial Planning & Analysis (FP&A): Blue Teams should forge strong alliances with their organization's FP&A department. These teams are invaluable for quantifying potential financial losses from security incidents. By leveraging FP&A's existing data on hourly revenue, service disruption costs, and holiday impacts, defenders can quickly transform abstract risks into concrete monetary figures, providing a powerful argument for executive budget allocation.
4. Strategic Report Crafting and Presentation:

• Targeted Language: Reports to executives should use internal company terminology and phrasing that resonates with non-technical leaders.
• Visual Impact: Utilize diagrams, especially "chains of effects" and "customer view" diagrams, to visually communicate the cascading impact of a security incident across different business units. Visuals often convey more effectively than dense technical prose.
• Cost-Benefit Analysis: Always present remediation costs alongside the much larger potential costs of inaction. This frames security spending as an investment to prevent greater financial harm.
• Highlight Strengths: Counterintuitively, including positive feedback on the security team's existing controls or progress in the report builds credibility. This prevents the "doom and gloom" perception and makes the identified critical flaws stand out, increasing executive receptiveness.

1. Establish Clear Communication Channels: During the pentest, maintain open and immediate lines of communication with the pentesters. This ensures rapid response if an accidental real-world impact occurs or if a critical, unknown vulnerability is discovered, allowing for swift remediation.
2. Understand Executive Risk Appetite: Defenders must recognize that executives are ultimately responsible for determining the acceptable level of risk for the organization. While security teams aim for maximum protection, executives weigh security investments against other business priorities. Understanding this dynamic helps defenders tailor their proposals and, if necessary, identify when a company's risk tolerance might exceed acceptable professional standards, prompting a re-evaluation of one's own career path.

By adopting this strategic approach, Blue Teams can transform the annual pentest from a passive audit into a proactive force for meaningful security improvement, securing the budget and executive support needed to build a robust defensive posture.

Key Takeaways

• Refocus Pentests for Business Impact: Shift the primary goal of annual penetration tests from merely identifying new vulnerabilities to strategically justifying budget and resources for known, persistent security issues.
• Collaborate with Pentesters: Work closely with pentesters to narrowly scope engagements around specific, high-priority issues, directing them to enumerate the full business impact rather than just technical findings.
• Translate Technical Risk to Financial Terms: Convert security vulnerabilities and exploits into quantifiable business consequences such as revenue loss, operational disruption, and reputational damage to resonate with executive leadership.
• Leverage FP&A for Monetary Impact: Partner with your organization's Financial Planning & Analysis (FP&A) department to obtain accurate monetary figures for potential losses, as they already possess data on service disruptions and revenue impacts.
• Craft Strategic Executive Reports: Design reports with executive summaries, clear cost-benefit analyses, visual diagrams (chains of effects, customer views), and use internal company language. Crucially, include positive feedback on existing security strengths to build credibility.
• Understand Executive Risk Appetite: Recognize that executives are "gamblers" who weigh risks against potential gains. Frame security spending as an investment to prevent larger, quantifiable losses, but also understand when an organization's risk tolerance may exceed acceptable levels.

About the Speaker(s)

Sean Juroviesky, known online as ‘4dw@r3’, is a Senior Security Engineer currently contributing his expertise at SoundCloud, a popular music streaming service. Beyond his corporate role, Sean is deeply involved in the cybersecurity community. He is one of the dedicated organizers of BBSE, a vibrant local cybersecurity group. BBSE hosts regular meetings across seven locations in the Chicagoland area, as well as in Minneapolis, Galway, Ireland, and Las Vegas, providing a platform for professionals to network, share ideas, and foster community within the industry. Sean's passion lies in helping security teams effectively communicate the value and necessity of their work to broader business leadership.