Hackers vs. Testers: A Comparison of Software Vulnerability Discovery Processes
Daniel Votipka (PhD candidate · University of Maryland)
Swiss Cyber Storm 2019 · Day 1 · Management
Overview
In this insightful talk from Swiss Cyber Storm, Daniel Votipka, a PhD candidate at the University of Maryland, presented his research on the distinct, yet often overlapping, approaches taken by professional software testers and security hackers when searching for vulnerabilities. The core premise of the research is to demystify the "adversarial mindset" – a concept frequently championed in cybersecurity but rarely understood in practical terms – by comparing how security experts (hackers) and non-security experts (testers) identify flaws in software.

Key moments
- 0:40 Why traditional testers struggle finding vulnerabilities
- 3:45 Benefits of bug bounties and hiring hackers
- 6:00 Challenges in engaging hackers: timeliness, diversity, communication
- 9:40 The 6-step common vulnerability discovery process
- 12:10 Four factors differentiating hacker and tester processes
- 16:00 How hackers gain superior vulnerability discovery experience
- 20:10 Recommendations: Increase developer security exposure in context
- 22:00 Recommendations: Consistent communication with external hackers
Hackers vs. Testers: A Comparison of Software Vulnerability Discovery Processes
Speakers: Daniel Votipka, PhD candidate, University of Maryland
Conference: Swiss Cyber Storm
YouTube: https://www.youtube.com/watch?v=KZ2nZcGpWO4
Overview
In this insightful talk from Swiss Cyber Storm, Daniel Votipka, a PhD candidate at the University of Maryland, presented his research on the distinct, yet often overlapping, approaches taken by professional software testers and security hackers when searching for vulnerabilities. The core premise of the research is to demystify the "adversarial mindset" – a concept frequently championed in cybersecurity but rarely understood in practical terms – by comparing how security experts (hackers) and non-security experts (testers) identify flaws in software.
The talk highlights a critical challenge in software development: ensuring code is free of exploitable vulnerabilities before it reaches the market. Traditional methods, such as relying on developers or generalist testers, often fall short due to a lack of specialized security expertise. Votipka’s work delves into the processes, experiences, and environments that shape vulnerability discovery, aiming to extract actionable insights that can enhance defensive strategies and foster better collaboration between development teams and security researchers.
By dissecting the methodologies of both groups, Votipka and his colleagues sought to identify fundamental differences that contribute to effective vulnerability discovery. The findings offer valuable guidance for organizations looking to integrate security more effectively into their software development lifecycle, whether by cultivating an adversarial mindset internally or by improving engagement with external security researchers.
Background
▶ Watch: Why traditional testers struggle finding vulnerabilities (0:40)
The pervasive challenge of software security stems from the inherent difficulty in writing flawless code and subsequently identifying its weaknesses. Prior research underscores this difficulty: a study by Edmondson et al. found that even 30 developers examining a web application with introduced vulnerabilities missed several basic issues like cross-site scripting and SQL injection. Similarly, Olivera et al. observed that hundreds of developers struggled to find vulnerabilities in small code snippets, particularly concerning security-related APIs. Even professional developers undergoing months of security training, as observed in Votipka’s own prior work, continued to introduce errors in medium-sized software projects. These findings collectively demonstrate that relying solely on generalist developers and testers, even with some security training, is often insufficient for robust vulnerability detection.
This reality has led to the widespread adoption of an adversarial mindset – approaching software with the intent of an attacker. One common manifestation of this is engaging professional pen testers or leveraging bug bounty programs. These programs offer several benefits: they facilitate responsible disclosure, provide an ethical framework for vulnerability discovery, and, crucially, bring a vast number of "extra eyeballs" to the problem. Research by Mailer et al. on the HackerOne platform suggested that each security researcher offers unique skillsets and mindsets, making diverse contributions vital. Finister et al.'s study on Firefox and Chrome bug reports further validated this, showing that 27% of Chrome bugs and 24% of Firefox bugs originated from bug bounties in their early stages, with external contributions eventually rivaling internal ones. Remarkably, the cost of running these bug bounty programs was found to be comparable to the salary of a single browser security team member, highlighting their cost-effectiveness in attracting diverse expertise.
Despite these benefits, significant challenges persist. Timeliness remains an issue, as security reviews often occur late in the development cycle, leading to costly rework. Cognitive diversity, while a goal of bug bounties, is often limited in practice; contributions frequently come from a small, non-diverse group of top researchers. Finally, communication barriers are prevalent. Hackers often struggle to convey the criticality of their findings to developers, leading to frustration, while companies frequently report receiving false positives or poorly articulated reports. It is within this context of both the promise and the pitfalls of adversarial testing that Votipka's research sought to understand the fundamental differences between internal testers and external hackers to bridge these gaps.
Key Findings
▶ Watch: Challenges in engaging hackers: timeliness, diversity, communication (6:00)
To systematically understand the nuances of vulnerability discovery, Daniel Votipka and his team employed a rigorous qualitative research methodology. They conducted task analysis interviews, a method borrowed from psychology, with participants. This involved asking individuals to describe their process step-by-step, breaking down complex actions into smaller components to uncover even rote or intuitive behaviors. Alongside this, they inquired about tools, skills, and community involvement. The recruitment process was extensive, involving outreach to 106 organizations, ultimately yielding 10 professional software testers and 15 security hackers for hour-long interviews.
The research yielded several key findings:
- Similar Discovery Process: Surprisingly, both testers and hackers followed a fundamentally similar six-step process for vulnerability discovery:
- Gather Context: Understand the program, identify libraries, review update history, and look at previous bug reports. As one tester noted, "bugs tend to cluster," guiding their focus.
- Understand Program: Build a mental map of how the program works, its components, and interactions with the environment (network, file system, memory). A hacker described this as "touching a little bit of everything, organizing it to structure in your head."
- Identify Attack Surface: Pinpoint all direct and indirect input points to the system.
- Explore Attack Surface: Systematically test identified inputs through fuzzing, manual testing, or code/binary review. This step often loops back to "Understand Program" as new insights emerge.
- Recognize Vulnerability: Identify a problem, often described as an "intuition-based" recognition derived from prior experience.
- Report Vulnerability: Document the issue, emphasizing its importance and advocating for remediation. Both groups stressed the need to "convince someone that there's a risk," highlighting the communication challenge.
- Four Influential Factors: While the process was similar, four factors significantly impacted how it was carried out:
- Vulnerability Discovery Experience: The quantity and variety of vulnerabilities previously encountered.
- Underlying System Knowledge: Proficiency in programming languages, APIs, and memory management.
- Access to Development Process: Involvement in design, communication with developers, and access to documentation.
- Motivation: Whether the goal was comprehensive security or specific rewards (e.g., bug bounties).
- Impact on Success vs. Process: These four factors grouped into two categories based on their effect:
- Factors influencing reported success rate: Vulnerability discovery experience and underlying system knowledge directly correlated with a higher success rate in finding vulnerabilities.
- Factors influencing process execution (but not necessarily success rate): Access to the development process and motivation affected how the discovery process was performed, but not necessarily the ultimate success in finding bugs.
- Key Differentiators: The most significant differences between hackers and testers were found in their vulnerability discovery experience and their access to the development process. These two factors became the primary focus for understanding the unique strengths and weaknesses of each group.
Technical Deep Dive
▶ Watch: Four factors differentiating hacker and tester processes (12:10)
The core of Votipka's findings lies in the detailed breakdown of how vulnerability discovery experience and access to the development process critically differentiate hackers from testers. This distinction fundamentally shapes their approach to identifying and reporting security flaws.
Vulnerability Discovery Experience: The Breadth and Depth of Knowledge
Hackers, by their nature, possess a significantly broader and deeper well of vulnerability discovery experience compared to traditional software testers. This experience manifests in three crucial phases of the discovery process:
- Attack Surface Identification (Considering Indirect Inputs): More experienced individuals, particularly hackers, are far more likely to consider indirect inputs – data flows that aren't immediately obvious but can still influence the system. Votipka provided a compelling example from a tester who, after accidentally discovering an SQL injection vulnerability via an apostrophe in LinkedIn profile data being imported into their database, subsequently made it a point to test Facebook and Twitter inputs for similar issues. This "happenstance" learning transformed their approach, demonstrating how a single exposure to an indirect path broadened their attack surface considerations for future projects. Hackers, routinely exposed to a vast array of such scenarios, inherently incorporate these indirect vectors into their initial threat modeling.
- Exploration (Informed Test Case Selection): Experience directly informs the selection of test cases during the exploration phase. Hackers, having seen various vulnerability patterns, can craft more targeted and effective inputs, whether through manual testing or fuzzing. Their mental models of common weaknesses guide them to specific data structures, edge cases, or protocol deviations that are historically prone to bugs. Testers, with less exposure, might rely on more generic or functional test cases, inadvertently overlooking security-critical inputs.
- Vulnerability Recognition (Intuition and Checklists): The ability to "recognize a vulnerability" is often described as intuitive, but this intuition is built on a foundation of prior experience. Both hackers and testers reported having mental (and sometimes physical) checklists of vulnerabilities they look for in different system types. This mapping of observed behavior to known vulnerability patterns is a direct consequence of exposure. A hacker's mental database of past exploits allows them to quickly identify familiar "smells" or anomalous behaviors that signal a potential flaw.
The source of this experience further clarifies the disparity:
- Employment: Testers typically focus on one or a few systems within their company, leading to deep but narrow expertise. Hackers, conversely, inspect a wide variety of systems across numerous organizations, fostering broad, diverse experience.
- Hacking Exercises: Hackers actively participate in dozens of Capture The Flag (CTF) competitions and war games, learning about known vulnerabilities in controlled, gamified environments. Testers might attend one or two company-mandated trainings.
- Community: Hackers are deeply embedded in vibrant security communities, constantly sharing experiences and learning from peers. Testers often have limited security-focused networks.
- Bug Reports: Hackers extensively analyze public bug reports from platforms like HackerOne or major projects like Firefox and Chrome. Testers are usually limited to internal company bug reports, which may offer fewer examples of diverse vulnerabilities.
This "wide gap of experience and variety" is the most significant differentiating factor, equipping hackers with a richer pattern-matching capability for vulnerability discovery.
Access to Development Process: A Double-Edged Sword
The second major differentiator is access to the development process, where testers generally have an advantage, though it comes with caveats.
- Time-Saving vs. Black Box: Testers, being internal, typically have direct communication channels with developers and access to comprehensive documentation. This saves them significant time and effort that hackers, operating in a black box or gray box scenario, must spend on reverse engineering to understand system functionality. While this access streamlines initial understanding, it can also be a "double-edged sword."
- Cognitive Bias (The "Hard to Ignore" Problem): As one tester articulated, "it's hard to ignore certain details once you know about certain areas already." Access to internal knowledge can inadvertently create cognitive bias, making it difficult to step back and view the system from a truly external, adversarial perspective. Hackers, by virtue of their limited access, are forced to think outside the box, often uncovering vulnerabilities that internal teams, too close to the code, might overlook. This highlights the importance of maintaining both internal and external perspectives.
- Reporting and Rapport: Testers benefit from established rapport with developers. They can engage in ongoing dialogue, provide feedback, and build relationships that facilitate smoother vulnerability remediation. One tester noted, "you can get feedback to your developers, you're coming back with information and then they react on it." Hackers, in contrast, often have a one-time interaction with a company. They frequently need to invest significant time in building proof-of-concept exploits to convince developers of an issue's validity, and even then, face resistance or dismissal. This struggle with communication and trust underscores a major friction point in external bug bounty programs. However, experienced hackers who report multiple times to the same company can eventually build this rapport, as exemplified by a participant who reported Android formware problems and no longer needed to write exploits, simply pointing to code lines for quick fixes.
In essence, while internal access provides efficiency, it can hinder the adoption of a truly adversarial mindset. The hacker's external perspective, born from necessity, often leads to novel findings, even if the reporting process is more arduous.
Demo / Proof of Concept
▶ Watch: How hackers gain superior vulnerability discovery experience (16:00)
This talk presented research findings rather than a live demonstration or proof of concept of a specific exploit or tool. Instead, the "proof" lies in the qualitative data gathered from interviews with security experts and testers, illuminating their distinct approaches to vulnerability discovery. The insights derived from this comparative analysis serve as a foundational "proof" for understanding the cognitive and procedural differences between these two groups.
Defensive Implications
▶ Watch: Recommendations: Consistent communication with external hackers (22:00)
The research offers concrete, actionable advice for organizations aiming to strengthen their software security posture by leveraging the insights gleaned from both hackers and testers.
Empowering Developers and Testers with Security Exposure
The primary recommendation for internal teams is to increase security exposure in context, minimizing additional workload. This means integrating security learning directly into existing workflows:
- Hire Hackers into Testing Teams: The most compelling defensive implication is the strategic integration of security experts into traditional testing teams. Votipka shared a success story where a company, after a pen test revealed numerous vulnerabilities, hired two mid-level hackers. These hackers worked alongside testers, recognizing potential issues and guiding the team to investigate. Over two years, this collaborative approach fostered a "security culture" within the company. When the same pen testing team returned, they found significantly fewer vulnerabilities, demonstrating the transformative power of embedding adversarial expertise. This approach builds internal capacity and transfers tacit knowledge effectively.
- Utilize Bug Report-Based Exercises: To provide hands-on, context-relevant security training, companies should create exercises based on their own internal bug reports. This leverages existing, familiar codebases and vulnerability types, making the learning directly applicable. Votipka mentioned Dropbox's internal capture-the-flag (CTF) competition as an example, where developers engage with real-world scenarios. Early results from studies around such initiatives indicate that they significantly improve developers' understanding of security and help them identify where to focus security efforts. This method provides the "hacking exercise" experience that external researchers gain, but within the secure and relevant context of the company's own software.
Fostering Better Communication with External Hackers
For organizations engaging with external security researchers through bug bounty programs or responsible disclosure initiatives, improving communication is paramount:
- Consistent Communication with a Single Point of Contact: A major pain point for hackers is inconsistent or unclear communication. Establishing a single point of contact for all vulnerability reports can drastically improve the experience. This contact person, ideally someone familiar with both developer terminology and hacker language, can act as an effective translator and build rapport over time. This consistency reduces frustration for hackers and ensures that valuable vulnerability reports are understood and prioritized by developers. The example of the Android formware reporter who built a relationship and no longer needed to provide elaborate exploits illustrates the power of sustained, consistent interaction.
By implementing these strategies, organizations can bridge the gap between traditional testing and adversarial security research, creating a more robust and responsive security ecosystem.
Key Takeaways
- Similar Process, Different Execution: Both software testers and security hackers follow a remarkably similar six-step process for vulnerability discovery, but the way they execute these steps differs significantly.
- Experience is Paramount: Vulnerability discovery experience and underlying system knowledge are critical determinants of success in finding bugs. Hackers possess a broader and deeper range of this experience due to diverse exposure.
- Access is a Double-Edged Sword: While access to internal documentation and developers saves time for testers, it can also introduce cognitive biases, making it harder to adopt a truly adversarial, external perspective.
- Hackers' Unique Value: Hackers excel at considering indirect inputs, crafting informed test cases, and intuitively recognizing vulnerabilities due to their extensive and varied experience, often gained through employment across diverse systems, CTFs, community engagement, and public bug report analysis.
- Integrate Adversarial Expertise: Companies should consider embedding security experts (hackers) directly into their testing and development teams to foster a security culture and transfer critical knowledge in context.
- Improve Hacker Engagement: Consistent communication through a single point of contact is vital for building trust and rapport with external security researchers, leading to more effective and efficient vulnerability remediation.
About the Speaker(s)
Daniel Votipka is a PhD candidate at the University of Maryland, where his research focuses on understanding how security experts and non-security experts approach the discovery of software vulnerabilities. His work aims to extract practical insights from these comparisons to improve software defenses and foster better collaboration within the security ecosystem. Votipka's ongoing research involves studying the processes, tools, skills, and communities that shape vulnerability discovery, often inviting participation from industry professionals to gather real-world data. He is actively involved in exploring methods to expose developers and testers to security in context, such as through bug report-based exercises and internal CTF competitions.
Reviews
Dr. Zero (Offensive Security Researcher) — SOLID
Votipka did the academic legwork to prove what offensive operators have known for twenty years: QA testers look for regressions, while hackers look for weird machines. It's a usable-security study that formalizes the gap between functional testing and adversarial threat modeling. Solid academic validation for managers trying to justify security headcount, but it won't change how you actually audit a codebase.
Heather Calloway (CISO) — STRONG ACCEPT
Votipka puts empirical data behind a problem every CISO knows: standard QA teams struggle to find security bugs, and external researchers struggle to talk to developers. It is a clear, data-driven look at why we fail to staff product security effectively. Required viewing for whoever runs your AppSec or bug bounty programs.