SusParams: Hypercharge your web testing with DATA

Jason Haddix (CISO and Hacker in Charge · BuddoBot), Gunnar Andrews

AppSec Village @ DEF CON 31 · Day 2 · AppSec Village

Overview

In the dynamic and often opaque world of application security testing, experienced hackers frequently rely on an intuitive "spidey sense" to pinpoint vulnerabilities. This intuition, however, is a product of years of experience and contextual understanding, making it notoriously difficult for newcomers to acquire. The talk "SusParams: Hypercharge your web testing with DATA" by Jason Haddix and Gunnar Andrews directly addresses this challenge, presenting a novel, data-driven approach to democratize web hacking knowledge and accelerate the learning curve for security professionals.

Watch on YouTube

Visual summary for SusParams: Hypercharge your web testing with DATA by Jason Haddix, Gunnar Andrews
Visual summary for SusParams: Hypercharge your web testing with DATA by Jason Haddix, Gunnar Andrews

Key moments

  1. 0:00 Introduction and problem: 'Spidey Sense' of hacking
  2. 2:50 Original 'Hunt' research and data methodology
  3. 5:45 SusParams: New research leveraging HackerOne data with AI
  4. 8:00 AI parsing methodology: GPT-4 for reduction, regex for extraction
  5. 10:45 Burp Suite integration with Gap extension for flagging
  6. 19:00 Key vulnerable parameters by type (File Inclusion, IDOR, SQLi, XSS)
  7. 23:00 Future data sources and project expansion plans
  8. 29:45 Q&A: Industry-specific parameter categorization feasibility

SusParams: Hypercharge your web testing with DATA

Speakers: Jason Haddix (CISO and Hacker in Charge, BuddoBot); Gunnar Andrews

Conference: AppSec Village @ DEF CON 31

YouTube: https://www.youtube.com/watch?v=zInDWK43yQ8

Overview

In the dynamic and often opaque world of application security testing, experienced hackers frequently rely on an intuitive "spidey sense" to pinpoint vulnerabilities. This intuition, however, is a product of years of experience and contextual understanding, making it notoriously difficult for newcomers to acquire. The talk "SusParams: Hypercharge your web testing with DATA" by Jason Haddix and Gunnar Andrews directly addresses this challenge, presenting a novel, data-driven approach to democratize web hacking knowledge and accelerate the learning curve for security professionals.

The core of their research and tooling, dubbed SusParams, revolves around identifying and categorizing web parameters that are statistically most likely to be vulnerable to common web application flaws. Building upon Haddix's foundational "Hunt" research from DEF CON 26, this presentation introduces significantly enhanced data sets, leverages cutting-edge AI for automated parsing, and offers practical integrations to empower both seasoned and aspiring security testers. By converting years of collective vulnerability discovery into actionable lists, SusParams aims to provide a tangible roadmap for efficient and effective web application assessments, moving beyond anecdotal experience to concrete, data-backed guidance.

This work matters deeply to the cybersecurity community as it tackles a fundamental problem in talent development and testing efficiency. In an industry grappling with a severe skills gap, providing tools and methodologies that shortcut the arduous path to expertise is invaluable. SusParams offers a systematic way to prioritize testing efforts, ensuring that critical parameters, statistically proven to harbor vulnerabilities like SQL injection, cross-site scripting, and IDOR, receive the attention they deserve. This not only streamlines the testing process but also fosters a more analytical and evidence-based approach to application security.

Background

▶ Watch: Introduction and problem: 'Spidey Sense' of hacking (0:00)

The genesis of SusParams lies in Jason Haddix’s earlier research presented at DEF CON 26, titled "Hunt: Data Driven Web Hacking and Manual Testing." That initial endeavor aimed to quantify the elusive "spidey sense" that seasoned application security hackers develop over years – the ability to instinctively know where to look for vulnerabilities on a given website. This intuition, while powerful, is deeply rooted in individual experience and context, making it a difficult skill to transfer or teach to new practitioners. Haddix noted that simply telling new hackers, "you'll develop this skill set in 15 years," was unhelpful and unsustainable.

The core problem identified was the lack of structured guidance for identifying high-priority testing areas. While a URL consists of a protocol, subdomain, domain, resource path, and crucially, parameters and their values, the latter were pinpointed as the key to unlocking contextual insights. Parameters often dictate how an application processes data, making them prime candidates for various injection and logic flaws.

For the original "Hunt" research, Haddix, then working for Bugcrowd, leveraged an unprecedented dataset: anonymized URL data from over 600 bug bounty programs. This massive collection, comprising approximately 18,000 parameters associated with identified vulnerabilities, was meticulously processed. The data engineers at Bugcrowd stripped domain information to ensure anonymity, and Haddix parsed out parameters linked to specific bug classes from P1 (critical) and P2 (high severity) vulnerabilities. This raw data was then reduced, sorted by recurring instances, and manually reviewed by a team of five experienced hackers. Their collective expertise allowed for the inclusion of common permutations and the enrichment of the lists with data from public fuzzing repositories like PentestDB, FuzzDB, and SecLists (which Haddix co-runs with Daniel Miessler). The output of this painstaking process was a set of lists detailing parameters frequently found vulnerable to specific attack types, such as next for redirects or id for IDOR and SQL injection. These lists became a foundational resource, informally shared among security professionals as a guide for targeted testing.

Key Findings

▶ Watch: SusParams: New research leveraging HackerOne data with AI (5:45)

The "SusParams" project represents a significant evolution and expansion of the original "Hunt" research, bringing renewed focus, enhanced data, and innovative tooling to the forefront. The key findings and contributions of this latest iteration are multi-faceted:

  1. Expanded Data Sources: While the original "Hunt" project relied on Bugcrowd data, SusParams significantly augments this by incorporating publicly disclosed reports from HackerOne's "Hacktivity" feed. This new dataset comprises over 15,000 public reports, offering a fresh and extensive perspective on contemporary web vulnerabilities and the parameters associated with them. The speakers noted that HackerOne's Hacktivity provided a much larger public data set compared to Bugcrowd's crowdstream for this specific purpose.
  1. AI-Driven Data Parsing: A groundbreaking aspect of SusParams is its successful application of Artificial Intelligence (AI), specifically GPT-4, for parsing unstructured text from bug bounty reports. While many consider AI parsing for structured output challenging, the team demonstrated its effective use. GPT-4 was employed to distill verbose vulnerability descriptions into concise snippets containing potentially vulnerable parameters. This dramatically reduced the manual effort required to extract relevant information from thousands of reports, overcoming the inherent difficulty of obtaining uniform output directly from AI models.
  1. Release of Updated Lists and Tooling: The project culminates in the public release of the SusParams lists, which combine the insights from both the original Bugcrowd data and the new HackerOne analysis. These lists are available in various formats, including GF patterns for integration with command-line tools and a dedicated list for a Burp Suite extension. The accompanying scripts and methodology are also open-sourced, encouraging other organizations to generate their own bespoke "sus param" lists.
  1. Integration with Burp Suite's Gap Extension: A major practical enhancement is the direct integration of SusParams lists into xnl_hacker's Gap Burp Suite extension. This integration allows the Gap extension, which automatically collects parameters flowing through the proxy, to flag "sus" parameters in real-time. When a parameter encountered during testing matches an entry in the SusParams lists, Gap creates a Burp issue, effectively providing automated, experience-driven guidance to the tester on potential vulnerability classes for that specific parameter. This acts as an immediate, context-aware prompt for focused manual testing.
  1. Categorized Vulnerable Parameters: The research yielded specific lists of parameters most commonly associated with distinct vulnerability types. For instance:
  • File Inclusion: lang, type, text, domain, name, token, URL, path, file, default, language, directory.
  • Insecure Direct Object Reference (IDOR): id, user_id, username, email, data_id.
  • Redirects: next, redirect, return, host, url.
  • SQL Injection: id, email, refer, limit, search, token, username, password. (A specific "bounty tip" was given to always try SQLi in the refer header).
  • Server-Side Request Forgery (SSRF): url, source, start, host.
  • Server-Side Template Injection (SSTI): While HackerOne data didn't yield strong SSTI parameters, the combined GF patterns still incorporate older "Hunt" data for this category.
  • Cross-Site Scripting (XSS): Specific parameters most likely to be vulnerable to XSS were also identified.

These findings collectively bridge the gap between anecdotal experience and empirical data, providing a powerful resource for enhancing web application security testing.

Technical Deep Dive

▶ Watch: Burp Suite integration with Gap extension for flagging (10:45)

The technical ingenuity of the SusParams project lies in its sophisticated approach to data acquisition, AI-driven parsing, and subsequent analytical processing. The methodology is designed to be both scalable for large public datasets and adaptable for internal, proprietary security data.

Data Acquisition from HackerOne

The initial step involved gathering a substantial dataset of vulnerability reports. The team targeted HackerOne's public "Hacktivity" feed, which contains publicly disclosed bug bounty reports. A critical insight was that Hacktivity is powered by a GraphQL query in its backend. This allowed for programmatic scraping, circumventing the complexities of traditional web scraping dynamic JavaScript-rendered pages. Gunnar Andrews developed a script (hackerone.py) to execute these paginated GraphQL queries, effectively pulling all available report IDs.

Furthermore, each HackerOne report, when accessed via its URL, also provides its content in JSON format by simply appending .json to the URL (e.g., https://hackerone.com/reports/12345.json). This crucial detail meant that the entire content of over 15,000 public reports could be downloaded and saved as structured JSON files, bypassing the need for complex text extraction from HTML. This resulted in a comprehensive vols.json file, serving as the raw input for the next stage. The scripts are designed to allow users to filter by date, enabling incremental updates rather than re-scraping the entire dataset.

AI Parsing with GPT-4

The most innovative technical aspect was the utilization of AI for parsing the unstructured text within these JSON reports. The challenge, as highlighted by the speakers, is that AI models, particularly earlier versions like GPT-3.5 Turbo, struggle to consistently produce uniform, structured output (e.g., perfect JSON). Simply asking the AI to "give me the parameters" from a verbose report often resulted in inconsistent formatting, making automated extraction difficult.

The solution involved leveraging GPT-4 via the OpenAI API. While GPT-3.5 Turbo was attempted, GPT-4 demonstrated significantly superior understanding of HTTP requests, URLs, code, and JSON, leading to much cleaner intermediate output. The parse_poc.py script implemented a chat completion function. For each report's description text, GPT-4 was prompted to identify and extract vulnerable parameters. Crucially, to manage token limits and prevent the AI from "hallucinating" or becoming confused by overly long conversational histories, the chat context was cleared after processing each individual report. This ensured that each parsing operation was a fresh interaction, maximizing consistency and efficiency across 15,000 distinct reports.

The AI's output, while not perfectly uniform JSON, was designed to be highly parseable. It often produced phrases like "the vulnerable parameter is..." followed by the parameter name enclosed in quotes, bulleted lists, or other semi-structured formats. This transformation effectively reduced mountains of free-form text into concise, targeted snippets, and simultaneously filtered out reports where no parameters were identified.

Post-AI Processing and List Generation

Following the AI parsing, the regex_parse.py script took over. This script applied a series of regular expressions (regex) to the AI-generated snippets. The regex patterns were specifically crafted to capture parameter names from the common formats GPT-4 tended to produce—e.g., text within single quotes, double quotes, after colons, or in REST API-like paths (e.g., between backslashes). The speakers noted that ChatGPT itself was helpful in generating these regex patterns. This regex stage achieved an impressive 98% catch rate, effectively extracting the parameter names despite the AI's non-uniform output.

The extracted parameters were then categorized based on the original bug class labels provided by HackerOne (e.g., XSS, SQL injection, IDOR). These categorized lists were then combined with the existing "Hunt" data from the Bugcrowd research. The final output included:

  • Categorized text files: E.g., file_inclusion.txt, idor.txt, sqli.txt, containing the most frequently identified vulnerable parameters for each bug class.
  • GF patterns: These are specific patterns designed for use with the GF tool (Tom Nom Nom's GF), enabling automated grepping for "sus" parameters in raw HTTP traffic or other text files.
  • SusParams list for Burp Gap extension: A consolidated list formatted for direct integration into xnl_hacker's Burp Suite Gap extension, providing real-time alerts during manual testing.

The entire process, from data acquisition to final list generation, is open-sourced in the project's GitHub repository, allowing other researchers and organizations to replicate the methodology or apply it to their own internal datasets. The cost of running the GPT-4 parsing for 15,000 HackerOne reports was a remarkably low ~$30 using a personal OpenAI token, demonstrating the economic viability of this approach.

Demo / Proof of Concept

▶ Watch: Key vulnerable parameters by type (File Inclusion, IDOR, SQLi, XSS) (19:00)

During the talk, Gunnar Andrews provided a comprehensive walkthrough of the SusParams GitHub repository, serving as a live demonstration of the project's tooling and the resulting data. This practical demonstration underscored the accessibility and utility of their work.

The demo began by showcasing the repository's structure, highlighting key files. It first pointed out the GF patterns at the top, emphasizing their utility for integration with automated scanning tools. Next, the original GraphQL query used to scrape HackerOne's Hacktivity was displayed, providing transparency into the data acquisition method.

The hackerone.py script was then introduced as the primary tool for pulling HackerOne data. Gunnar explained that while the script takes time due to the paginated GraphQL queries, it generates a large vols.json file containing all the raw report data. He noted that the repository already includes a pre-pulled vols.json from their last run, but users could easily regenerate it or filter by date for incremental updates.

The core of the AI parsing was demonstrated through the parse_poc.py script. This script, as explained, takes the JSON report data and feeds it to the OpenAI API using GPT-4. The output, stored in a test directory (e.g., xss_test.txt), was then showcased. Gunnar illustrated that while the AI output was not perfectly uniform, it exhibited clear, consistent patterns (e.g., "the vulnerable parameter is" followed by a quoted parameter). This non-uniformity was a deliberate design choice, acknowledging AI's limitations in structured output while still making the data amenable to the next processing step.

The regex_parse.py file was presented next, demonstrating how a series of carefully crafted regular expressions efficiently extracted the parameter names from the AI-generated text. The script included comments explaining each regex pattern, and it was noted that tools like ChatGPT were used to assist in their creation. This stage effectively transformed the semi-structured AI output into clean, actionable lists.

Finally, the result of this entire pipeline was shown: categorized parameter lists (e.g., file_inclusion.txt, idor.txt, sqli.txt). The speakers then projected a screenshot of the Gap Burp Suite extension in action. This screenshot visually demonstrated how the integrated SusParams lists would automatically flag parameters encountered during web traffic (e.g., id) and generate a Burp issue, indicating potential vulnerability classes (e.g., XSS, IDOR, SQLi, SSTI). This real-time flagging serves as a powerful proof of concept for how SusParams directly aids testers in prioritizing their manual efforts, providing immediate, data-driven context for their investigations.

Defensive Implications

▶ Watch: Q&A: Industry-specific parameter categorization feasibility (29:45)

The SusParams project offers significant and actionable implications for defenders, ranging from individual security testers to large organizational security teams and even developers.

  1. Prioritized Testing and Code Review: For security testers, both internal and external, SusParams provides a statistically validated roadmap for prioritizing testing efforts. Instead of exhaustively testing every parameter for every vulnerability, testers can now focus on parameters identified as "sus" for specific bug classes. This allows for more efficient use of time and resources, increasing the likelihood of discovering critical vulnerabilities in high-risk areas. Similarly, development teams can use these lists to prioritize code reviews, scrutinizing the handling of these common "sus" parameters for potential flaws.
  1. Enhanced Static and Dynamic Analysis: The SusParams lists can be integrated into existing security tools to enhance their effectiveness. For instance, SAST (Static Application Security Testing) tools could be configured with custom rules to flag code interacting with these high-risk parameters. DAST (Dynamic Application Security Testing) tools and fuzzers can leverage these lists to generate more targeted payloads for specific parameters, improving their coverage and precision.
  1. Internal Bug Bounty Program Improvement: Organizations running internal bug bounty programs or managing incident response data can adopt the SusParams methodology. By using the open-sourced scripts, they can generate company-specific "sus param" lists tailored to their unique technology stack and historical vulnerabilities. This creates a feedback loop, turning past incidents into proactive defense mechanisms and helping to identify systemic weaknesses within their applications.
  1. Security Training and Onboarding: SusParams provides an invaluable resource for training new security engineers and developers. Instead of relying on abstract concepts, newcomers can be given concrete examples of parameters that historically lead to specific vulnerabilities. This significantly shortens the learning curve, providing practical context and guiding them towards high-impact areas during their initial assessments. It helps to instill that "spidey sense" much faster.
  1. Proactive Threat Modeling: During threat modeling exercises, the SusParams lists can inform discussions around potential attack surfaces. Knowing which parameters are commonly exploited for, say, SSRF or file inclusion, allows teams to proactively design mitigations or implement more stringent validation controls for those specific inputs.
  1. Supply Chain Security: For organizations integrating third-party components or APIs, the SusParams lists can be used to assess the risk of parameters exposed by these external services. If a third-party API uses a parameter known to be "sus" for a particular vulnerability, it warrants additional scrutiny and potentially heightened monitoring.

In essence, SusParams transforms reactive vulnerability discovery into a more proactive, data-informed defense strategy. By understanding where vulnerabilities are statistically most likely to occur, defenders can allocate their resources more effectively, build more resilient applications, and empower their security teams with actionable intelligence.

Key Takeaways

  • Data-Driven Web Hacking: SusParams revolutionizes web application security testing by replacing anecdotal "spidey sense" with statistically validated lists of parameters most likely to be vulnerable, accelerating expertise for both new and experienced testers.
  • AI for Unstructured Data: The project successfully demonstrates how AI, specifically GPT-4, can effectively parse and reduce massive amounts of unstructured security report text into actionable, semi-structured data, even when direct structured output is challenging.
  • Actionable Vulnerability Lists: SusParams provides immediately usable lists of high-risk parameters categorized by common vulnerability classes (e.g., IDOR, SQLi, XSS, SSRF, File Inclusion), offering clear guidance for focused testing efforts.
  • Seamless Burp Suite Integration: The integration with xnl_hacker's Gap Burp Suite extension offers real-time identification of "sus" parameters during manual testing, providing instant, context-aware alerts to potential vulnerabilities.
  • Open-Sourced Methodology: The entire data acquisition, AI parsing, and list generation methodology is open-sourced, empowering organizations to create their own tailored "sus param" lists from internal incident data or bug bounty reports.
  • Cost-Effective Data Processing: Leveraging modern AI APIs like OpenAI's GPT-4 for such a large dataset (15,000+ reports) proved remarkably cost-effective, demonstrating the accessibility of advanced data processing techniques.

About the Speaker(s)

Jason Haddix is a highly respected figure in the application security community, known for his extensive experience and contributions to offensive security research. With over 15 years in the field of web application testing, Haddix has developed a deep understanding of vulnerability patterns and exploitation techniques. He previously presented foundational research on data-driven web hacking, titled "Hunt," at DEF CON 26. Beyond his research, Jason is also recognized for his role in co-running SecLists with Daniel Miessler, a widely used collection of useful payloads and wordlists for security testing. His work on SusParams continues his mission to democratize security knowledge and provide practical tools for the hacking community.

Gunnar Andrews is a security researcher and collaborator on the SusParams project. He played a pivotal role in the technical implementation, particularly in developing the data acquisition scripts for HackerOne and pioneering the innovative AI-driven parsing methods. His work on overcoming the challenges of extracting structured information from unstructured text using GPT-4 was central to the project's success, demonstrating his expertise in leveraging modern technologies for security research.

Reviews

Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT

Haddix and Andrews didn't drop a zero-day here, they dropped a massive timesaver. Taking 15,000 public HackerOne reports, beating them into shape with GPT-4 and regex, and shoving the output straight into a Burp extension is exactly the kind of pragmatic tooling the appsec community needs.

Heather Calloway (CISO) — STRONG ACCEPT

Jason Haddix and Gunnar Andrews used a $30 GPT-4 run to parse 15,000 bug reports and build statistically validated lists of vulnerable application parameters. This isn't just a win for manual penetration testers; it's a blueprint for scaling AppSec. If your testing program relies purely on the expensive intuition of senior engineers, you are wasting budget and missing coverage.

→ Top-rated talks at AppSec Village @ DEF CON 31

All talks from AppSec Village @ DEF CON 31