AI Is Writing Your Bug Reports. Can You Tell?

AI Is Writing Your Bug Reports. Can You Tell?

Speakers: Khushali Dalal, Senior Vulnerability Analyst, VulnCheck

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=aNC7rsAB5es

Overview

In an era where artificial intelligence increasingly permeates every facet of technology, its impact on vulnerability management and bug bounty programs has become a critical concern. Khushali Dalal, a Senior Vulnerability Analyst at VulnCheck, delivered a compelling talk at VulnCon, addressing the escalating challenge of distinguishing between genuinely researched vulnerability reports and those expertly crafted by AI. The central premise of her presentation, "AI Is Writing Your Bug Reports. Can You Tell?", is not to condemn AI but to equip security professionals with the discerning eye needed to navigate this new landscape.

Dalal highlighted a significant and growing gap between the rapid evolution of vulnerability intake processes and the slower adaptation of triage methodologies. The influx of highly polished, yet often unsubstantiated, AI-generated reports is creating a signal-to-noise ratio crisis, leading to wasted resources and the potential oversight of legitimate security issues. Her talk served as a crucial guide for vulnerability analysts and PSIRT teams, emphasizing the shift from evaluating the fluency of a report to rigorously assessing its underlying evidence and confidence.

The proliferation of AI-assisted reporting poses an urgent operational issue, threatening the efficiency and integrity of vulnerability disclosure ecosystems. Dalal's insights are particularly vital for organizations grappling with increased report volumes and the nuanced complexities of AI-driven communication. By focusing on practical, evidence-based triage habits, her presentation offers a strategic framework for maintaining robust security postures in the face of evolving AI capabilities.

Background

▶ Watch: Introduction: The AI-driven vulnerability report gap (0:00)

The landscape of vulnerability reporting underwent a profound shift in late 2025 and early 2026, driven by three interconnected factors that transformed what was once a manageable intake process into an urgent operational challenge. First, there was an unprecedented volume of reports. Not only did the sheer number of submissions increase, but they also arrived more frequently and with a consistently higher level of polish. This surge was directly attributable to AI's ability to generate comprehensive-looking reports rapidly.

Second, the fluency of these reports reached new heights. AI models became adept at mimicking the language and structure of experienced security researchers. Reports began to sound as if they were penned by seasoned professionals, employing appropriate technical vocabulary and presenting findings with a level of sophistication that was previously indicative of expert human analysis. This made initial differentiation incredibly difficult for human triagers.

Third, a pervasive bias emerged. Humans are inherently predisposed to trust information that sounds confident, urgent, and well-articulated. When AI-generated reports presented vulnerabilities with high severity scores, detailed descriptions, and a sense of immediate threat, triagers were naturally inclined to prioritize them. This psychological bias, coupled with AI's linguistic prowess, led to a critical problem: the risk was not merely slow triage, but rather fast, yet incorrect triage, consuming valuable resources on non-existent or exaggerated issues.

The real-world consequences of this shift are already evident. Khushali Dalal cited prominent examples, underscoring the severity of the issue. Curl, a tool utilized by billions of devices globally, was forced to shut down its public bug bounty program as of January 31, 2026. The primary reason? A catastrophic collapse in the signal-to-noise ratio, rendering their program unsustainable due to the overwhelming volume of low-quality or AI-generated submissions. Similarly, GitHub reported a staggering 224% increase in vulnerability report volume over a single 90-day period. GitHub's internal teams flagged the quality of these AI-generated reports as a major operational issue, highlighting the immense burden placed on triage teams to sift through the noise to identify legitimate threats. These examples illustrate a clear pattern: more polished intake, increased burden on triage, and a growing need to delay trust until verifiable artifacts can be found within a report.

Key Findings

▶ Watch: Goal: Build triage habits, not AI detection (4:00)

The central finding of Dalal's research and presentation is that the increasing sophistication of AI in generating vulnerability reports has fundamentally altered the challenge of vulnerability triage. While AI can produce highly fluent and polished narratives, this often masks a critical lack of verifiable evidence and can even introduce contradictory or hallucinated details. The core insight is that stronger facts always beat stronger writing in the context of effective vulnerability management.

Dalal categorized reports into three distinct patterns, each requiring a specific triage approach:

1. Mostly AI-generated reports: These are characterized by smooth narratives and confident language, often including high CVSS scores and mapped CWEs. However, upon closer inspection, they frequently contain weak validation, contradictory statements within the same report (e.g., claiming no authentication needed while also requiring admin privileges), and can even cite hallucinated CVE IDs that do not exist or are unrelated. These reports are designed to appear credible at first glance but fall apart under scrutiny due to their lack of underlying factual coherence.

1. Human-found, AI-packaged reports: This pattern represents a particularly dangerous scenario. A genuine vulnerability is discovered by a human researcher, but the report is then passed through an AI to enhance its narrative, making it sound more impactful or urgent. While the underlying bug is real, the AI's embellishments can inflate the claimed blast radius or severity beyond what the evidence supports. Triagers might dismiss such a report due to its inflated language, thereby missing a legitimate security flaw. The challenge here is to separate the confirmed bug from the AI-generated narrative.

1. Mostly human-written reports: These reports, while perhaps less polished or "impressive" in their language, are distinguished by their high signal-to-noise ratio. They focus on specific details, concrete reproduction steps, observed behavior, and lack inflated certainty or borrowed severity language. They prioritize factual evidence over eloquent prose, making them far more actionable and efficient for triage teams. Such reports might even start with hesitant phrasing like "maybe," but their strength lies in their verifiable content.

The overarching contribution is the emphasis on building a triage habit centered on scoring confidence based on evidence, rather than being swayed by the fluency or packaging of the text. This involves a systematic approach to questioning claims, demanding artifacts, and focusing solely on validated information.

Technical Deep Dive

▶ Watch: Case Study 2: Human found, AI packaged (most dangerous) (12:00)

Khushali Dalal's presentation delved into the nuances of AI-generated vulnerability reports through three compelling case studies, each designed to illustrate the specific signals and pitfalls that triagers must learn to identify. These examples served as a practical "training" for the audience to discern the origin and reliability of reported vulnerabilities.

Case 1: Critical Unauthenticated RCE in Acme Controller GraphQL API

The first example presented a report titled "Critical unauthenticated RCE in Acme controller GraphQL API" with an alarming CVSS score of 9.8 and a claim that "all versions are affected" and "internet exploitable." The description was fluent, well-structured, and included typical elements like impact and affected versions.

• Initial Impression: At first glance, the report appeared highly credible, mirroring the structure and confidence of a professional security researcher. The inclusion of a high CVSS score and broad impact claims (unauthenticated RCE) immediately flags it as critical.
• Technical Red Flags: The critical flaw emerged from contradictory statements within the report itself. One sentence asserted, "This issue is exploitable without authentication," while a subsequent sentence stated, "in my POC attached, successful exploitation required admin only privileges." This glaring inconsistency is a hallmark of AI-generated content; while AI can assemble plausible-sounding technical terms and structures, it often struggles with maintaining logical coherence and factual consistency across different parts of a detailed claim. Furthermore, Dalal noted that a CVE ID attached to this report turned out to be hallucinated upon research.
• Analysis: This report was classified as mostly AI-generated. AI excels at generating a "smooth narrative" but frequently lacks the underlying understanding to ensure technical accuracy and consistency. A human researcher would either not include such a contradiction or would provide immediate clarification. The lack of genuine validation beneath the polished text is the key differentiator.

Case 2: Privilege Escalation via Support Bundle Upload

The second case study involved a report detailing a "Privilege escalation via support bundle upload," specifically tested on a particular version and indicating exploitation by a low-privilege role. This report was more specific than Case 1, mentioning observed behavior like modifying a manifest to influence an admin interface.

• Initial Impression: This report provided more concrete details, such as a specific version and a low-privilege context, which are typically strong indicators of human research. However, the claimed impact was "massive," including "admin compromise," "full environment takeover," and "sensitive data access."
• Technical Red Flags: While the description provided specific observed actions (e.g., "modified the manifest and influenced admin interface"), the claimed "full environment takeover" blast radius was not adequately substantiated by the provided details. The language used to describe the ultimate impact felt inflated, going beyond the direct evidence.
• Analysis: This report was deemed human-found and AI-packaged. This is considered the most dangerous pattern because it contains a genuine vulnerability, but the AI's embellishment of the impact can lead triagers to dismiss it due to the "inflated language." The defensive strategy here is to separate the confirmed bug from the claimed blast radius, scoring only the verified findings (e.g., the ability to influence an admin interface through manifest modification) and trimming the exaggerated narrative.

Case 3: Maybe Authentication Bypass Export Issue in API V2 Reports Download

The final case presented a report with a hesitant title: "Maybe authentication bypass export issue in API V2 reports download." It lacked a CVSS score or grand claims of total compromise. Instead, it described a specific scenario: "I was logged in as a viewer-only user and downloaded a report from reports page. I copied the request from the browser network tab and replayed it after changing the ID parameter to another report ID. The application returned this this this."

• Initial Impression: This report was notably "low polish" and "less impressive" in its presentation. The use of "maybe" and the absence of a CVSS score might initially lead a triager to undervalue it.
• Technical Green Flags: The strength of this report lay in its high signal. It provided specific endpoints (API V2 reports download), concrete roles (viewer-only user), exact reproduction steps (copying request, changing ID parameter, replaying), and observed behavior ("application returned this this this"). There was "no inflated certainty," "no fancy wording," and "no borrowed severity language."
• Analysis: This was identified as mostly human-written. Its value comes from the ease of replication and validation. Triagers can quickly follow the precise steps to confirm the vulnerability in "10-15 minutes." This highlights that stronger facts always beat stronger writing. The detailed, verifiable steps are far more valuable than a highly polished, yet vague, description.

Across these cases, the technical deep dive reveals that AI excels at generating plausible form but struggles with consistent substance. Triagers must develop a habit of scrutinizing specific details: looking for product versions, environment setups, authentication states, expected versus actual behavior, and crucially, demanding raw data, proofs of concept (POCs), screenshots, and logs. These artifacts are the bedrock of verifiable truth, distinguishing genuine security research from AI-generated noise.

Demo / Proof of Concept

▶ Watch: Stronger facts always beat stronger writing (17:00)

While Khushali Dalal's talk did not feature a live technical demonstration or a specific Proof of Concept tool in the traditional sense, she employed an equally effective interactive method to illustrate her points. The presentation was structured around three real-world case studies of vulnerability reports, which served as interactive examples for the audience.

For each case, Dalal displayed the raw text of a vulnerability report on screen and then prompted the audience to vote, via a QR code, on whether they believed the report was:

1. AI-generated
2. Human-written
3. A combination of human-found and AI-assisted

This audience participation served as a dynamic "proof of concept" for the talk's central thesis: even experienced security professionals can be swayed by the fluency of AI-generated text, and it takes a deliberate focus on specific signals and evidence to correctly identify the report's origin and veracity. The voting results, which Dalal shared in real-time from her phone, allowed for immediate feedback and reinforced her analysis of the technical and linguistic cues present in each report. This interactive approach effectively demonstrated the challenges faced by triage teams daily without requiring a live code or exploit demonstration.

Defensive Implications

▶ Watch: The PAUSE mnemonic for effective vulnerability triage (20:00)

The rise of AI-generated vulnerability reports necessitates a proactive and systematic shift in defensive strategies for PSIRT teams and vulnerability management programs. Khushali Dalal outlined several crucial recommendations, focusing on both the intake process and ongoing triage habits.

At Intake:

The first line of defense is to make the submission process more rigorous. Organizations should require more specific data upfront. This includes:

• Exact product versions tested.
• Details about the environment setup (e.g., cloud provider, OS, specific configurations).
• Authentication states used during testing (e.g., authenticated as admin, low-privilege user, unauthenticated).
• A clear description of expected versus actual behavior observed during the vulnerability discovery.

To facilitate this, organizations should update their submission templates to explicitly demand this information. Making these fields mandatory will immediately filter out many shallow, AI-generated reports that cannot provide such granular detail.

During Follow-up:

Once a report is received, the follow-up phase is critical for validating its claims. Defenders should:

• Ask one or two domain-specific questions. A shallow AI model, even a sophisticated one, typically lacks the deep contextual understanding to bluff convincingly on highly specific technical questions related to the reported vulnerability. A genuine researcher who has reproduced the bug will respond promptly and accurately. As Dalal noted, "a padded report won't survive the follow-up."
• Separate the confirmed bug from the claimed blast radius. Triagers must learn to isolate the verifiable observations from exaggerated impacts. Score only the verified truth—what the researcher actually observed and could prove—rather than the "imagined worst-case scenario" or inflated language about "full system compromise." This prevents over-prioritization of non-critical issues and ensures resources are allocated effectively.

Concrete Triage Habits (The PAUSE Mnemonic):

To embed these principles into daily operations, Dalal introduced the PAUSE mnemonic, a structured approach to initial report assessment:

• Parse the claim: Reduce the entire report to one testable statement. What is the single, most critical assertion that can be verified? This helps cut through narrative fluff.
• Ask for artifacts: Always demand proof of concept (POC) code, screenshots, logs, or other raw data. These are undeniable pieces of evidence that AI struggles to fabricate credibly.
• Understand the precondition: What are the specific versions? What features need to be enabled? What is the authentication state? Replicate the exact conditions under which the vulnerability was found.
• Score confirmed impact: Base the vulnerability score (e.g., CVSS) solely on the evidence in hand and what has been validated, not on hypothetical maximum impact. The score can be adjusted later if more details emerge.
• Evolve the workflow: Implement checkpoints, metrics, and feedback loops within PSIRT and vulnerability management teams. Continuously refine intake templates, triage procedures, and training based on observed patterns of AI-generated reports. This ensures the defensive posture adapts as AI capabilities evolve.

Furthermore, general best practices include:

• Rerunning exact requests mentioned in reports to confirm behavior.
• Testing the same request with different low-privilege roles to confirm scope and impact.
• Verifying against other versions of the product before assigning final severity.

The overarching defensive implication is a shift from implicit trust in report fluency to explicit demand for verifiable evidence. By adopting these strategies, organizations can regain control over their vulnerability intake pipelines, reduce wasted effort on false positives, and ensure that genuine security threats are identified and addressed promptly.

Key Takeaways

• AI-driven report fluency is a critical operational challenge: The ability of AI to generate polished, confident, and seemingly credible vulnerability reports has led to a collapse in the signal-to-noise ratio within bug bounty and vulnerability intake programs, exemplified by Curl shutting down its program and GitHub's 224% increase in report volume.
• Fluency does not equate to substantiation: A well-written report does not automatically mean it is well-substantiated. Triagers must learn to look beyond the packaging and language quality, focusing instead on the underlying evidence, consistency, and verifiable facts.
• Identify report patterns: Distinguish between purely AI-generated reports (smooth narrative, weak validation, contradictions, hallucinated CVEs), human-found but AI-packaged reports (real bug, inflated impact), and genuinely human-written reports (low polish, high signal, specific steps, observed behavior). The "human-found, AI-packaged" reports are particularly dangerous as they can lead to dismissal of real vulnerabilities.
• Implement stringent intake and follow-up protocols: Require specific details like product versions, environment setups, and authentication states in submission templates. During follow-up, ask domain-specific questions that shallow AI models cannot convincingly answer, and always demand raw data, POCs, screenshots, and logs.
• Adopt the PAUSE mnemonic for effective triage: This framework—Parse the claim, Ask for artifacts, Understand the precondition, Score confirmed impact, Evolve the workflow—provides a structured, evidence-first approach to quickly and accurately assess vulnerability reports.
• Prioritize facts over writing quality: In the battle against AI-generated noise, stronger, verifiable facts will always outperform stronger, more persuasive writing. Develop a habit of seeking concrete, replicable evidence to drive triage decisions and resource allocation.

About the Speaker(s)

Khushali Dalal is a Senior Vulnerability Analyst at VulnCheck, specializing in CV farming and coordinated vulnerability disclosures. Her work primarily focuses on the intake side of vulnerability programs, where she has observed firsthand the increasing challenges presented by the volume and nature of vulnerability reports. Dalal's expertise lies in understanding the evolving landscape of vulnerability submissions and developing strategies to effectively triage and manage them, particularly in the context of emerging AI capabilities. Her insights are crucial for organizations seeking to maintain robust security operations in the face of these new challenges.