Before the CVE: Mining Open-Source Commits for Negative Day Vulnerabilities
Dmitrijs Trizna (AISLE)
[un]prompted.au 2026 · Day 2 · Main Hall
Overview
In a revealing talk at [un]prompted.au, Dmitrijs Trizna from AISLE unveiled a groundbreaking approach to proactively identify security vulnerabilities before they are publicly disclosed through the Common Vulnerabilities and Exposures (CVE) program. Titled "Before the CVE: Mining Open-Source Commits for Negative Day Vulnerabilities," the presentation delved into the critical window between a security patch being committed to an open-source repository and its official CVE issuance—a period Trizna terms the "negative day vulnerability" window. This often-overlooked timeframe presents a significant opportunity for threat actors, as traditional Software Composition Analysis (SCA) tools remain blind to these fixes until a CVE is published.

Key moments
- 3:40 What are negative day vulnerabilities?
- 7:40 Applying AI for vulnerability discovery: AISLE's 400 CVEs
- 10:20 Comparing LLM performance for identifying security patches
- 14:40 Scaling vulnerability detection across the software ecosystem
- 18:00 Protecting against prompt injections in Git-fed LLMs
- 21:40 The future of CVEs in the AI era
- 27:00 Q&A: LLM cost trajectory and training data contamination
Before the CVE: Mining Open-Source Commits for Negative Day Vulnerabilities
Speakers: Dmitrijs Trizna, AI & ML Engineer, AISLE
Conference: [un]prompted.au
YouTube: https://www.youtube.com/watch?v=kdJI5DemkSk
Overview
In a revealing talk at [un]prompted.au, Dmitrijs Trizna from AISLE unveiled a groundbreaking approach to proactively identify security vulnerabilities before they are publicly disclosed through the Common Vulnerabilities and Exposures (CVE) program. Titled "Before the CVE: Mining Open-Source Commits for Negative Day Vulnerabilities," the presentation delved into the critical window between a security patch being committed to an open-source repository and its official CVE issuance—a period Trizna terms the "negative day vulnerability" window. This often-overlooked timeframe presents a significant opportunity for threat actors, as traditional Software Composition Analysis (SCA) tools remain blind to these fixes until a CVE is published.
The core of Trizna's research and AISLE's methodology lies in leveraging advanced Artificial Intelligence and Machine Learning techniques, specifically Large Language Models (LLMs), to automate the detection of security-relevant patches within the vast landscape of open-source software commits. This innovative application of AI aims to transform a previously intractable problem into a scalable and cost-effective defensive strategy. By shifting from reactive vulnerability management to proactive patch discovery, organizations can significantly reduce their exposure to exploitable flaws, gaining a crucial advantage in the ever-escalating cybersecurity arms race.
The talk not only illuminated the existence and impact of negative day vulnerabilities but also provided a detailed technical deep dive into the AI models and methodologies employed. Trizna demonstrated how their AI-native approach, refined through extensive experience in vulnerability discovery, can effectively sift through millions of commits, identify security fixes, and enable early patching. Furthermore, the discussion extended to the critical defensive implications, including strategies for securing AI agents against adversarial inputs and the broader future of vulnerability disclosure in an AI-driven world.
Background
▶ Watch: What are negative day vulnerabilities? (3:40)
The modern software supply chain is a complex web of interconnected components, with the vast majority of application logic residing within third-party libraries and dependencies. While developers often manage these dependencies by version tags, the underlying codebases are dynamic, with continuous commits and patches occurring between major releases. It is within this constant evolution that the concept of negative day vulnerabilities emerges.
A negative day vulnerability refers to the critical window between a security patch being publicly committed to a version control system (like Git) and the official public disclosure of that vulnerability via a CVE. During this period, the fix for a security flaw is available, but users and automated security tools remain unaware of the underlying vulnerability. For threat actors, this window represents a golden opportunity: a vulnerability that is effectively "patched but undisclosed," allowing them to develop exploits with a significant head start before defenders can react. Trizna distinguishes this from zero-day vulnerabilities (unknown to vendors) and likens it more to N-day exploits, where a patch exists but users haven't updated.
The problem is exacerbated by cases of "silent patches," where maintainers fix a vulnerability without ever requesting a CVE, leaving users permanently in the dark. Trizna provided several compelling examples:
- OpenClaw RCE: A remote code execution vulnerability patched two weeks prior to the talk, with no CVE issued at the time, leaving users unknowingly exposed.
- Authlib
alg:none: A severe vulnerability (CVE-9.8 severity) where a patch landed on February 6th, but the CVE was not issued until March 6th—a full 30-day head start for attackers. Thealg:nonevulnerability type is so common it has become an industry joke. - Axios CVE: An extreme case where a patch was committed on October 30th, but the CVE was not issued until April 8th of the following year, creating a nearly five-month negative day window.
Traditional Software Composition Analysis (SCA) tools, such as Dependabot and Snyk, are inherently reactive. They rely on publicly available vulnerability databases and CVEs to identify known issues in dependencies. Consequently, they are completely blind during the negative day window, leaving organizations unprotected. Research cited by Trizna, which analyzed over one million commits, found that while 2% of commits carried a security signal (a fix), a staggering 77% of these security-related patches were released before their corresponding CVEs went public. This highlights the widespread nature of the negative day problem and the diligence of maintainers in shipping fixes rapidly, often ahead of bureaucratic disclosure processes. This long-standing issue has historically been "untractable" due to the sheer volume and complexity of analyzing open-source commit histories at scale.
Key Findings
▶ Watch: Comparing LLM performance for identifying security patches (10:20)
AISLE's research and development have yielded several significant findings, demonstrating the viability and effectiveness of an AI-native approach to identifying negative day vulnerabilities. Building on their extensive experience in vulnerability discovery—having secured over 400 CVEs in high-criticality software like FFmpeg, FreeBSD, Firefox, and Signal—AISLE adapted its existing AI harness for this novel application.
The methodology involved creating a target list of over 1,000 common libraries across various ecosystems. For each customer's specific dependency versions, AISLE's system scanned all future commits that landed after that version. These commits were then fed into their AI harness, not to find new vulnerabilities, but to identify patches that addressed security issues. A crucial step involved backtracking the CVE program to confirm whether these identified patches later correlated with officially issued CVEs, thus validating their "negative day" status.
To rigorously test their approach, AISLE downsampled their extensive findings into a tangible dataset comprising 40 high-use libraries (including Django, Flask, and various crypto libraries) and 300 commits. This dataset included both actual patch commits and "clean" commits, with an evaluation window of 12 hours around each patch to assess model precision.
The performance of various LLMs on this task revealed critical insights:
- Suboptimal Performance: High-profile models like Astra and Fable performed "horribly." Fable, in particular, exhibited "cyber policy violations," refusing to analyze commits related to security fixes, deeming them "bad actor" activities—a significant limitation for security-focused applications. This underscored that state-of-the-art (SOTA) models are not always suitable for every task, especially in specialized domains.
- Strong Contenders: While Sonnet 5 achieved the best overall performance, its cost implications made it less ideal for scalable deployment. More surprisingly, open-source models like Qwen and the newly released DeepSeek demonstrated "surprisingly good" recall, making them highly feasible options.
- Cost-Effectiveness Champion: Luna, described as the "smallest GPT kind of architecture," emerged as a standout. It performed "quite well" while being the cheapest, smallest, and fastest model. This finding is crucial for practical, large-scale implementation.
A Pareto frontier analysis, which balances recall against cost, clearly positioned Luna and DeepSeek as optimal choices for organizations prioritizing both effectiveness and scalability. The research also explored different harness approaches:
- Simple Agent: An agent capable of exploring a repository, reading, and grepping files. This was used for initial experiments.
- Single Call: A minimalist approach where only the commit diff is fed to the LLM, with a direct query: "Is there a patch?"
- State-of-the-Art Harness: AISLE's sophisticated harness used for general CVE discovery.
While the state-of-the-art harness achieved slightly higher recall, the "single call" approach with just the commit diff proved remarkably effective. Despite a modest 5% drop in recall, it offered a 10x improvement in price and speed. This efficiency gain is paramount for scaling the solution to cover vast numbers of commits. For instance, using Luna with the single-call method, scanning 100,000 commits could cost as little as $100, making it accessible even for "marginal players" without extensive AI lab resources.
Furthermore, the team identified a critical scaling advantage: scanning common libraries once allows for reuse across multiple projects. By analyzing 50,000 Software Bill of Materials (SBOMs) across ecosystems like NPM, PyPI, NuGet, and Composer, they found a significant "common core." In NPM, for example, the most common library is reused across 92% of projects, and the top 20 libraries cover 85% of projects. This means scanning a relatively small number of widely used packages can provide broad coverage for a substantial portion of the open-source ecosystem, making the approach highly efficient and impactful.
Technical Deep Dive
▶ Watch: Scaling vulnerability detection across the software ecosystem (14:40)
The technical underpinning of AISLE's approach to identifying negative day vulnerabilities is rooted in an AI-native methodology that leverages the analytical capabilities of Large Language Models (LLMs) to parse and interpret code changes. The process begins with establishing a comprehensive target list of over 1,000 widely used open-source libraries, spanning various programming language ecosystems such as NPM, PyPI, NuGet, and Composer.
For a given customer's software, which typically relies on specific versions of these libraries, the system initiates a continuous monitoring process. Instead of focusing on published CVEs, the core task is to analyze all future commits that land in the upstream repositories after the customer's currently deployed dependency versions. These commits represent the raw material for detection.
The commits are then fed into AISLE's specialized AI harness. While AISLE possesses a highly sophisticated harness for general CVE discovery—which has been instrumental in securing 400 CVEs—the negative day vulnerability detection task required a tailored application. The goal here is not to discover new vulnerabilities but to identify security-relevant patches within these commits. This distinction is crucial: the AI is looking for indicators of a fix, rather than an exploitable flaw itself.
To evaluate the effectiveness of various LLMs and harness configurations, a carefully curated dataset was constructed. This dataset comprised 40 high-use libraries (e.g., Django, Flask, various cryptographic libraries) and 300 individual commits. For each commit identified as a potential negative day patch, an "evaluation window" of 12 hours was established, including the actual patch commit and surrounding "clean" commits. This allowed for precise measurement of recall (the ability to identify true positives) and precision (avoiding false positives).
Several prominent LLM architectures were tested, including Astra, Fable, Sonnet 5, Qwen, DeepSeek, and Luna, alongside other models like GPT-4o. The performance varied significantly:
- Fable, for example, exhibited "cyber policy violations," refusing to process security-related queries, rendering it unsuitable for this domain. This highlights the challenges of applying general-purpose LLMs to sensitive security tasks without specialized fine-tuning or access.
- Sonnet 5 showed high recall but was deemed too costly for large-scale deployment.
- Qwen and DeepSeek demonstrated strong performance, particularly for open models, indicating the increasing viability of open-source alternatives.
- Luna, a smaller GPT-style architecture, proved to be an optimal choice due to its balance of good recall, speed, and significantly lower cost.
The talk emphasized the importance of harness design for scalability. Three main approaches were explored:
- Simple Agent: This involved an agent capable of basic repository exploration, such as reading files and using
grepcommands. While effective, it introduced more complexity and potential attack surface. - Single Call Approach: This was the most streamlined and efficient method. The raw commit diff (the textual representation of changes between two code versions) was directly fed to the LLM with a simple query asking whether the diff contained a security patch. This approach significantly reduced computational overhead and complexity.
- State-of-the-Art Harness: AISLE's full-fledged vulnerability discovery harness, offering maximum analytical depth but also higher resource consumption.
Crucially, the "single call" approach, despite a minor 5% drop in recall compared to the more complex harnesses, delivered a 10x improvement in both speed and cost-efficiency. This makes it the most feasible option for widespread, continuous monitoring. For instance, scanning 100,000 commits with Luna using this method could be achieved for approximately $100, democratizing access to this advanced capability.
A key enabler for scaling this approach across the entire open-source ecosystem is the concept of Software Bill of Materials (SBOMs). SBOMs provide a standardized, machine-readable inventory of software components, including their versions and dependencies. By aggregating and analyzing 50,000 anonymized SBOMs from various customers and environments, AISLE identified a significant common core of widely reused libraries. For example, in the NPM ecosystem, the top-most common library is used across 92% of projects, and the top 20 libraries are still used by over 85%. This means that by scanning these common core dependencies once, the results can be reused across a vast number of projects, significantly reducing redundant processing and further driving down costs. Specific examples of these highly reused libraries include debug, lodash, and react for NPM, and requests, urllib3, and certifi for Python. While Python's dependency graph is less centralized than NPM's, the principle of scanning common components still yields substantial coverage.
The talk also addressed the critical security concerns associated with feeding untrusted Git data directly into AI agents. Trizna highlighted the "wild west" nature of public repositories, where prompt injections can be found in issues, pull requests (PRs), and even commit messages. These adversarial inputs can hijack LLMs, leading to data exfiltration or other supply chain compromises. To mitigate this, specific defensive strategies were outlined:
- Architectural Isolation: Employing simple, single-query LLM agents minimizes the impact of a potential hijack, as there are no tools or secrets to exfiltrate.
- Instruction Isolation: Using dynamically generated, unique XML tags to encapsulate untrusted input, making it harder for attackers to craft escape sequences.
- Agent Isolation: Running agents within sandboxes with restricted network access, only allowing communication with trusted internal resources like an Artifactory.
- Monitoring: Implementing robust monitoring of LLM prompts and tool traces, potentially using solutions like OpenTelemetry or even smaller, aligned LLMs for "scalable oversight."
This multi-faceted technical strategy allows organizations to harness the power of AI for proactive vulnerability detection while managing the inherent risks of processing untrusted external data.
Demo / Proof of Concept
▶ Watch: The future of CVEs in the AI era (21:40)
While the talk did not feature a live, interactive demonstration in the traditional sense, the entire presentation served as a comprehensive conceptual proof of concept for AISLE's methodology. Dmitrijs Trizna meticulously detailed the experimental setup, the construction of their custom dataset, the testing of various LLM models, and the analysis of their performance metrics (recall, cost, speed).
The results, presented through scatter plots and bar graphs, effectively demonstrated that their AI-native approach is not only feasible but also scalable and cost-effective. The rigorous evaluation of different harness configurations and the identification of optimal models like Luna and DeepSeek, alongside the quantitative analysis of cost savings (e.g., $100 for 100,000 commits), provided compelling evidence of the system's practical viability. The discussion of real-world examples like Authlib and Axios further solidified the relevance and impact of detecting these "negative day" vulnerabilities.
Defensive Implications
▶ Watch: Q&A: LLM cost trajectory and training data contamination (27:00)
The implications of AISLE's research for cybersecurity defenders are profound, offering a paradigm shift from reactive vulnerability management to proactive risk mitigation. By leveraging AI to identify negative day vulnerabilities, organizations can significantly bolster their defensive posture across multiple fronts.
Firstly, the most immediate implication is the ability to enable proactive patching. Instead of waiting for a CVE to be published, which can take weeks or months, defenders can now identify security-relevant patches as soon as they land in public repositories. This allows for internal patching or updates to dependencies long before the vulnerability becomes widely known, effectively closing the window of opportunity for attackers. This directly addresses the "dark matter" of dependencies, bringing visibility and control to a previously opaque area of the software supply chain.
Secondly, this AI-powered approach serves as a crucial augmentation for traditional SCA tools. Existing SCA solutions like Dependabot and Snyk are indispensable but inherently limited by their reliance on public CVE databases. By integrating AI-driven commit analysis, organizations can supplement these tools, gaining early warning for vulnerabilities that have been patched but not yet disclosed. This creates a more comprehensive and forward-looking security posture for software supply chain integrity.
Thirdly, the research provides a blueprint for strengthening supply chain security against sophisticated threats. The ability to scan and understand the security implications of commits, even before official disclosure, contributes to a more resilient software ecosystem. This includes the potential to detect backdoored or hijacked commits, as hinted by Trizna in the Q&A, by scrutinizing unusual or suspicious changes that might not immediately trigger a CVE but could indicate malicious intent.
Crucially, the talk highlighted several critical LLM security considerations for any organization adopting AI for such tasks:
- Architectural Isolation: Employing simple, single-query LLM agents for tasks like commit analysis minimizes the attack surface. If an agent is compromised by an adversarial commit, the impact is contained, as it lacks access to sensitive tools, secrets, or broader system functions.
- Instruction Isolation: Implementing robust input sanitization and isolation techniques, such as dynamically generating unique XML tags around untrusted input, can prevent prompt injection attacks where malicious code within a commit message could hijack the LLM's instructions.
- Agent Isolation (Sandboxing): Running LLM agents in highly restricted sandboxes is paramount. These environments should deny internet access and limit communication solely to trusted internal resources (e.g., an Artifactory for fetching code), preventing data exfiltration or arbitrary code execution.
- Monitoring: Comprehensive monitoring of LLM interactions, including tool calls, prompts, and responses, is essential. Solutions like OpenTelemetry can be integrated to ship traces centrally. Furthermore, the concept of "scalable oversight," where smaller, aligned LLMs monitor the behavior of larger, more cognitively capable models, offers a promising avenue for detecting rogue agent behavior.
Finally, the work underscores the increasing importance of SBOM adoption. By providing a structured inventory of software components, SBOMs enable organizations to efficiently identify their common dependencies. This allows for the strategic scanning of core libraries, whose results can be reused across a multitude of projects, maximizing coverage and cost-efficiency. This proactive approach also implicitly challenges the limitations of the current CVE program, suggesting that while CVEs remain vital, they may not be keeping pace with the rapid development cycles and AI-driven analysis capabilities of the modern era. Organizations should explore internal databases and proactive scanning to supplement traditional CVE-based vulnerability management.
Key Takeaways
- Negative Day Vulnerabilities are a Critical Blind Spot: The window between a security patch landing in public open-source repositories and its official CVE disclosure creates a significant head start for attackers, leaving traditional SCA tools and defenders unaware.
- AI/LLMs Offer a Scalable Solution: Large Language Models can effectively identify these pre-CVE security patches in open-source commits, transforming a previously intractable problem into a manageable and scalable defensive capability.
- Cost-Effectiveness and Open Models are Key: Simple LLM agents, particularly smaller, efficient models like Luna, when combined with a single-call analysis of commit diffs, can analyze hundreds of thousands of commits for a minimal cost (e.g., $100 for 100k commits), making this approach accessible to many organizations.
- SBOMs Enable Ecosystem-Wide Coverage: Leveraging Software Bill of Materials (SBOMs) to identify common dependencies allows for efficient scanning of core libraries, with results reusable across a vast number of projects, significantly reducing operational overhead.
- Securing AI Agents is Paramount: When processing untrusted Git data, robust security measures for LLM agents—including architectural isolation, instruction isolation, sandboxing, and continuous monitoring—are crucial to prevent prompt injection attacks and supply chain compromises.
- Proactive Defense is the Future: This methodology empowers organizations to shift from reactive vulnerability management to a proactive stance, identifying and addressing security flaws before public disclosure and gaining a critical advantage against evolving threats.
About the Speaker(s)
Dmitrijs Trizna is an AI & ML Engineer at AISLE, bringing a wealth of experience from both the offensive and defensive sides of cybersecurity. With 15 years in the information security field, Dmitrijs has a strong background in professional red teaming, equipping him with a deep understanding of attacker methodologies. He transitioned into the realm of Artificial Intelligence and Machine Learning even before the widespread adoption of LLMs, when the field was primarily referred to as ML.
Dmitrijs is a recognized speaker, having presented at prestigious conferences such as Black Hat and DEF CON AI Village. He holds the OSCP (Offensive Security Certified Professional) certification, further underscoring his practical expertise in penetration testing. His work at AISLE focuses on applying AI-native approaches to vulnerability discovery, a field where his team has achieved significant success, having been issued over 400 CVEs in high-profile software. AISLE is notably a researcher CNA (CVE Numbering Authority), allowing them to issue CVEs more efficiently due to their specialized expertise.
Reviews
Dr. Zero (Offensive Security Researcher) — SOLID
We can drop the marketing term 'negative day' immediately—it's just a silent patch. But automating silent patch discovery across thousands of repos using cheap LLMs for a hundred bucks is a massive win for defenders, and Trizna brought the receipts and cost analyses to prove it works.
Heather Calloway (CISO) — MUST SEE
This research exposes a structural failure in how the industry handles open-source vulnerabilities: the multi-week gap between a patch landing and a CVE publishing. It proves that relying solely on traditional SCA tools leaves a massive window of exposure, and demonstrates a cost-effective, LLM-driven method to close it. This is required viewing for your AppSec and engineering leadership.