Prompt Obfuscation for Large Language Models

"I wasn't sure if this is indeed a security risk": Data-driven Understanding of Security Issue Reporting in GitHub Repositories of Open Source npm Packages

Speakers: Rajdeep Ghosh, Shiladitya De, Mainack Mondal, Researchers at IIT Kharagpur

Conference: USENIX Security 2025

Overview

This paper presents a critical, data-driven investigation into the effectiveness and challenges of security issue reporting within the vast npm (Node Package Manager) ecosystem. Given npm's pivotal role in modern JavaScript development and its dense interdependencies, understanding how security vulnerabilities are identified, reported, and ultimately resolved is paramount. The research, conducted by Rajdeep Ghosh, Shiladitya De, and Mainack Mondal from IIT Kharagpur, exposes a significant disconnect between the actual prevalence of security-related issues and how they are formally recognized and addressed by package maintainers on GitHub.

The core of this work lies in its large-scale analysis of over 10 million issues across nearly 45,500 diverse npm packages hosted on GitHub. Through a combination of manual analysis and sophisticated machine learning (ML) models, the researchers uncovered that a staggering 14.8% of all reported issues are, in fact, security-related, a figure dramatically higher than the mere 0.13% explicitly tagged as such by developers. This 114-fold difference underscores a systemic problem of under-tagging and highlights the limitations of current security management practices, including the widespread but often ineffective use of bots.

The findings offer invaluable insights for enhancing security management in open-source ecosystems. By meticulously analyzing user-developer interactions and the factors influencing issue resolution, the study reveals that issues with clearly defined solutions, such as those referencing CVE (Common Vulnerabilities and Exposures) IDs, are resolved significantly faster. Conversely, even reproducible security issues without known solutions often languish, becoming stale. This research not only bridges a critical gap in understanding real-world security reporting but also provides actionable recommendations for improving tooling, collaboration, and overall security posture for npm package owners, bot developers, security researchers, and end-users.

Background

The npm ecosystem stands as the largest and most crucial package manager for JavaScript development, underpinning millions of applications and services. Its inherent design, characterized by deep and complex interdependencies, creates a fertile ground for the rapid propagation of security vulnerabilities. A single compromised package can potentially expose a vast network of dependent projects, making effective vulnerability management a cornerstone of open-source security.

Prior research has extensively explored various facets of security vulnerability management in package ecosystems like npm, PyPI, and RubyGems. These studies often focused on theoretical frameworks or proposed automated tools for vulnerability reporting, patch propagation, and detection. Topics such as upstream vulnerabilities, delays in patch delivery, and the critical importance of regular dependency updates have received considerable attention. Furthermore, the role of bots in automating developer workflows and enhancing productivity has been investigated, albeit with mixed conclusions regarding their efficacy in security-specific contexts.

However, a significant gap persisted in understanding the "ground reality" of security issue reporting. Most existing work tended to prescribe ideal automated solutions without adequately analyzing what mechanisms were actually in use, how effective they were, and whether the public nature of open-source platforms truly aided in detecting and mitigating security flaws. Specifically, it was unclear whether users or bots effectively reported or suggested mitigations for security issues, and if these reports had a tangible impact on improving package security. While features like Private Vulnerability Reporting (PVR) exist, they are relatively new and not widely adopted, meaning that public GitHub issues remain a primary channel for security disclosures. This research directly addresses these unanswered questions by conducting a large-scale, data-driven analysis of user-generated data, focusing on publicly available npm packages and their corresponding GitHub repositories.

Key Findings

The study uncovered several critical findings that shed light on the state of security issue reporting and resolution in the npm ecosystem:

• Massive Under-tagging of Security Issues: Despite collecting over 10.9 million issues across 45,466 npm packages, only a minuscule 0.13% (13,835 issues) were explicitly tagged as security-related by GitHub repository owners. However, the researchers' sophisticated machine learning models identified an additional 1,617,738 issues as security-related, representing a staggering 14.8% of all issues. This means that 114 times more security issues exist than are formally recognized through tagging, indicating a severe systemic problem in issue classification and visibility.
• Ineffectiveness of Current Bots: While bots, particularly Dependabot, are highly active in reporting security issues (accounting for 55.9% of all bot-reported security-tagged issues), their overall effectiveness in contributing to resolution is limited. The analysis revealed that most bots operate on predefined, rule-based logic, with only a handful (four out of 51 examined bots) specifically addressing security concerns. Even these security-focused bots are often paid services or in beta, and primarily rely on static analysis, lacking advanced AI/ML capabilities. Their primary role in user-reported issues is frequently to mark them as stale due to inactivity rather than to facilitate meaningful discussion or resolution.
• Challenges in Developer Responsiveness and Resolution: User-reported security issues often face significant hurdles in being addressed. Many issues are ignored by developers or closed without clear justification, particularly if they become stale. The study found that over 23% of security issues receive no comments at all, highlighting a lack of community engagement.
• Factors Influencing Resolution Time:
• CVE Mentions Expedite Resolution: Issues that explicitly mention a CVE ID are resolved significantly faster. These issues had a mean time to close of 174.08 days (median 70 days), compared to 307.19 days (median 127 days) for issues without CVE mentions. This indicates that publicly recognized vulnerabilities are prioritized due to their critical nature.
• Reproducibility Can Hinder Resolution (Counter-intuitively): While reproducibility is generally considered helpful, the study found that reproducible security issues often take longer to close (mean 105.16 days vs. 47.21 days for non-reproducible) and are more likely to become stale. This counter-intuitive finding suggests that reproducibility alone is not sufficient; without a known solution (like a CVE) or strong developer motivation, these issues can languish due to verification overhead or lack of capacity.
• Bot Involvement Correlates with Staleness: Bot activity is strongly correlated with increased staleness and a lower likelihood of successful resolution. Bots frequently intervene in later stages of an issue's lifecycle, often marking it as stale or automatically closing it due to inactivity, effectively halting further resolution efforts.
• Limited Active Maintainer Engagement: The resolution of an issue was found to be independent of the number of active maintainers. The study observed a low number of active maintainers engaging in discussions, with mean and median numbers of active maintainers at 4.83 and 2, respectively, and even fewer actively commenting (3.78 mean, 1 median).

Technical Deep Dive

The research methodology combined extensive data collection, qualitative analysis, and advanced machine learning techniques to uncover the intricate dynamics of security issue reporting.

Data Collection and Stratified Sampling:

The study began by collecting data from the npm registry in May 2024, identifying 4.3 million public JavaScript packages. To manage this massive dataset, the researchers employed stratified sampling based on the number of dependents (a proxy for package popularity and impact). They divided packages into six buckets (e.g., 0 dependents, 1-10, 10-100, etc.) and sampled 20,000 packages from each of the lower-dependent buckets, while including all packages from higher-dependent buckets (e.g., all packages with >100 dependents). This resulted in a curated set of 45,466 npm packages.

For these packages, the team collected 37,278 distinct GitHub repository links. Using the GitHub API, they then systematically gathered 10,907,467 issues from these repositories, with an overwhelming majority (92.3%) being closed issues. For each issue, they extracted the title, body (description), and crucial metadata such as associated tags and the usernames of those who posted issues or comments.

Initial Identification of Security-Related Issues via Tags:

GitHub allows repository owners to apply arbitrary tags to issues. Initially, the researchers explored using these tags to identify security issues. They found that only 13,031 out of 37,278 repositories utilized tags, and approximately 50% of all issues (5,454,149) had no tags at all. Manual review of common tags revealed that security-related terms like "vulnerability," "exploit," or "cve" were rarely used, appearing in fewer than five repositories. The term "security" itself was only used as a tag in 196 repositories.

To overcome the arbitrary nature of tags, the researchers employed Word2Vec embeddings. They calculated the cosine similarity between each unique tag's embedding and the embedding of the term "security." A similarity threshold of 0.8 (chosen after experimentation) identified 25 semantically security-related tags. This method initially classified 13,835 issues (0.13% of the total) as security-related. Further analysis showed that 7,731 of these were bot-reported (with "bot" in the username, 87.6% by Dependabot) and 6,104 were user-reported.

Scaling Security Issue Detection with Machine Learning:

Recognizing the severe under-tagging, the researchers developed an ML-based pipeline to identify untagged security issues.

1. Ground Truth Creation: A gold-standard dataset of 2,000 issues was manually labeled by two coders (Cohen's kappa 0.74). This dataset included 1,000 issues with security-related tags and 1,000 issues without, classified as "security-related" or "non-security-related."
2. Model Training and Selection: The problem was framed as a two-class classification task using issue titles and descriptions. Various pre-trained transformer models were fine-tuned, including BERT, RoBERTa, CodeBERT, FLAN-T5, and DeBERTa. Large Language Models (LLMs) like Mistral, Qwen, Meta-LLaMA, and Gemma were also evaluated in zero-shot and few-shot In-Context Learning (ICL) settings.
3. Optimal Model: The fine-tuned RoBERTa model emerged as the best performer, achieving an impressive F1-score of 0.94, 94% accuracy, and a ROC-AUC score of 0.94 on the validation set. DeBERTa showed similar metrics but produced more false positives and negatives. Manual validation on an unseen set of 100 issues confirmed RoBERTa's high accuracy, misclassifying only 4 issues.
4. Scaling Application: This validated RoBERTa model was then applied to the 9,131,800 user-reported issues that had no security-related tags. The model successfully identified an additional 1,617,738 issues (14.8% of all issues) as potentially security-related, a dramatic 114-fold increase over the initially tagged issues.

Qualitative Analysis and Theme Annotation:

To understand the lifecycle of user-reported security issues, the researchers performed a qualitative analysis on 6,104 user-reported security issues and their associated comments.

1. Phase Extraction: Issues were divided into three phases: Creation (user description), Discussion (user-developer interaction), and Resolution (last comment before closure).
2. Open Coding and Affinity Diagramming: Two researchers open-coded 100 quotes from each phase, developing codebooks (Cohen's kappa 0.85). They then used affinity diagramming to group these codes into a four-level hierarchy of themes, explaining user/developer actions and interactions (Table 4 in the paper).
3. Scaling Theme Annotation: To apply these themes to the expanded dataset of 1.6 million security issues and their 4,461,934 comments, the problem was modeled as a multi-label classification task for Level-2 themes (L-1 themes were too broad, L-3/L-4 too granular for automated classification with limited labeled data). Again, RoBERTa outperformed other models, achieving approximately 80% accuracy in identifying themes.

Factors Correlating with Resolution:

To understand what impacts issue resolution, the study developed three Generalized Linear Mixed Models (GLMMs). These models assessed the correlation of factors like CVE mentions, number of comments, weekly downloads, active maintainers, issue reproducibility, and bot involvement with three dependent variables: time to close, staleness, and successful resolution. Key findings from this analysis included the significant positive impact of CVE mentions on faster resolution and the counter-intuitive finding that reproducibility, despite its perceived helpfulness, often increased time to close and staleness.

Demo / Proof of Concept

This paper presents a rigorous data-driven understanding and a robust machine learning pipeline for identifying and characterizing security issues, rather than a traditional live demonstration of a specific exploit or a new security tool. The core "demonstration" of this work lies in the empirical evidence derived from their extensive data analysis and the validated performance of their developed machine learning models.

Specifically, the "proof of concept" is the successful application of their fine-tuned RoBERTa model, which, with an F1-score of 0.94, accurately identified over 1.6 million previously untagged security-related issues within a dataset of over 10 million GitHub issues. This effectively demonstrates the significant gap in current manual tagging practices and the potential of ML to uncover hidden security concerns at scale. The second "proof of concept" is the effective application of their theme classification model, also based on RoBERTa, which successfully annotated millions of comments and issue bodies with thematic categories related to issue creation, discussion, and resolution. While no executable tool was released as a "demo," the methodology, trained models, and the resulting dataset are available for verified researchers, proving the feasibility and impact of their approach.

Defensive Implications

The findings of this study offer crucial actionable insights for various stakeholders involved in the npm ecosystem's security. Implementing these recommendations can significantly enhance the detection, management, and resolution of security vulnerabilities.

1. Standardized and Consistent Tagging for Maintainers: The discovery that 14.8% of issues are security-related but only 0.13% are tagged as such is a glaring indictment of current practices. npm package owners and maintainers must adopt a standardized tagging system for GitHub issues. Clear guidelines for applying security-related tags (e.g., "security," "vulnerability," "CVE-related") should be enforced across repositories. This will improve the visibility of critical issues, streamline triage, and prevent important security concerns from being overlooked.
2. Developing Smarter, AI/ML-Driven Bots: Current bots are largely ineffective for security issue resolution, primarily serving to automate dependency updates or mark issues as stale. Bot developers for the npm ecosystem should pivot towards creating AI/ML-powered bots that can intelligently assist in the security workflow. These bots could:

• Automatically identify potential security issues from issue descriptions (similar to the RoBERTa model developed in this paper).
• Suggest relevant CVEs or similar issues from other repositories to reduce duplication.
• Aid in issue reproduction by suggesting potential steps or highlighting missing information.
• Perform early detection of false positives.
• Crucially, these advanced bots should be developed as free and open-source solutions to ensure broad accessibility and adoption across the diverse npm community.

1. Prioritizing CVE-Tagged Issues and Enhancing Developer Engagement: The correlation between CVE mentions and faster resolution times is a clear signal for maintainers. Issues referencing known CVEs should be automatically prioritized and receive immediate attention. Furthermore, maintainers need to increase their active engagement in issue discussions. The finding that many issues, even reproducible ones, become stale due to lack of feedback or capacity underscores the need for more proactive responses and clearer communication channels with users.
2. Improving User Reporting Quality: Users play a critical role in identifying security issues. To facilitate faster and more effective resolution, users should be encouraged and educated to:

• Provide clear, step-by-step reproduction guides and, where possible, code snippets.
• Include relevant system information (Node.js version, npm version, OS, etc.).
• Be responsive to maintainer queries for clarification.
• Conduct basic research to avoid reporting duplicate or false positive issues.

1. Tools for Security Researchers and the Community: The security research community has a vital role in developing tools that go beyond basic vulnerability detection. These tools should focus on:

• Identifying and prioritizing overlooked security issues that lack formal tags or CVEs.
• Automating the reproduction and verification of reported vulnerabilities.
• Facilitating early detection of false positives to reduce maintainer workload.
• Improving the overall accuracy and efficiency of vulnerability triage for both users and maintainers.

1. Addressing Reproducibility Challenges: The counter-intuitive finding that reproducible issues can take longer to resolve highlights a need for better processes. While users should provide reproducible steps, maintainers need better tools or workflows to quickly verify and act on these reproducible issues, especially when they lack a corresponding CVE. This might involve dedicated maintainer capacity or specialized bots to automate reproduction environments.

Key Takeaways

• Massive Under-Reporting: A staggering 14.8% of issues in npm GitHub repositories are security-related, but only 0.13% are explicitly tagged as such, demonstrating a severe systemic failure in issue classification.
• Ineffective Bots: Current bots, while active in reporting, are predominantly rule-based and largely ineffective in assisting with the resolution of complex security issues, often contributing to staleness rather than solutions.
• CVEs Drive Resolution: Security issues that include CVE mentions are resolved significantly faster, highlighting the importance of formal vulnerability identification for prioritization.
• Reproducibility Paradox: Counter-intuitively, reproducible security issues often experience longer resolution times and are more prone to becoming stale, especially if they lack a corresponding CVE or sufficient developer engagement.
• Call for Smarter Tools and Standardization: There is an urgent need for standardized tagging systems, advanced AI/ML-driven bots, and improved collaboration among all stakeholders to enhance the detection, triage, and resolution of security issues in the npm ecosystem.
• User Responsibility: Users can significantly aid resolution by providing detailed, reproducible steps and actively engaging with maintainer queries, reducing the likelihood of issues being closed due to staleness.

About the Speaker(s)

The research was conducted by Rajdeep Ghosh, Shiladitya De, and Mainack Mondal, all affiliated as Researchers at the Indian Institute of Technology (IIT) Kharagpur. Their work focuses on understanding and improving software security, particularly within open-source ecosystems. Their expertise spans areas such as large-scale data analysis, machine learning applications in software engineering, and the qualitative assessment of developer workflows and community interactions. This paper exemplifies their commitment to bridging the gap between theoretical security solutions and the practical challenges faced in real-world software development environments.