From Noise to Signal: Precisely Identify Affected Packages of Known Vulnerabilities in npm Ecosystem

From Noise to Signal: Precisely Identify Affected Packages of Known Vulnerabilities in npm Ecosystem

Speakers: Yingyuan Pu

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=7Tad-Vyzkag

Overview

The npm ecosystem contains over 3 million packages with deeply nested dependency chains, and research shows approximately one quarter of all package versions depend on packages with known vulnerabilities. Yet 80% of enterprise dependencies remain unpatched for over a year despite fixes being available for over 95% of vulnerabilities. The root cause is alert overload: existing Software Composition Analysis (SCA) tools like npm audit and Dependabot perform package-level analysis that cannot determine whether vulnerable code is actually called by an application, leading to massive false positive rates.

This talk introduces VulTracer, a system that enables function-level reachability analysis at ecosystem scale to determine whether vulnerable code is actually invoked. Through a novel "analyze once, reuse many" architecture leveraging the immutability of npm package versions, VulTracer achieves an F1 score of 0.905 with perfect precision (zero false positives), a 99.4% success rate on real-world projects (versus Jelly's 37%), and reveals that over 68% of package-level vulnerability alerts are false positives -- the vulnerable code is never actually called.

Background

▶ Watch: Background: npm's 3 million packages and supply chain risk (0:00)

The npm ecosystem's structure is characterized by deep nesting and intricate dependency chains, meaning a single vulnerability in a foundational package can have massive downstream impact. A remote code execution vulnerability in the resolve package, for instance, put over 285,000 GitHub repositories at risk. This reality has made software supply chain security a critical concern.

To manage this risk, the industry relies on SCA tools that report vulnerable dependencies in a project's dependency graph. However, these tools create a critical question for developers: "Am I really affected?" The answer requires understanding whether the vulnerable code path is actually reachable from the application's entry points -- a determination that package-level tools fundamentally cannot make.

Existing approaches to function-level analysis face three major challenges. First, scalability: tools like Jelly (the state-of-the-art JavaScript call graph analyzer) use whole-program analysis that is computationally prohibitive, achieving only a 37% success rate under a 4GB memory limit with most failures caused by out-of-memory errors. Second, JavaScript's dynamic nature: functions as first-class citizens, higher-order functions, and callbacks obscure call targets, making static analysis inherently imprecise. Third, dual module systems: CommonJS with mutable exports and dynamic require expressions versus ECMAScript modules with static imports, plus interoperability between them, creates additional analytical complexity.

Key Findings

▶ Watch: Key insight: npm package immutability enables analyze-once-reuse-many (4:00)

The ecosystem-scale study -- the largest function-level vulnerability impact measurement on npm to date -- covers 3.2 million unique packages, 34.7 million distinct versions, and 900 million dependency links across 27 surveyed CVEs. The results fundamentally change our understanding of vulnerability propagation:

• 68% of package-level alerts are over-approximations: At both single-hop and multi-hop levels, over two-thirds of alerts from package-level tools do not represent real threats. Roughly two out of every three alerts that developers receive are noise.
• Up to 90%+ reduction for specific CVEs: For some vulnerabilities, the reduction from package-level to function-level analysis is dramatic -- from over 21,000 affected libraries at package level down to just 1,000 at function level.
• API breadth drives attenuation: For broad API libraries like lodash (242 functions), the vulnerable function ranks only 49th in usage, accounting for just 0.3% of all calls. In contrast, narrow API libraries like debug have vulnerabilities in core functions, with over 98% of importing packages actually calling the vulnerable function.
• 20-40% of dependencies are unused: Over 20% of direct dependencies in high-impact CVEs and over 40% globally declare a vulnerable dependency in package.json but never import any code from it. For lodash alone, over 131,000 packages declare the dependency but never use it.
• True propagation is shallow: While package-level analysis shows vulnerabilities traveling up to 32 hops with an average of 7.6 hops, function-level analysis reveals actual propagation averages just 0.71 hops with a maximum of 8. Over 96% of truly affected packages are within just 4 hops.

Technical Deep Dive

▶ Watch: Results: F1 of 0.905 with perfect precision vs Jelly's 0.731 (10:00)

VulTracer's architecture is built on a key insight: npm package versions are immutable. Once published, a package version's content never changes. This enables an "analyze once, reuse many" model where rich analysis results are precomputed and cached for each package version.

The system operates in three phases:

Phase 1 - Rich Semantic Graph (RSG) Generation: Each package version is analyzed in isolation to produce an RSG that captures not only internal function calls but also boundary information. Traditional call graphs discard boundary information, losing semantic context of external calls and failing to model the public API. The RSG addresses this with three vertex types: programmatic entities (modules and functions), invocation points (refined as vertices rather than just edges), and export anchors (modeling the public API). It also defines three edge types: lexical nesting, call resolution edges (internal or external), and export resolution edges. The critical innovation is making external calls and API exports explicit nodes in the graph.

Phase 2 - Interface Contract Extraction: From each RSG, formal interface contracts are extracted consisting of an export manifest (mapping API paths to exposed functions) and an import manifest (cataloging external dependencies and mapping use paths to invocation points). These contracts serve as machine-readable specifications that enable semantic matching during composition and decouple internal implementation from external interface.

Phase 3 - Compositional Synthesis: When analyzing a specific application, the algorithm performs a reverse topological sort of the dependency graph, ensuring each package's dependencies are resolved before it is processed. For each package, its precomputed RSG is loaded and composed with already-resolved dependency graphs. The core operation is interface stitching -- matching use patterns from a caller's import manifest with definition paths from a callee's export manifest, handling both direct calls and transitive reexports.

VulTracer achieves an F1 score of 0.905 against ground truth from dynamic analysis with Node profiling, compared to Jelly's 0.731. Critically, VulTracer maintains perfect precision of 1.0 across all benchmarks -- zero false positives. An ablation study confirms that removing formal contracts causes a 45% drop in inter-package coverage, making it the most critical component.

Demo / Proof of Concept

▶ Watch: 68% of package-level alerts are false positives (12:00)

The practical evaluation demonstrates VulTracer's effectiveness on real-world projects. Using the same 12 applications from the GEM benchmark, npm audit reported 532 vulnerable packages and 75 propagation paths. VulTracer reduced this to just 53 alarms, and manual verification confirmed 21 true positives with only 2 false positives.

For scalability testing, VulTracer was compared against Jelly on 99 real-world projects affected by a single CVE under a 4GB memory limit. Jelly succeeded on only 37% of packages (most failing from memory exhaustion), while VulTracer achieved a 99.4% success rate. Critically, VulTracer's analysis time remains stable regardless of project complexity, while Jelly's time degrades significantly as code size increases.

The ecosystem-scale study covering 3.2 million packages and 27 CVEs represents the largest study of its kind, transforming the understanding of how vulnerabilities actually propagate through the npm dependency graph.

Defensive Implications

▶ Watch: True propagation is shallow: 0.71 hops average vs 7.6 at package level (16:00)

VulTracer has immediate practical implications for development and security teams:

• Triage prioritization: By reducing false positives by 68% or more, security teams can focus patching efforts on dependencies where the vulnerable code is actually called, addressing the alert fatigue that causes 80% of vulnerabilities to remain unpatched for over a year.
• CI/CD integration: The precomputed analysis model means VulTracer can operate at CI/CD speed, providing function-level reachability results without the computational overhead of whole-program analysis for each build.
• Dependency hygiene: The finding that 20-40% of declared dependencies are never imported suggests that automated dependency pruning could eliminate a significant portion of supply chain risk with zero functional impact.
• Risk modeling: The revelation that true vulnerability propagation averages just 0.71 hops (versus 7.6 at package level) fundamentally changes how organizations should model supply chain risk -- the actual blast radius is far smaller and shallower than package-level analysis suggests.
• Vulnerability management SLAs: Organizations can set more realistic patching timelines based on actual reachability rather than theoretical exposure, reserving urgent response for the roughly one-third of alerts that represent genuine risk.

Key Takeaways

• Over 68% of package-level vulnerability alerts in npm are false positives -- the vulnerable code is never actually called by the application
• VulTracer's "analyze once, reuse many" approach leverages npm package immutability to achieve function-level analysis at ecosystem scale
• Perfect precision (zero false positives) with an F1 of 0.905 dramatically outperforms Jelly's 0.731 while achieving 99.4% success rate versus Jelly's 37%
• True vulnerability propagation in npm is shallow (average 0.71 hops, max 8) versus the deep propagation (average 7.6 hops, max 32) suggested by package-level analysis
• API breadth is the strongest predictor of whether a vulnerability actually propagates: broad libraries like lodash see minimal real impact while narrow API libraries like debug show 70%+ propagation rates
• Over 131,000 packages declare lodash as a dependency but never import any code from it, highlighting massive dependency hygiene issues across the ecosystem

About the Speaker(s)

The presentation was delivered by a representative from Tsinghua University on behalf of the authors. The research was conducted by Lingling from a technology research institute and Yatungu from Tsinghua University. The work addresses what is considered one of the top five challenges in software supply chain security: patching paralysis from imprecise alerts.