We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs

Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala

34th USENIX Security Symposium · Day 1

This distinguished paper, "We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs," delves into a critical and emerging threat to the software supply chain: **package hallucinations**. Authored by Joseph Spracklen and colleagues from the University of Texas at San Antonio, the University of Oklahoma, and Virginia Tech, this research illuminates how Large Language Models (LLMs) used for code generation can inadvertently recommend non-existent software packages. These erroneous recommendations create a novel vector for **package confusion attacks**, where malicious actors can register packages with hallucinated names, tricking unsuspecting developers into downloading and integrating harmful code into their projects.

AI review

Solid empirical work that finally quantifies a real supply chain threat everyone's been hand-waving about. The 576K sample corpus across 16 models is no joke, and the mitigation comparisons—especially the fine-tuning vs. code quality tradeoff—are the kind of uncomfortable findings practitioners actually need.