From Chatbots to Phishbots?: Phishing Scam Generation in Commercial Large Language Models
Sayak Saha Roy, Poojitha Thota, Krishna Vamsi Naragam, Shirin Nilizadeh
IEEE Symposium on Security and Privacy 2024 · Day 1 · Continental Ballroom 4
The advent of commercial Large Language Models (LLMs) such as ChatGPT, Claude, and Bard has revolutionized various industries, offering unprecedented capabilities in content generation, data analysis, and even source code production. These models are increasingly accessible, often free, and require minimal technical expertise to operate, making them powerful tools in the hands of both legitimate users and potential adversaries. This talk, presented by Sayak Saha Roy and his co-authors at IEEE S&P, delves into a critical security concern: the potential for these sophisticated LLMs to be exploited for the automated generation of highly effective phishing scams.
AI review
This research exposes a critical, scalable threat: commercial LLMs can be weaponized to generate sophisticated, evasive phishing attacks by systematically bypassing content moderation. The work demonstrates automated prompt generation for these attacks and proposes a highly effective, context-aware detection model, providing a vital countermeasure.