Analyzing 100 Hacker's scripts using Claude Sonnet and ChatGPT

Analyzing 100 Hacker's scripts using Claude Sonnet and ChatGPT

Speakers: Arik Atar, Senior CTI Researcher, Radware

Conference: DefCamp

YouTube: https://www.youtube.com/watch?v=954qJamqRAc

Overview

In an insightful DefCamp presentation, Arik Atar, a Senior CTI Researcher at Radware, unveiled groundbreaking research on the efficacy of Artificial Intelligence (AI) in analyzing real-world credential stuffing scripts. Atar, who dedicates his professional life to immersing himself in underground hacker forums and collecting Open-Source Intelligence (OSINT) via 86 distinct hacker profiles established since 2017, sought to answer a pivotal question: Can AI models like ChatGPT and Claude Sonnet effectively automate the complex task of analyzing credential stuffing scripts, a core component of his work for over a decade?

The research involved the meticulous collection of 100 actual credential stuffing scripts—those that had been used in attacks, subsequently expired, and then released for free by threat actors. This unique dataset provided a rare glimpse into the practical methodologies employed by attackers. Atar's findings, while presenting a mixed bag of optimism and challenges, offer invaluable insights for security professionals contemplating the integration of AI into their threat intelligence and defensive strategies. This article delves into the methodologies, key findings, and defensive implications of this research, providing a comprehensive analysis of the evolving landscape of credential stuffing and the role of AI in combating it.

Background

▶ Watch: Introduction: Automating credential stuffing script analysis with AI (0:00)

Credential stuffing remains one of the most pervasive and damaging bot attacks, consistently ranking just below Distributed Denial of Service (DDoS) attacks in prevalence. The underlying premise of this attack is deceptively simple yet highly effective: threat actors leverage lists of compromised usernames and passwords obtained from various data breaches, then programmatically attempt to use these credentials against other online services. This strategy exploits a fundamental human vulnerability: the widespread practice of password reuse. Statistical data cited by Atar indicates that a staggering 84% of adults recycle the same password across 12 or more different online platforms, creating a fertile ground for credential stuffing success. Beyond mere account takeover, these attacks are increasingly used as an initial access vector for more sophisticated operations, including ransomware deployments against organizations.

The anatomy of a credential stuffing script is typically multi-faceted, designed to mimic legitimate user behavior and bypass rudimentary bot detection mechanisms. Atar's analysis revealed that while scripts can vary significantly in length—from a mere 3 lines to an extensive 368 lines—most adhere to a three-step process:

1. Pre-authentication: This initial phase, present in 100% of the analyzed scripts, focuses on gathering necessary session information. Attackers cannot simply hit a login page directly. They must first obtain CSRF tokens and authentication cookies to appear as a legitimate human user initiating a session. This often involves navigating to a URL, parsing the response, and extracting specific values.
2. Cookie Replay: Once authentication cookies and tokens are acquired, the script proceeds to replay these values. This replay attack is crucial for maintaining the illusion of a continuous, legitimate user session as the bot moves towards the actual login attempt.
3. Credential Insertion: This is the core of the attack, where the compromised username and password pairs are programmatically inserted into the login form, often alongside other parameters like CAPTCHA form IDs.

These scripts are not written in conventional programming languages like Python; instead, they often use proprietary scripting languages tailored for specific botting tools. A prominent example highlighted in the talk is Lolly Script, which is the language for the widely used botting tool known as Silver Bullet.

Silver Bullet is an open-source tool that has become a staple in the credential stuffing ecosystem. Its user-friendly interface, constant updates, and powerful capabilities make it highly attractive to threat actors. Key features of Silver Bullet include:

• Massive Bot Generation: Capable of generating thousands of concurrent bots to target any application, API endpoint, or mobile device.
• Sophisticated Spoofing: Can effectively spoof various user agents and device identities, rendering JavaScript (a common bot mitigation technique), and utilizing libraries like Puppeteer to emulate browser behavior convincingly.
• Configurability: Allows attackers to load proxylists (lists of proxy IPs to evade IP-based rate limiting), wordlists (lists of common passwords like "123456" or "Iloveyou3," which statistically have a ~14% success rate even with random users), and specialized configs (the credential stuffing scripts themselves).
• Integration: Can interact with various cracking software, further enhancing its utility for account compromise.

The threat actor ecosystem behind credential stuffing is a tripartite structure, each role distinct yet interdependent:

1. The Config Developer: This is the true "brain" of the operation. A skilled hacker, they perform reverse engineering on target websites and applications. This involves understanding the victim's business logic, identifying third-party defenders (e.g., bot mitigation systems), mapping their bypasses, and pinpointing rate limits. Their output is the meticulously crafted script, or "config."
2. The Account Cracker: Less technically proficient, this individual purchases configs (often for around $50 in Bitcoin Cash) and simply runs the automated tool. Their objective is to generate breached accounts.
3. The Account Buyer: This final party monetizes the breached accounts. Buyers range from individuals seeking free access to premium services (e.g., Spotify) to sophisticated financial fraudsters. These fraudsters exploit accounts for monetary gain, draining balances, converting loyalty points or airline miles into cash, or ordering goods. A critical and often overlooked consequence is that successful attackers frequently add their own two-factor authentication (2FA) mechanisms to compromised accounts, permanently locking out the original user and making recovery exceedingly difficult, if not impossible.

The focus of Atar's research, therefore, was primarily on the config developer, understanding their methods and the intricate logic embedded within their scripts. The lifecycle of these attack scripts is rapid and dynamic:

1. Creation: Configs are developed in response to demand for specific targets (e.g., Perplexity, Salesforce). This involves testing with active, aged accounts (less likely to be monitored) and new accounts (to check for grace periods), followed by intensive reverse engineering via methods like man-in-the-middle interception (using tools like Burp Suite) or decompiling Android applications. The goal is to understand rate limits, account actions per minute per IP, the victim app's business logic, and conditions that trigger failure or detection. This stage can involve inserting deliberate delays (e.g., a 35-second pause) to bypass specific detection triggers.
2. Sale: Once created, configs are sold, typically having a functional lifespan of only 3 to 4 weeks before targets patch their vulnerabilities. Sales are often limited to maintain a low "fingerprint" and avoid widespread detection.
3. Free Release: After their success rate diminishes, approximately 20% of configs are released for free. This acts as a "premium model" for threat actors, allowing aspiring "script kiddies" to taste success, fostering excitement, and encouraging future purchases. These free configs remain perpetually available in underground forums and Telegram channels, serving as invaluable intelligence for researchers like Atar.

Key Findings

▶ Watch: Threat actors: config developer, account cracker, account buyer (10:00)

Arik Atar’s analysis of 100 credential stuffing scripts, with 80% of the heavy lifting performed by AI models like Claude Sonnet and ChatGPT and 20% by human analysts for training, revealed a dynamic and evolving threat landscape. The research illuminated several critical aspects of modern credential stuffing attacks, from the sophistication of threat actors to the most targeted industries and the advanced techniques employed.

One of the most striking findings was the concentration of expertise among advanced threat actors. Atar observed that just three individuals were responsible for nearly half of the 100 scripts analyzed over a six-month period. This indicates a significant knowledge gap between the top-tier config developers and the rest, with these few individuals effectively dominating the market. Crucially, Atar confirmed that all of these advanced threat actors are actively leveraging AI, having been observed in Telegram channels discussing and asking questions about how to use AI in their operations. This underscores AI's growing dual-use potential, empowering both defenders and attackers.

The research identified distinct specializations among these elite config developers:

• SVB config maker (Silver Bullet): This actor specializes in bypassing security for AI platforms such as ChatGPT, Claude, Perplexity, and Gemini. Their expertise includes advanced CAPTCHA handling, bypassing service redirects, and sophisticated CSRF token handling, which accounted for 31% of their work.
• Mobile API Guy: As the name suggests, this individual focuses on exploiting mobile APIs, particularly targeting Samsung and other media accounts.
• Magic CKG: This actor is dedicated to compromising Microsoft cloud services, including Office 365 and other corporate SaaS programs. Their primary target audience includes ransomware groups seeking initial access to corporate networks, signifying a shift from individual consumer accounts to high-value enterprise targets.

The target industries for credential stuffing also presented clear trends:

• Tech (specifically AI platforms) emerged as the number one target industry during the research period. This spike was directly attributed to the rise of AI tools; without them, the tech sector would rank fourth. This highlights the immediate attractiveness of new, high-value online services for attackers.
• Financial Services (FSI) and Government (Gov) sectors followed closely, due to the high potential for exploitation.
• Travel saw a seasonal spike, particularly during the European travel season. Attackers target travel accounts to exploit accumulated mileage, loyalty points, or active credit cards for fraudulent bookings and resale.
• Retail and E-commerce experienced a spike in January 2025. This was linked to configs developed during the 2024 holiday season, which were then released for free approximately a month later, allowing researchers to capture them. These accounts are prized for gift card points, which are easily abused and less traceable than credit card transactions.
• Media (Mobile Carriers) accounts were also significant targets, primarily for SIM swapping attacks. By gaining control of a mobile carrier account, attackers can request new SIM cards or port numbers, thereby intercepting 2FA codes and completely locking out legitimate users.

Atar’s research also shed light on two distinct attack tactics: "hit-and-run" versus "low-and-slow." Every script includes suggested bot counts and a maximum "combos per minute" (CPM) rate, which reflects the config developer's understanding of the target's rate limits.

• Media accounts were subjected to hit-and-run attacks, averaging 107 concurrent bots. These are rapid, high-volume attacks designed to compromise tens of thousands of accounts in a short five-minute window, typical for SIM swapping.
• Financial services, conversely, were targeted with low-and-slow attacks, using significantly fewer bots. This tactic is necessitated by the stringent IP rate limits and security regulations prevalent in the financial sector. Attackers are content with compromising 10 to 100 accounts per day, running their operations for days or even weeks, meticulously staying below detection thresholds.

A critical evolution in credential stuffing is the prevalence of API targeting. A staggering 84% of the analyzed scripts explicitly targeted API endpoints at various stages of the attack.

• Post-authentication API targeting was observed in 50% of these cases. After successfully breaching an account, attackers would hit specific APIs to extract sensitive information such as account balances, full PII, or the last four digits of credit cards. This data is crucial for account buyers to determine the value of the compromised account, effectively transforming credential stuffing into a PII breach.
• Alarmingly, 20% of API-targeting configurations specifically sought out older API versions. For instance, a script targeting Bluejet was found to use V2 of its API instead of the current V6. These older APIs are often neglected, unmonitored by SOC teams, and less likely to be updated by third-party providers, presenting an "out of sight, out of mind" vulnerability that hackers readily exploit.

Finally, the research highlighted the increasing sophistication of evasion techniques, particularly multi-device identity spoofing. While device spoofing is not new, Atar found that 62% of attacks dynamically transitioned between multiple device identities (Android, iPhone, Mac, Windows) within a single attack session. The attackers' strategy is to identify and exploit the "weakest" device in terms of security for each specific phase of the attack. For example, a CAPTCHA might be solved using a Windows user agent, but the resulting token is then replayed using a Mac user agent. The most successful transition observed was from Windows to mobile (32% of cases) to bypass detection. This technique has become so prevalent that it's no longer an advanced tactic but a mainstream evasion method, indicating a gap in how defenders holistically view user sessions across different device fingerprints.

Technical Deep Dive

▶ Watch: Attack script lifecycle: creation, sale, and free release (13:00)

The technical intricacies of modern credential stuffing attacks, as revealed by Atar's research, underscore a sophisticated understanding of web application logic and bot detection mechanisms by config developers. The core of these operations lies in the custom-built scripts, often written in specialized languages like Lolly Script, which are executed by advanced botting tools such as Silver Bullet.

A typical Lolly Script, as demonstrated, orchestrates a multi-stage attack flow. The initial phase, pre-authentication, is critical for establishing a legitimate-looking session. This involves sending HTTP requests to the target application's login page or an associated endpoint to retrieve essential session parameters. These parameters include CSRF (Cross-Site Request Forgery) tokens and authentication cookies. Modern web applications heavily rely on these tokens to prevent automated submissions and maintain session integrity. The script must parse the HTML or JSON response to extract these dynamic values, which might be embedded in hidden form fields or HTTP headers. The goal is to appear as a genuine user initiating a session, bypassing initial checks that might flag direct login attempts.

Following pre-authentication, the script enters the cookie replay phase. Here, the extracted authentication cookies and tokens are meticulously re-inserted into subsequent HTTP requests. This maintains the illusion of a continuous, authorized session, allowing the bot to navigate through the application's workflow without being flagged for session inconsistencies. This technique is a cornerstone of many web-based attacks, exploiting how stateless HTTP protocols manage state through cookies.

The final and most direct step is credential insertion. This involves constructing an HTTP POST request to the login endpoint, embedding the compromised username and password. Crucially, the script also includes any other required form parameters, such as a CAPTCHA form ID or other dynamic fields that the application might demand. The example script for "Mobile Mouth OAU" illustrated this straightforward yet effective sequence, which, despite its simplicity, had been used in real-world attacks for months.

The Silver Bullet tool itself is a marvel of attacker-centric engineering. Its open-source nature and constant development by a community of threat actors ensure it remains cutting-edge. Beyond basic bot generation, Silver Bullet incorporates advanced capabilities to evade detection:

• JavaScript Rendering: Many bot mitigation systems rely on JavaScript execution in the browser to detect automated agents. Silver Bullet's ability to render JavaScript, often by integrating headless browser functionalities (like those provided by Puppeteer), allows it to execute client-side code and mimic a full browser environment, thus bypassing these checks.
• API and Mobile Spoofing: The tool can convincingly spoof various user agents, operating systems, and device types (e.g., iPhone, Android, Mac, Windows). This is critical for the multi-device identity spoofing technique, where the bot dynamically changes its perceived device fingerprint mid-attack.
• Proxy Management: Silver Bullet integrates proxylists, allowing attackers to rotate IP addresses frequently. This circumvents IP-based rate limiting and blacklisting, making it difficult for defenders to block the attack based on source IP.
• Wordlists and Configs: The tool's ability to ingest both general wordlists (for common password spraying) and specific configs (the credential stuffing scripts) makes it highly versatile for various attack scenarios.

The config developer's process is a testament to their reverse engineering prowess. When targeting a new application, they embark on a methodical reconnaissance phase:

1. Victim Selection and Active Account Testing: Developers often request "active accounts" from their networks. These accounts, typically six months old or more, are less likely to be under intense scrutiny compared to newly registered accounts, which might trigger grace period checks or immediate fraud detection.
2. Reverse Engineering Techniques:

• Interception (Man-in-the-Middle): Using tools like Burp Suite, developers intercept and analyze all HTTP/S traffic between a legitimate user's browser and the target application. This reveals the exact sequence of requests, headers, parameters, and responses, including how CSRF tokens, cookies, and CAPTCHAs are handled.
• Decompiling Applications: For mobile targets, decompiling Android (or sometimes iOS) applications allows developers to inspect the source code, understand the underlying API calls, and uncover hidden logic or vulnerabilities.

1. Mapping Business Logic and Rate Limits: The goal is to understand the application's "business logic"—the intended flow of a legitimate user. This involves identifying API endpoints, their expected parameters, and critical actions. Concurrently, developers meticulously map rate limits (e.g., how many login attempts per minute per IP), account actions (e.g., how many password resets per hour), and conditions that cause the application to fail or trigger detection. This deep understanding allows them to craft scripts that operate just below detection thresholds. For example, a 35-second timeout might be embedded in a script if the developer discovers that a specific detection mechanism resets or drops after that duration.

The script lifecycle itself is a tactical game. A config's effective lifespan is often limited to 3 to 4 weeks because targeted organizations eventually patch the vulnerabilities or update their bot mitigation. To avoid detection and maintain a low "fingerprint," config developers limit the number of copies sold, preventing the attack from scaling too widely too quickly. Once a script's success rate declines, it is often released for free, serving as a "premium model" to attract new, less experienced "script kiddies" into the ecosystem, ensuring a continuous supply of potential customers for future, more advanced configs.

The specific TTPs identified highlight this technical sophistication:

• AI Platform Bypass: The SVB config maker's focus on AI platforms involved intricate understanding of their specific CAPTCHA implementations, service redirects, and how they handle tokens to maintain session continuity. This requires deep knowledge of each platform's unique security architecture.
• Targeting Older APIs: The discovery of scripts targeting V2 of an API when V6 was current (e.g., Bluejet) is a critical insight. Older API versions are often deprecated but not fully decommissioned, or they may be maintained by third-party services that are slow to update. These endpoints are typically unmonitored by security teams, offering a stealthy attack vector.
• Multi-Device Spoofing Mechanics: The technique of solving a CAPTCHA on a Windows user agent and then replaying the token with a mobile user agent is not merely about changing a header. It implies an understanding of how different device types are perceived by the target application's security stack. Attackers exploit potential weaknesses or inconsistencies in detection logic that might apply stricter rules to desktop user agents versus mobile ones, or vice versa, depending on the specific phase of the login process. This dynamic manipulation of identity within a single session demonstrates a sophisticated approach to evading behavioral analytics that expect consistent user fingerprints.

Demo / Proof of Concept

▶ Watch: API targeting for PII breach and older API versions (26:00)

While Arik Atar's presentation was primarily an analytical exposé rather than a live demonstration of an active attack, he effectively provided "proof of concept" through compelling visual evidence drawn directly from his research.

He showcased a screenshot of the Silver Bullet botting tool (specifically version 1.1.4), providing the audience with a direct view of the attacker's operational interface. This image vividly displayed key metrics such as "total tries" and "total hits" (representing breached accounts), along with running logs of successful and failed attempts. It also highlighted the tool's capability to launch concurrent sessions, with an example showing 60 bots running simultaneously against an LS Hosting VPS target. This visual underscored the scale and automation capabilities of such tools in the hands of threat actors.

Furthermore, Atar presented a sample of a Lolly Script, the proprietary language used by Silver Bullet. He specifically chose the shortest script from his dataset to illustrate the fundamental three-step process of pre-authentication, cookie replay, and credential insertion in an easily digestible format. This provided a concrete example of the code being analyzed by his AI models.

Finally, to contextualize the commercial aspect of this threat, he displayed screenshots of Telegram channel advertisements for credential stuffing configs. These ads, often boasting features like "100% custom work," "fast turnaround," "anti-detection optimized," and "24-hour support," revealed the professionalized nature of the config development market and the explicit use of AI (e.g., ChatGPT) for crafting persuasive ad copy. These artifacts collectively served as a powerful "demo" of the real-world tools, scripts, and marketing tactics employed by the threat actors under investigation.

Defensive Implications

▶ Watch: Multi-device identity spoofing for advanced evasion (29:00)

The detailed analysis of 100 credential stuffing scripts by Arik Atar and his AI models provides critical intelligence for defenders. The evolving TTPs and the professionalization of the credential stuffing ecosystem necessitate a multi-layered and adaptive defensive strategy.

1. Mandatory Two-Factor Authentication (2FA): This remains the single most effective deterrent against credential stuffing. As Atar highlighted, attackers immediately add their own 2FA to compromised accounts, locking out legitimate users permanently. Implementing mandatory 2FA across all critical services significantly raises the bar for attackers, as even with valid credentials, they cannot gain full access without the second factor. Organizations should educate users on its importance and simplify its adoption.

1. Comprehensive API Monitoring and Management: The finding that 84% of scripts target API endpoints, with 20% specifically targeting older API versions, is a stark warning. Organizations must:

• Map all API endpoints: Conduct thorough inventories of all active APIs, including those used by third-party integrations or legacy systems.
• Monitor authentication endpoints rigorously: Implement real-time monitoring for unusual activity on login, password reset, and account modification APIs.
• Prioritize patching and decommissioning of older APIs: Assume that unmonitored or outdated APIs will be discovered and exploited. Proactively update or retire old versions (e.g., move from V2 to V6) to remove these blind spots.
• Scrutinize Post-Authentication API Calls: Pay close attention to API calls made after a successful login, especially those designed to extract PII, financial data, or loyalty points. Unusual volume or patterns of such calls could indicate a successful credential stuffing attack turning into a data breach.

1. Holistic Session Analysis and Advanced Bot Mitigation: The prevalence of multi-device identity spoofing (62% of attacks) indicates that simple device fingerprinting is no longer sufficient. Defenders need:

• Behavioral analytics: Employ bot mitigation solutions that analyze user behavior across an entire session, looking for inconsistencies like rapid transitions between device types, unusual navigation patterns, or atypical time delays (e.g., the 35-second pause).
• Cross-device tracking: Develop or integrate systems that can correlate activities across different device fingerprints for a single user session, flagging suspicious changes. A human would never log in from Windows, solve a CAPTCHA, and then replay a token from a mobile device within seconds.
• Contextual awareness: Understand the typical user journey for your application and flag deviations that attackers might exploit, such as solving a CAPTCHA on a less secure platform to bypass stronger controls on another.

1. Adaptive Rate Limiting and Business Logic Protection: While financial services use strict IP rate limits, attackers adapt with "low-and-slow" tactics.

• Dynamic rate limiting: Implement adaptive rate limiting that considers factors beyond just IP address, such as user agent, geographical location, and historical behavior.
• Business logic protection: Proactively reverse-engineer your own application's business logic to identify potential bypasses or vulnerabilities that config developers might exploit. Understand how attackers analyze your application's flow, including potential timeouts or grace periods that they can weaponize.

1. Proactive Threat Intelligence and Rapid Response: The short 3-4 week lifespan of active configs before patching highlights the need for agility.

• Monitor underground forums: Engage in CTI activities to monitor hacker forums and Telegram channels for discussions, ads, and free releases of configs targeting your industry or specific applications. Tools like those used by Atar can be invaluable here.
• Rapid patching and mitigation: Establish processes for quickly analyzing new TTPs and deploying patches or mitigation strategies within weeks, not months.
• Understand attacker motivations: Recognize that attackers specialize (e.g., AI platforms, mobile APIs, Microsoft cloud) and tailor your defenses to these specific threats if you are a target in those sectors.

By adopting these defensive postures, organizations can move beyond reactive measures to proactively counter the sophisticated and constantly evolving threat of credential stuffing, turning the intelligence gleaned from hacker scripts into actionable security improvements.

Key Takeaways

• AI as a Dual-Use Tool: Artificial intelligence, exemplified by Claude Sonnet and ChatGPT, is proving to be an indispensable tool for both CTI researchers in analyzing complex attack scripts and for advanced threat actors in developing sophisticated bypasses and streamlining their operations.
• Specialized Threat Actors Drive Innovation: The credential stuffing landscape is dominated by a few highly skilled config developers who specialize in specific targets (e.g., AI platforms, mobile APIs, Microsoft cloud services) and continuously innovate evasion techniques, including advanced CAPTCHA and CSRF token handling.
• Evolving Attack Tactics: Modern credential stuffing attacks increasingly leverage sophisticated TTPs, including widespread targeting of API endpoints (especially older, unmonitored versions and post-authentication data exfiltration for PII), and dynamic multi-device identity spoofing within a single attack session to bypass detection.
• Critical Importance of 2FA: Two-factor authentication remains the most effective defense against credential stuffing, as attackers actively bypass or re-configure it to lock out legitimate users, highlighting its non-negotiable role in account security.
• Proactive API Security is Paramount: Defenders must meticulously map and continuously monitor all API authentication endpoints, particularly those that are older or less frequently used, as these represent prime, often overlooked, targets for attackers.
• Holistic Session Monitoring and Rapid Response: Effective defense requires a holistic view of user sessions to detect multi-device transitions and other behavioral anomalies, coupled with agile threat intelligence gathering from underground forums and rapid patching cycles to counter the short lifespan of active attack configs.

About the Speaker(s)

Arik Atar is a Senior CTI Researcher at Radware, specializing in the intricate world of cyber threat intelligence. His work involves deep dives into underground hacker forums and Telegram channels, where he meticulously collects Open-Source Intelligence (OSINT). Since 2017, Atar has established and maintained 86 distinct profiles to interact with threat actors, gaining unique insights into their methodologies and tools. He then collaborates with Radware's R&D and marketing teams to analyze this intelligence, translating real-world hacker tactics into enhanced security products and strategies. Atar's research focuses on understanding the evolving threat landscape to build more effective defenses against sophisticated cyberattacks.