PentestGPT: Evaluating and Harnessing Large Language Models for Automated Penetration Testing

PentestGPT: Evaluating and Harnessing Large Language Models for Automated Penetration Testing

Speakers: Gelei Deng; Yi Liu; Víctor Mayoral-Vilches; Peng Liu; Yuekang Li; Yuan Xu; Martin Pinzger; Stefan Rass

Conference: USENIX Security '24

YouTube: https://www.youtube.com/watch?v=eGqjYo_vdTg

Overview

The talk "PentestGPT: Evaluating and Harnessing Large Language Models for Automated Penetration Testing" introduces an innovative, open-source solution that leverages the power of Large Language Models (LLMs) to automate the complex and human-intensive process of penetration testing. Presented by Gelei Deng, this collaborative work primarily from Nanyang Technological University, with contributions from the Singapore Agency for Science Technology and Research and Al University of New South Wales, addresses a critical gap in cybersecurity: the need for reproducible and scalable penetration testing that doesn't solely rely on the subjective expertise of individual human testers.

PentestGPT is designed to perform end-to-end automated penetration testing, capable of identifying, exploiting, and generating Proof-of-Concepts (PoCs) for vulnerabilities, and subsequently producing comprehensive penetration testing reports—all without requiring direct human expert intervention during the testing phase. With over 7,000 stars on GitHub, extensive documentation, and industrial partner integrations, the project signifies a significant step towards autonomous security assessment.

This article delves into the foundational concepts, technical architecture, and empirical results of PentestGPT, highlighting its potential to revolutionize how organizations approach vulnerability identification and risk management. By mimicking the structure of a human penetration testing team, PentestGPT aims to overcome the inherent limitations of raw LLMs, offering a robust and highly effective platform for automated security analysis.

Background

▶ Watch: Introduction to PentestGPT: automated penetration testing solution (0:00)

Penetration testing stands as one of the most widely adopted black-box approaches in the cybersecurity industry for identifying vulnerabilities within systems. The traditional procedure involves a security professional, often referred to as a "pen tester," systematically probing a target system using various testing tools. They interpret the results, identify potential vulnerable services, attempt to exploit these vulnerabilities, and if successful, proceed with privilege escalation to gain control of the system. If an exploit fails, the tester revisits earlier steps, employs different tools, and iterates through the process until a compromise is achieved or all avenues are exhausted.

Despite its effectiveness, this human-centric approach presents several significant drawbacks. A primary challenge highlighted by the speaker is the heavy reliance on the individual pen tester's domain knowledge. Different experts, when confronted with the same service or vulnerability, may employ divergent strategies, tools, and interpretations of test results. This variability leads to a lack of reproducibility in penetration testing procedures, making it difficult to guarantee consistent outcomes across different assessments or even within repeated assessments by the same individual. Furthermore, training proficient pen testers requires substantial time and resources, as they must acquire extensive knowledge across diverse systems, services, and exploitation techniques.

In response to these limitations, the industry has explored automated solutions, including those based on reinforcement learning and other traditional machine learning systems. However, the advent of Large Language Models (LLMs) has introduced a new paradigm, offering unprecedented opportunities to leverage their vast embedded knowledge and reasoning capabilities to automate complex tasks that traditionally required human cognitive input. PentestGPT emerged from the hypothesis that LLMs could potentially encapsulate and apply this expert knowledge to overcome the reproducibility and scalability issues inherent in conventional penetration testing.

Key Findings

▶ Watch: Initial LLM findings for pentesting: capabilities and limitations (3:00)

Before developing PentestGPT, the research team conducted an initial evaluation to assess the raw capabilities and limitations of existing LLMs (specifically GPT-3.5, GPT-4, and Bard/Gemini) in a penetration testing context. This evaluation employed a "human-in-the-loop" procedure, where a human operator merely executed commands provided by the LLM and fed the outputs back, without offering expert feedback or strategic input. The goal was to understand how well LLMs could perform pen testing tasks autonomously. The experiments were conducted on a synthetic benchmark comprising real-world penetration testing targets from platforms like Hack The Box and TryHackMe.

The study yielded three primary findings:

1. Tool Proficiency: Large Language Models demonstrated a surprising aptitude for utilizing common penetration testing tools. They could effectively generate commands for traditional tools such as Nmap for service scanning, Burp Suite for web vulnerability analysis, and Metasploit for exploitation. This indicated that LLMs possess a foundational understanding of tool syntax and their general application in a security context.
2. Long-Term Memory Challenges: A significant limitation observed was the LLMs' struggle with long-term memory. At the time of the study, most LLMs had restricted token sizes, typically around 4,000 or 8,000 tokens for GPT-3.5 and GPT-4, respectively. This limited context window meant that as the penetration testing process progressed and more interactions occurred, the models would "forget" earlier findings or states, hindering their ability to maintain a comprehensive understanding of the target system.
3. Preference for Recent Tasks: Related to the memory issue, LLMs tended to overly focus on the most recent conversation or interaction. The speaker noted that even after explicitly informing the LLM about vulnerable services identified earlier, subsequent interactions would cause the model to shift its focus, often forgetting to pursue or exploit those previously identified vulnerabilities. This "short-sightedness" made it challenging for raw LLMs to conduct a coherent, multi-stage penetration test that requires persistent tracking of multiple potential attack paths.

These findings underscored that while LLMs possess inherent capabilities for understanding and generating security-related commands, their architectural limitations, particularly regarding context management and memory, prevented them from performing effective, end-to-end automated penetration testing. This realization became the driving force behind the agent-based design of PentestGPT, aiming to engineer a system that could overcome these identified drawbacks.

Technical Deep Dive

▶ Watch: PentestGPT's agent-based design mimicking a human team (5:50)

To address the limitations of raw LLMs, PentestGPT employs an innovative agent-based design that intelligently orchestrates LLM interactions, mimicking the structure and workflow of a human penetration testing team. This design partitions the complex task into specialized roles, much like a leader assigns tasks to individual testers, thereby mitigating issues like token limits and short-term memory. The core architecture comprises three interconnected modules: the Parsing Module, the Reasoning Module, and the Generation Module.

The design rationale centers on creating a hierarchical structure. A "pentest leader" (represented by the Reasoning Module) maintains a high-level understanding of the overall testing objective and discovered system state, while "pentesters" (represented by the Generation Module) focus on executing specific, detailed tasks.

1. Parsing Module:

The primary challenge for LLMs in handling penetration testing outputs is the sheer volume of data generated by security tools. Tools like Nmap or vulnerability scanners often produce verbose results that can quickly exceed an LLM's token limit. The Parsing Module is specifically engineered to address this. Its function is to condense the often lengthy and detailed outputs from testing tools into concise, key information relevant for the LLM's subsequent processing. Instead of feeding the entire raw output, which would exhaust the token window and introduce irrelevant noise, this module intelligently extracts critical findings, vulnerable points, service banners, or error messages. This process ensures that only pertinent data is passed to the next stage, optimizing token usage and focusing the LLM's attention on actionable intelligence.

1. Reasoning Module:

This module serves as the "brain" or the "pentest leader" of the PentestGPT system. Its crucial role is to overcome the long-term memory and "recent task preference" issues identified in raw LLMs. The Reasoning Module maintains a Penetration Testing Tree, an abstract, natural language-formatted tree structure that represents the current understanding of the target system and the ongoing penetration test. This tree keeps track of identified services, potential vulnerabilities, exploitation paths, and overall progress.

Upon receiving the condensed information from the Parsing Module, the Reasoning Module analyzes it to:

• Update the Tree: Integrate new findings into the existing knowledge graph of the target system. This persistent state representation is vital for maintaining long-term memory.
• High-Level Reasoning: Based on the updated tree structure, the module performs high-level strategic reasoning to determine the most valuable next step in the penetration testing procedure. This involves prioritizing potential attack vectors, deciding which service to investigate next, or determining if a previously identified vulnerability should now be exploited. By maintaining a global view of the test, it prevents the system from getting stuck in local optima or forgetting earlier crucial discoveries.

1. Generation Module:

Once the Reasoning Module has determined a specific task to be performed (e.g., "scan port 80 for web vulnerabilities," "attempt to exploit CVE-XXXX-YYYY on service Z"), it passes this task to the Generation Module. This module acts as the "individual pentester" in the team. Its responsibility is to translate the high-level task into concrete, executable commands for the testing environment.

The Generation Module generates the specific command-line instructions or tool configurations that a human operator (in the initial evaluation phase) or an automated execution environment would follow. A key aspect of this module is its design to reduce hallucination. By focusing on a specific, well-defined task provided by the Reasoning Module, the Generation Module is less prone to generating irrelevant or incorrect commands, ensuring the generated actions are precise and aligned with the overall testing strategy.

The entire process operates in a continuous loop:

1. The Reasoning Module identifies a task.
2. The Generation Module produces the corresponding command.
3. The command is executed in the testing environment.
4. The testing environment returns results.
5. The Parsing Module condenses these results.
6. The condensed results are fed back to the Reasoning Module, which updates its tree and plans the next step.

This loop continues until the target system is successfully compromised, or the Reasoning Module determines that all feasible branches of the Penetration Testing Tree have been enumerated and no further progress can be made. This modular, agent-based design effectively addresses the inherent limitations of raw LLMs, transforming them into a powerful and coherent automated penetration testing system.

Demo / Proof of Concept

▶ Watch: Passing module: condensing test results to overcome token limits (6:45)

The efficacy of PentestGPT was rigorously evaluated against a set of real-world penetration testing scenarios, demonstrating a significant leap in performance compared to direct applications of raw LLMs. The benchmarks utilized platforms commonly employed by human penetration testers, providing a realistic assessment of its capabilities.

The primary demonstration and proof-of-concept involved two main evaluation scenarios:

1. Hack The Box Targets: The system was tested against a set of 10 machines from Hack The Box, a popular online platform for cybersecurity training and penetration testing challenges. At the time of the evaluation, these machines were public but had no readily available solutions online, presenting genuine "open challenges." PentestGPT successfully compromised 4 out of the 10 machines. This achievement is particularly notable because these are complex, multi-stage targets designed to challenge human experts. The speaker highlighted a total cost of $131 for conducting these tests, likely referring to the computational costs associated with running the LLMs and the testing infrastructure. This demonstrated a substantial increase in vulnerability capturing capability, with the system showing an improvement of more than 200% compared to directly using plain GPT-3.5 or GPT-4 models without PentestGPT's architectural enhancements.

1. PicoCTF 2022: PentestGPT was also deployed against challenges from PicoCTF 2022, an online Capture The Flag (CTF) competition. This environment, while open-source and with solutions available post-competition, provided a structured and diverse set of challenges to benchmark the system's problem-solving abilities. PentestGPT achieved a score of 1,500 points out of a possible 4,200. This performance placed it 24th among 248 participating teams, a remarkable feat for an automated system competing against human teams in a complex security competition.

These results collectively underscore PentestGPT's ability to not only leverage LLMs for basic tool usage but to orchestrate them into a sophisticated, goal-oriented system capable of navigating complex penetration testing scenarios. The substantial performance increase over raw LLMs validates the agent-based design and its modules—parsing, reasoning, and generation—as critical components for effective automated security assessment.

Defensive Implications

▶ Watch: Reasoning module: high-level planning and updating the testing tree (7:25)

The emergence of sophisticated automated penetration testing tools like PentestGPT carries profound implications for defensive cybersecurity strategies. As these LLM-powered systems become more capable and widely adopted, the landscape of cyber threats will undoubtedly evolve, requiring defenders to adapt their approaches.

Firstly, the primary implication is the acceleration and sophistication of attacks. Automated tools can operate 24/7, tirelessly probing for vulnerabilities, and executing exploits at speeds and scales far beyond human capabilities. This means organizations will face a higher volume of automated reconnaissance and exploitation attempts. Defenders must anticipate that known vulnerabilities, even those with lower severity, could be rapidly identified and exploited by such automated agents, demanding faster patch management cycles and more stringent configuration enforcement.

Secondly, the ability of PentestGPT to chain together multiple steps, maintain context, and adapt its strategy based on observed system states (via its Reasoning Module) implies that simple, signature-based defenses may become less effective. Automated attackers will likely exhibit more dynamic and less predictable attack patterns. Defenders will need to shift towards more advanced detection mechanisms, including behavioral analytics, anomaly detection, and AI-driven threat hunting that can identify deviations from normal system behavior rather than just known malicious signatures.

Furthermore, the rise of LLM-powered pen testing tools also opens avenues for proactive defense using similar technologies. Defenders could leverage LLMs to:

• Automate vulnerability scanning and prioritization in their own environments, mimicking an attacker to identify weak points before they are exploited.
• Generate sophisticated detection rules by training LLMs on attack patterns or system logs to identify subtle indicators of compromise.
• Simulate complex attack scenarios to test the resilience of their defensive infrastructure and incident response plans, effectively using "red team" LLMs to improve "blue team" capabilities.
• Enhance security orchestration, automation, and response (SOAR) platforms by integrating LLM-driven analysis for incident triage and response playbook generation.

Finally, the concept of reproducible penetration testing also has defensive benefits. If automated tools can consistently identify vulnerabilities, defenders can use this consistency to objectively measure the effectiveness of their security controls and track improvements over time. The $131 cost associated with the Hack The Box challenges also hints at potentially more cost-effective security assessments for organizations, which could free up resources for other defensive investments. The defensive posture must evolve from merely reacting to threats to proactively anticipating and neutralizing them with equally advanced and automated capabilities.

Key Takeaways

• LLMs show promise but have inherent limitations for pen testing: Initial evaluations revealed that while raw LLMs can utilize security tools, they struggle significantly with long-term memory, context management, and maintaining focus on earlier findings due to token limits and a preference for recent interactions.
• Agent-based architecture is crucial for effectiveness: PentestGPT's success stems from its modular, agent-based design, which mimics a human pen testing team. This allows it to overcome the limitations of raw LLMs by separating high-level reasoning from detailed command generation and robustly managing context.
• Modular design enhances LLM capabilities: The system's three key modules—Parsing (for condensing output), Reasoning (for high-level strategy and memory via a Penetration Testing Tree), and Generation (for precise command creation)—work in concert to enable end-to-end automated penetration testing.
• Significant performance improvement over raw LLMs: PentestGPT demonstrated a >200% increase in vulnerability capturing compared to plain LLMs, successfully compromising 4 out of 10 challenging Hack The Box machines and ranking 24th among 248 teams in PicoCTF 2022.
• Open-source nature fosters collaboration and advancement: Being fully open-source with extensive documentation and industrial partner integrations, PentestGPT encourages community contributions and further development, accelerating the evolution of automated security tools.
• Implications for both attackers and defenders: The rise of such automated tools necessitates that defenders enhance their capabilities with equally sophisticated, possibly LLM-driven, detection, response, and proactive security measures to counter faster and more comprehensive automated attacks.

About the Speaker(s)

The work on PentestGPT was presented by Gelei Deng, who is credited as the primary speaker for this presentation. The project itself is a collaborative effort involving a team of researchers: Gelei Deng, Yi Liu, Víctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Martin Pinzger, and Stefan Rass.

The primary affiliation for this research is Nanyang Technological University. Additionally, the project benefited from collaborations with the Singapore Agency for Science Technology and Research and Al University of New South Wales, indicating a strong academic and industrial partnership in advancing LLM-powered security solutions. Gelei Deng's presentation highlighted expertise in developing and evaluating sophisticated AI-driven systems for complex cybersecurity tasks like automated penetration testing.