Offensive Security - Automation and AI

Offensive Security - Automation and AI

Speakers: Andrei Cristea (Manager for Offensive Security, Booking Holdings Romania); Carmen Ruset (Senior Penetration Tester, Booking Holdings Romania); Constantin-Danut Covrig (Senior Penetration Tester, Booking Holdings Romania)

Conference: DefCamp

YouTube: https://www.youtube.com/watch?v=5A6l0n5YlHk

Overview

In an era defined by an ever-expanding digital attack surface and the relentless pace of cyber threats, the DefCamp talk "Offensive Security - Automation and AI" by Andrei Cristea, Carmen Ruset, and Constantin-Danut Covrig from Booking Holdings Romania offered a timely exploration into how artificial intelligence and automation are revolutionizing offensive security practices. The speakers shared their practical experiences and insights, moving beyond the hype to demonstrate tangible applications of these technologies across the entire offensive security lifecycle—from threat intelligence and reconnaissance to threat modeling, testing, and reporting.

The presentation underscored a critical shift in the cybersecurity landscape: the need for security teams to augment their capabilities with intelligent automation, not to replace human expertise, but to enhance it. As companies grapple with increasing asset complexity and shrinking windows for defense, the traditional manual approach to offensive security is proving insufficient. This talk provided a roadmap for leveraging AI and automation to achieve scale, consistency, and responsiveness, ultimately enabling security professionals to focus on higher-value, creative problem-solving rather than repetitive tasks.

Background

▶ Watch: Welcome, team, and offensive security evolution (0:00)

The journey of offensive security has evolved significantly, from esoteric "dark arts" to a sophisticated service industry driven by compliance and the sheer volume of attacks. The advent of bug bounties further accelerated this evolution, recognizing the need for dedicated, long-term effort to secure applications. Today, the industry stands at the precipice of an AI cohabitation era, where human and artificial intelligences collaborate. However, persistent challenges remain. Companies continue to struggle with accurate asset inventory, often unaware of what they have exposed or the sensitive data (PII, PCI) residing within these assets. The sheer scale of modern infrastructure, spanning cloud and on-premise environments, makes keeping up a monumental task.

A critical issue highlighted by the speakers is the accelerating "time to exploit," which has dramatically reduced to "minus one day." This alarming metric signifies that attackers are often exploiting vulnerabilities even before patches are developed, rendering reactive manual defenses largely ineffective. While manual methods remain crucial for deep, creative, out-of-the-box thinking, they simply cannot match the pace required for initial discovery and continuous monitoring of an exponentially growing attack surface. This imbalance necessitates a strategic shift towards automation and AI to augment security processes, allowing human experts to provide depth and value where it truly counts. Offensive security is no longer just a skill, but an orchestration of tools and automation.

Key Findings

▶ Watch: Automated reconnaissance using Shodan and CT logs (10:00)

The core contributions of this talk centered on demonstrating practical, real-world applications of AI and automation to enhance offensive security workflows. The speakers showcased how these technologies can be integrated across the entire engagement lifecycle, providing actionable insights and improving efficiency at scale.

1. Automated Threat Intelligence: Leveraging low-code/no-code platforms like NA10, the team demonstrated how to automatically gather, filter, and summarize external threat intelligence from RSS feeds using LLMs. This capability extends to processing internal company data (Jira, code repositories) into valuable threat feeds, enabling proactive and context-aware testing.
2. Enhanced Continuous Reconnaissance: Workflows were presented that automate the discovery of exposed assets. This includes querying internet databases like Shodan API for known IP ranges and analyzing certificate transparency logs for new domains, with LLMs processing results for suspicious patterns or anomalies. This ensures an always-up-to-date understanding of the attack surface.
3. AI-Driven Threat Modeling: The talk highlighted a constant feedback loop for threat modeling, moving away from manual, resource-constrained processes. By pulling data from diverse internal sources and using AI orchestration, dynamic and comprehensive threat models can be generated, embedding this crucial activity into continuous development processes.
4. Orchestrated Penetration Testing: The speakers illustrated how traditional tools like Nmap, Nessus, Burp Suite, and Nuclei can be chained together and orchestrated by automation, with LLMs interpreting results and proposing actions. This allows for targeted, rapid scanning and retesting, significantly increasing coverage and responsiveness.
5. Streamlined Reporting with LLMs: A significant pain point in offensive security, reporting, was addressed through automation. Tools like Poundoc can be integrated with LLMs to generate structured findings and reports from raw assessment data, reducing manual effort and improving consistency, allowing human testers to focus on impact validation.
6. Augmentation, Not Replacement: A central tenet of the presentation was that AI and automation serve to augment human security professionals. By handling repetitive tasks and providing initial analysis, these technologies free up experts for more creative, in-depth analysis, research, and validation, ultimately amplifying their reach and results.

Technical Deep Dive

▶ Watch: AI-driven threat modeling for continuous security (14:00)

The speakers provided a comprehensive technical exploration of how AI and automation are integrated into various stages of offensive security. The core tool enabling many of these workflows is NA10, a low-code/no-code solution, lauded for its visual workflow builder and ability to execute JavaScript and Python code nodes.

Automated Threat Intelligence Gathering

The threat intelligence flow in NA10 starts with a time trigger, initiating recurrent news gathering from various RSS websites. Code nodes (JavaScript, soon Python) filter the raw news, retaining only relevant updates (e.g., "yesterday's news"). An LLM then acts as the "intelligence," processing the filtered news using natural language instructions to extract actionable insights for offensive security. The output, often in Markdown, is transformed into HTML and dispatched via SMTP, forming the basis of an automated newsletter.

A more advanced concept introduced was the MCP (Model Context Protocol), which enables LLMs to gain the ability to use tools. For NA10, this translates into an MCP server that can push and read flows via NA10 APIs. By combining this with a context server (providing API interaction guidelines) and an HTTP MCP (examples of working flows), AI can automatically generate NA10 workflows. This significantly accelerates automation setup, though it still requires substantial "engineering" for aspects like managing information timeliness (e.g., immediate alerts for zero days vs. weekly summaries). Furthermore, the speakers emphasized leveraging internal data sources such as quarterly planning documents, Jira projects, and code repository activity (merge requests) to create valuable internal threat feeds, informing strategic testing decisions.

Continuous Reconnaissance Workflows

Carmen Ruset detailed two key NA10 workflows for reconnaissance:

1. Weekly IP Range Scanning: This workflow queries internet databases like the Shodan API using known company IP ranges. The scan results, containing information about exposed services, are processed and sent to an LLM configured as an analyst. The LLM performs a comprehensive analysis, identifying interesting information, which is then formatted and sent via configured notification channels. This allows for scalable, natural language-driven analysis of external exposures.
2. Automated Domain Discovery: This workflow integrates with certificate transparency logs (CT logs), which are public records of issued TLS certificates. By cross-referencing these logs with a list of known company domains, newly appearing subdomains or related assets can be identified. For each match, additional DNS enrichment and HTTP probing are performed to understand what's running. The results are fed to an LLM to detect IOCs (Indicators of Compromise), suspicious patterns, or anomalies across newly discovered domains. Safe domains are then added to the monitoring list, enabling domain discovery on autopilot. This approach can be extended to monitor cloud infrastructure, service directories, and third-party services like GitLab repositories for misconfigurations.

AI-Driven Threat Modeling

Threat modeling, traditionally a manual and resource-intensive process, is automated through a continuous feedback loop. This involves:

1. Automated Scoping and Context Enrichment: Data is pulled from various organizational sources, including internal documentation, internal APIs, Jira issues, and ServiceNow incidents.
2. AI-Powered Discovery: This information is fed into an AI engine.
3. AI-Driven Model Generation: An AI orchestration, guided by specific instructions, generates an updated and dynamic view of the threat landscape.
4. Actionable Reporting: The process culminates in comprehensive and actionable reports.

The NA10 workflow acts as the engine, pulling data from diverse sources and using AI orchestration to generate a dynamic threat landscape. During the Q&A, it was clarified that while methodologies like STRIDE and PASTA serve as initial hints for the LLM, the team developed a custom methodology tailored to their specific needs, often cross-checking generated threats with code repositories using tools like Sourcegraph for validation.

Orchestrated Penetration Testing

Constantin-Danut Covrig elaborated on integrating core penetration testing tools with automation and AI. The key is not just running tools in isolation, but orchestrating them: the output of one tool automatically feeds into the next, with an LLM processing the aggregated data into a comprehensive report.

Examples included:

• Simple WordPress Scan: An NA10 flow utilizes an "execute build block" to run system commands (e.g., a WordPress scanner) against specific targets. The LLM then interprets the results and proposes actions.
• Targeted Nuclei Scans: Outputs from subdomain discovery tools can trigger targeted Nuclei scans using custom templates. Nuclei's extensive templating capabilities allow for customized testing, retesting, and advanced test cases for specific infrastructure.
• AI Agent for Continuous Testing: A more advanced concept involves an AI Agent with access to various tools, such as an HTTP request tool. The LLM can make requests, interpret results, choose appropriate system commands, and store findings in a local database. This allows for continuous testing, building upon previous results as new assets come online. The output can be formatted via code blocks or integrated with systems like Slack or Jira. The talk also referenced multiple open-source initiatives exploring LLM-powered agents for penetration testing, many built on the same "agentic testing" principle.

Automated Reporting

Reporting, often seen as the least enjoyable but most crucial part of an assessment, is a prime candidate for automation due to its time-consuming and error-prone nature. The speakers advocated for standardizing flows to ensure consistent, documented reports.

The solution presented involves feeding raw assessment data into a tool like Poundoc (an open-source vulnerability management and reporting tool, with a community-sustained version, PoundG). An NA10 flow or LLM, given a proper context, transforms the input into a structured finding, ready for review. This significantly reduces the time spent on writing, allowing pentesters to focus on validating findings and exploring their real impact.

A specific implementation involved creating an MCP server for Poundoc. Since Poundoc didn't natively offer an MCP, the speaker leveraged the Gemini CLI (command-line interface) to "vibe code" an MCP server. This custom server provided all of Poundoc's functionalities, allowing the LLM to interact with the Poundoc API, retrieve vulnerability data, and push new findings directly from the console, bypassing the need to switch between multiple tools. The LLM could access project directories containing logs and information from other tools, creating findings with minimal input (e.g., "SQL injection in a specific request parameter"). While initial descriptions might be minimal, fine-tuning the LLM with a clear context and potentially training it on an organization's existing issue database can yield high-quality, customized reports.

Demo / Proof of Concept

▶ Watch: Automating security report generation with LLMs (24:00)

While the presentation did not feature live, interactive demonstrations, the speakers effectively illustrated several proof-of-concept workflows and practical implementations through detailed screenshots and explanations of NA10 configurations and underlying logic.

The core of the "demo" was showcasing how NA10 acts as an orchestration engine. Examples included:

• Threat Intelligence Newsletter Workflow: A visual representation of the NA10 flow for gathering RSS feeds, filtering with JavaScript, summarizing with an LLM, and sending HTML reports via SMTP.
• Reconnaissance Workflows: Detailed breakdowns of NA10 flows for querying Shodan API for IP ranges and analyzing certificate transparency logs for domain discovery, with LLMs performing analysis on the results.
• Automated Threat Modeling Engine: An NA10 workflow illustrating the ingestion of diverse internal data sources and the use of AI orchestration to generate dynamic threat models.
• Testing Orchestration: Descriptions of NA10's "execute build block" for running tools like WordPress scanners or Nuclei, with LLMs interpreting output. The concept of an AI Agent using HTTP requests and local databases for continuous testing was also explained.
• Poundoc MCP Server: Constantin-Danut Covrig detailed the creation and functionality of his custom MCP server for Poundoc using the Gemini CLI. This demonstrated a tangible, working integration where an LLM could access and interact with a vulnerability management tool directly from the command line, generating structured findings from minimal input. This served as a concrete example of how to extend LLM capabilities to integrate with existing security tools, even if they lack native LLM integration.

These examples collectively served as strong evidence of the feasibility and practical benefits of integrating AI and automation into offensive security, moving beyond theoretical discussions to demonstrate functional blueprints.

Defensive Implications

▶ Watch: Future trends and addressing AI inconsistencies in Q&A (33:00)

The insights shared in this talk, though focused on offensive security, carry profound implications for defenders. The exponential growth of assets and the "minus one day" exploit window mean that traditional, reactive defense mechanisms are no longer sufficient. Defenders must embrace the same principles of automation and AI to protect their organizations effectively.

1. Continuous Visibility and Inventory: Just as attackers automate reconnaissance, defenders must automate asset discovery and inventory management. Leveraging tools that query Shodan API, certificate transparency logs, and internal asset databases continuously is crucial to know what's exposed.
2. Proactive Threat Intelligence: Automating the collection and analysis of external threat intelligence (e.g., zero-day alerts) and internal data (Jira, code repositories) can provide early warnings and inform defensive priorities, mirroring the offensive use of these feeds.
3. Automated Security Testing: Defenders should implement continuous, automated security testing (e.g., using Nuclei with custom templates) to identify vulnerabilities as soon as they appear, ideally before attackers do. This includes integrating security checks directly into development pipelines.
4. AI-Augmented Security Operations: LLMs and AI agents can assist in analyzing security logs, identifying anomalies, and triaging alerts, freeing up human analysts for complex incident response. The ability of LLMs to interpret results and propose remediations, as shown in offensive testing, can be mirrored for defensive operations.
5. Context-Rich Defense: The emphasis on correlating diverse data sources (internal and external) to build rich context for offensive actions is equally vital for defense. Understanding the full context of an asset, its data, and its connections can inform more effective security controls and incident response.
6. Embrace Augmentation: Defenders should view AI and automation not as a replacement for their expertise, but as a force multiplier. By offloading repetitive tasks, security teams can focus on strategic planning, advanced threat hunting, and the nuanced human elements of cybersecurity.
7. Build Resilient Systems: The talk highlighted that companies with a good balance between building, defending, and attacking tend to have fewer attacks. This means building systems with security in mind from the outset, informed by continuous threat modeling and security assessments.

Key Takeaways

• AI and Automation are Force Multipliers: These technologies augment, rather than replace, human offensive security professionals, enabling them to achieve greater scale, consistency, and depth in their work.
• Full Lifecycle Integration: AI and automation can be effectively applied across the entire offensive security engagement, from threat intelligence and reconnaissance to threat modeling, testing, and reporting.
• Low-Code/No-Code Platforms are Key: Tools like NA10 simplify the creation of complex workflows, making advanced automation accessible even for those without extensive coding backgrounds.
• Context is Paramount: Leveraging both external threat feeds and internal company data (Jira, code repos) provides crucial context for targeted and effective security assessments.
• Orchestration of Tools is Powerful: Chaining traditional security tools (e.g., Nmap, Nuclei, Burp Suite) with LLMs for interpretation and action allows for highly efficient and comprehensive testing.
• Automation Streamlines Tedious Tasks: AI can significantly reduce manual effort in areas like reporting, allowing security professionals to focus on higher-value activities such as validating findings and exploring their impact.

About the Speaker(s)

Andrei Cristea is the Manager for Offensive Security at Booking Holdings Romania. With extensive experience in the cybersecurity domain, Andrei provided the overarching framework for the presentation, discussing the evolution of offensive security, current challenges, and future trends. He is passionate about exploring how new technologies like AI can empower security teams.

Carmen Ruset is a Senior Penetration Tester at Booking Holdings Romania. Carmen delved into the practical applications of automation and AI in reconnaissance and threat modeling. Her expertise lies in leveraging tools and intelligent workflows to continuously monitor and assess the attack surface, ensuring comprehensive coverage and proactive defense.

Constantin-Danut Covrig is also a Senior Penetration Tester at Booking Holdings Romania. Constantin-Danut focused on the technical deep dive into automating penetration testing execution and, notably, the often-dreaded task of reporting. He shared his personal journey and practical solutions, including the creation of a custom MCP server for Poundoc using Gemini CLI, demonstrating innovative ways to streamline post-assessment activities.