Pentesting ChatOps

Dr. Melanie Rieback (CEO · radically open security)

Swiss Cyber Storm 2019 · Day 1 · Main

Overview

Dr. Melanie Rieback's talk, "Pentesting ChatOps," delivers a compelling argument for a radical shift in the penetration testing industry's methodology and ethics. As CEO of Radically Open Security, the world's first not-for-profit computer security consultancy, Rieback introduces ChatOps as a transformative approach that champions transparency, collaboration, and efficiency. This talk is not merely a technical exposition but a philosophical statement, challenging the traditional "black box" security consultancy model that often fosters client dependence and secrecy.

Watch on YouTube

Visual summary for Pentesting ChatOps by Dr. Melanie Rieback
Visual summary for Pentesting ChatOps by Dr. Melanie Rieback

Key moments

  1. 0:00 Introduction to Radically Open Security's non-profit model
  2. 2:30 Frustrations with traditional, opaque security consulting
  3. 7:00 Introduction to ChatOps and its benefits for distributed teams
  4. 15:00 “Peek over our shoulder”: Customers in pen test chat rooms
  5. 18:00 Gamifying pen tests with red/blue team competitions
  6. 22:00 Visualizing phishing attack results in real-time
  7. 27:00 Transparent time tracking and project progress with ChatOps
  8. 32:00 Competitor's response to the talk's core message

Pentesting ChatOps

Speakers: Dr. Melanie Rieback, CEO, radically open security

Conference: Swiss Cyber Storm

YouTube: https://www.youtube.com/watch?v=WNQ9xVjPUhc

Overview

Dr. Melanie Rieback's talk, "Pentesting ChatOps," delivers a compelling argument for a radical shift in the penetration testing industry's methodology and ethics. As CEO of Radically Open Security, the world's first not-for-profit computer security consultancy, Rieback introduces ChatOps as a transformative approach that champions transparency, collaboration, and efficiency. This talk is not merely a technical exposition but a philosophical statement, challenging the traditional "black box" security consultancy model that often fosters client dependence and secrecy.

Rieback demonstrates how integrating a chatbot into a distributed team's workflow can automate mundane tasks, streamline complex processes, and foster unprecedented levels of knowledge transfer between testers and clients. The core innovation lies in leveraging chat as a command-line interface, allowing for real-time interaction, immediate feedback, and a shared operational picture. This approach not only enhances the effectiveness of security engagements but also aligns with Radically Open Security's mission to contribute to digital rights and open-source security research, driven by a commitment to positive impact over profit.

The implications of ChatOps extend beyond mere technical convenience, fundamentally altering the dynamic between security providers and their clients. By inviting customers directly into their operational chat rooms, Radically Open Security cultivates a unique environment of complete openness, enabling rapid problem-solving, continuous learning, and even gamified security exercises. This talk is crucial for anyone in the security industry seeking to improve efficiency, transparency, and collaborative defense strategies, offering a blueprint for a more ethical and effective future in penetration testing.

Background

▶ Watch: Introduction to Radically Open Security's non-profit model (0:00)

The genesis of Radically Open Security and its adoption of ChatOps stems from Dr. Melanie Rieback's profound disillusionment with the traditional security consultancy industry. Recounting an incident during her tenure at ING Bank's cybercrime team, Rieback experienced firsthand the arrogance and opacity of external consultants who refused to share their methods or proprietary tools, even going so far as to delete bash history logs and avoid logging screen sessions. This experience highlighted a pervasive issue: a lack of transparency that hindered knowledge transfer, cultivated client dependence, and ultimately undermined the client's long-term security posture.

Further reinforcing this ethical crisis was the boycott of the Ohm hacker camp by the German Chaos Computer Congress and the Dutch hacker community. The boycott targeted market leader Fox IT due to concerns about their ethical practices, leading to the formation of "Noisy Square," an organization dedicated to integrating hacker values of openness and transparency into the security industry. These events crystallized Rieback's conviction that a different model was needed—one that prioritized ideals over profit. Thus, Radically Open Security was founded as a fiscal fundraising institution, a unique non-profit structure where all profits are reinvested into charities furthering digital rights and internet security research. This model fundamentally reorients operational decisions towards optimizing "positive impact" rather than financial gain.

Against this backdrop, the need for an efficient, transparent, and collaborative operational framework became paramount for Radically Open Security's globally distributed team of 40 ethical hackers. Rieback discovered ChatOps, a concept pioneered by GitHub for managing distributed operations. ChatOps leverages a chatbot (like Hubot, an open-source tool) as a UNIX-like command-line interface within a chat room, allowing teams to execute commands, get output, and intersperse human dialogue with automated tasks. Initially using basic tools like IRC and Etherpad, Radically Open Security soon adopted Rocket.Chat, an open-source, self-hosted alternative to Slack, combined with Hubot. This foundational setup enabled them to build a highly transparent and efficient workflow, directly addressing the shortcomings Rieback identified in the conventional security consultancy model.

Key Findings

▶ Watch: Introduction to ChatOps and its benefits for distributed teams (7:00)

Dr. Rieback's presentation unveils several key findings and contributions that demonstrate the transformative power of ChatOps in penetration testing:

  1. Enhanced Transparency and Knowledge Transfer: By integrating customers directly into their ChatOps channels (the "peek over our shoulder" workflow), Radically Open Security achieves unparalleled transparency. This real-time visibility allows clients to observe all testing activities, ask questions directly, and provide immediate context, vastly improving knowledge transfer and preventing time wastage on irrelevant paths.
  2. Significant Efficiency Gains Through Automation: The ChatOps platform enables the automation of numerous time-consuming, repetitive tasks. This includes the entire process of generating pen-test reports and quotations using tools like PenText, running passive and active vulnerability scans (e.g., Shodan, Censys, Nmap, w3af), and managing phishing campaigns. This automation frees up testers to focus on more complex, high-value activities.
  3. Real-Time, Collaborative Security Operations: ChatOps transforms security testing into a highly interactive and collaborative process. Teams can coordinate distributed efforts seamlessly, share findings instantly, and collectively analyze results. The integration of tools like GitLab for documentation and reporting ensures that all output is accessible and auditable by the entire team and the client.
  4. Gamified Learning and Skill Development: The implementation of red/blue team pen testing within the ChatOps environment gamifies security education. By dividing client teams into red and blue factions and having them compete under the guidance of professional pen testers, organizations can significantly improve their internal security posture and developers' understanding of vulnerabilities, with participants often remarking, "I will never look at code the same way again."
  5. Unprecedented Operational Visibility and Accountability: Tools integrated into the chatbot, such as a transparent time-tracking system (rawspot charge), provide real-time updates on project progress and resource consumption. This fosters accountability for both the pen testers and the client, ensuring clear communication about where time and money are being spent.
  6. A DevOps Mindset for Security Consultancies: The reliance on ChatOps for core operations effectively turns the security company into a "DevOps company." This necessitates robust, constantly evolving infrastructure, dedicated IT personnel, and a commitment to continuous refactoring and Dockerization for scalability and reliability, ensuring that the operational platform itself is resilient.

Technical Deep Dive

▶ Watch: Gamifying pen tests with red/blue team competitions (18:00)

Radically Open Security's ChatOps infrastructure is built upon a foundation of open-source tools, demonstrating a commitment to their core values. The primary components are Rocket.Chat, a self-hosted, open-source alternative to Slack, serving as the central communication hub, and Hubot, an open-source chatbot framework that acts as the command-line interface within Rocket.Chat. This setup allows testers to interact with a wide array of backend security tools and custom scripts directly from their chat window.

One of the most significant technical contributions is their PenText system, an OWASP project that automates pen-test documentation. This XML-based system allows testers to invoke commands like rawspot quickscope OFF <client_name> <project_name> directly in the chat. In the backend, this command triggers a custom builder tool, which leverages Saxon (an XSLT and XQuery processor) and XSLT stylesheets to transform a concise, A4-sized XML input into a comprehensive, long-form PDF report, complete with standard boilerplate and company branding. The generated PDF is automatically password-protected (a feature under re-evaluation due to evolving exploits) and uploaded to an internal GitLab instance, with a clickable link provided directly in the chat for immediate access by the team and client. Crucially, PenText also features a conversion script that allows pen testers to input findings directly into GitLab issues, which are then automatically converted into the required XML format, minimizing direct interaction with XML unless fine-tuning is needed.

For reconnaissance and scanning, the ChatOps platform integrates various tools. A custom passive vulnerability scanning tool correlates data from Shodan and Censys to provide quick insights into a target's exposed services. The output, if extensive, is automatically checked into GitLab with a chat link, maintaining a clean chat environment while ensuring data accessibility. The system is also designed to invoke active scanning tools like Nmap for network enumeration and w3af (Web Application Attack and Audit Framework) for web vulnerability scanning. The outputs from these scans are similarly directed to GitLab, with summary links appearing in the chat. More advanced tools like SQLMap for SQL injection and even password crackers utilizing rainbow tables can be orchestrated through the chatbot, making powerful offensive capabilities accessible from any web-enabled device, including a mobile phone.

The custom phishing suite, also open-sourced, demonstrates another innovative application. Security officers can provide target lists and instrumented links within the chat. When a target clicks on a link, the integrated mail server, leveraging an API (similar to Slack's but adapted for Rocket.Chat), sends a real-time notification to the chat room, detailing the email address, pretext name, and timestamp of the click. This provides immediate, visual feedback on the effectiveness of phishing campaigns, as vividly illustrated by a healthcare institution test where clicks overwhelmed the chat room. Instrumented forms gather additional data, even capturing humorous (and sometimes concerning) attempts at SQL injection from developers.

Project management is also integrated, using Kanban boards and a Zendesk-like support desk system. Instrumented email addresses ensure that all project-related correspondence is automatically tracked and linked within GitLab, with notifications appearing in the chat. This allows security officers to be automatically looped into all relevant project channels without extra effort. For billing and project oversight, the rawspot charge <hours> <description> command allows pen testers to declare their hours transparently. The bot then provides real-time feedback on remaining project hours and a progress bar, visible to both testers and clients, reinforcing the commitment to openness.

Underlying this entire infrastructure is a robust DevOps philosophy. The system employs role-based access control (RBAC) across the entire infrastructure, ensuring proper authorization. Error and debug logs are routed to separate chat rooms, allowing system administrators to troubleshoot issues without direct server access. Radically Open Security is actively undergoing a "giant DevOps refactoring," Dockerizing large parts of their system for easy spin-up and better isolation of customer data through individual containers, enhancing security and scalability. All these tools and the underlying infrastructure are open-sourced, embodying the company's commitment to community contribution.

Demo / Proof of Concept

▶ Watch: Visualizing phishing attack results in real-time (22:00)

While Dr. Rieback's talk does not feature a live, interactive demonstration in the traditional sense, the presentation itself serves as a comprehensive proof of concept. Throughout her discussion, she incorporates numerous screenshots and descriptions of the ChatOps system in action, illustrating its functionality and impact. These visual examples effectively showcase how commands are issued, and how the chatbot responds with relevant information or triggers backend processes.

Key instances highlighted include:

  • A lighthearted example demonstrating the bot's ability to respond to rawspot hug me with an image of a pug, illustrating basic bot interaction.
  • The invocation of the PenText system with rawspot quickscope OFF Melanie demo, followed by screenshots detailing the backend compilation process and the resulting clickable link to the generated, password-protected PDF report hosted on GitLab. This clearly illustrates the automation of documentation.
  • The red/blue team pen testing scoreboard is demonstrated with the command good job blue, showing the bot incrementing points and displaying a motivational image, proving the gamification aspect.
  • The phishing suite is described through the real-time visualization of clicks in the chat room. Rieback vividly recounts how security officers could observe employees falling for phishing lures, with chat messages appearing for each click, demonstrating the immediate feedback mechanism for security awareness campaigns.
  • The transparent time-tracking mechanism is shown with the rawspot charge <hours> <description> command, where the bot confirms the declaration, updates the remaining hours, and displays a progress bar, making project billing clear and accountable to both the tester and the client.

Through these detailed descriptions and accompanying visuals, the audience gains a clear understanding of how the ChatOps system operates, the types of tasks it can automate, and the real-time benefits it offers for transparency, collaboration, and efficiency in penetration testing. The "demo" is woven into the narrative, providing concrete evidence of the system's capabilities without requiring a separate, dedicated segment.

Defensive Implications

▶ Watch: Competitor's response to the talk's core message (32:00)

The ChatOps paradigm, as presented by Dr. Rieback, offers profound defensive implications for organizations beyond merely hiring a more transparent pen-testing company. Integrating these principles can fundamentally transform an organization's internal security posture and operational efficiency.

Firstly, the "peek over our shoulder" workflow is a game-changer for knowledge transfer and proactive defense. By observing pen testers in real-time, internal security teams, developers, and sysadmins gain invaluable insights into attacker methodologies, exploited vulnerabilities, and the specific weaknesses within their systems. This direct exposure, coupled with the ability to ask immediate questions and provide context, accelerates learning and fosters a security-first mindset among development teams. It shifts security from a post-mortem audit to an integrated, continuous learning process, enabling faster remediation and the implementation of more robust preventative controls.

Secondly, the gamified red/blue team exercises conducted within a ChatOps environment are a powerful tool for internal skill development and incident response training. By having their own teams compete, organizations can stress-test their defenses, identify blind spots in their monitoring and response capabilities, and significantly enhance the practical skills of their security and operations personnel. The real-time scoring and collaborative environment make learning engaging and impactful, preparing teams for actual attacks.

Thirdly, the automation of security tasks through a chatbot interface can drastically improve operational efficiency for internal security teams. Imagine internal security analysts invoking vulnerability scans (nmap, w3af), passive reconnaissance (Shodan, Censys), or even triggering playbooks for incident response directly from their team chat. This centralizes control, reduces manual overhead, and ensures consistent execution of security procedures. The automated generation of reports and documentation, similar to Radically Open Security's PenText, can free up internal teams from tedious administrative tasks, allowing them to focus on threat hunting and strategic defense.

Furthermore, the real-time feedback from tools like the phishing suite provides immediate, actionable intelligence on employee susceptibility to social engineering. This allows security awareness programs to be rapidly adapted and targeted, moving beyond generic training to address specific, observed weaknesses within the organization.

Finally, while adopting ChatOps offers immense benefits, defenders must also consider the security of the ChatOps platform itself. As Rieback acknowledges, the platform becomes a critical component of the organization's infrastructure. Implementing robust Role-Based Access Control (RBAC), regular security audits, and architectural decisions like containerization to isolate sensitive data (as Radically Open Security is doing with Docker) are paramount. A compromised ChatOps environment could expose sensitive operational details, client data, and even provide an attacker with a direct interface to internal tools. Therefore, treating the ChatOps system as a high-value target and securing it rigorously is a fundamental defensive implication.

Key Takeaways

  • ChatOps revolutionizes penetration testing by transforming it into a highly transparent, collaborative, and efficient process, moving away from traditional "black box" consultancy models.
  • Automation of core pen-testing tasks such as documentation, reporting, and various scans (Nmap, Shodan, Censys) via a chatbot interface significantly boosts productivity and consistency.
  • Integrating customers directly into the ChatOps workflow ("peek over our shoulder") fosters unprecedented knowledge transfer, enables rapid feedback loops, and accelerates remediation efforts.
  • Gamified red/blue team exercises conducted within ChatOps environments provide an effective and engaging method for internal teams to develop practical security skills and improve their defensive posture.
  • Radically Open Security's non-profit, open-source business model challenges conventional industry ethics, demonstrating how prioritizing positive impact and shared knowledge can lead to highly effective and innovative security services.
  • Organizations should consider adopting ChatOps for their internal security operations to enhance efficiency, collaboration, and incident response, while rigorously securing the ChatOps platform itself through measures like RBAC and containerization.

About the Speaker(s)

Dr. Melanie Rieback is the visionary CEO of Radically Open Security, an organization she founded as the world's first not-for-profit computer security consultancy. Her unique business model, where all profits are reinvested into charities promoting digital rights and open-source security research, stems from her earlier experiences in the industry, including working in the cybercrime team at ING Bank.

Before establishing Radically Open Security, Dr. Rieback was an Assistant Professor of Computer Science at Vrije Universiteit Amsterdam, specializing in computer security. Her academic background provides a strong foundation for her analytical and innovative approach to security challenges. She is a passionate advocate for openness, transparency, and the integration of hacker values into the security industry.

Dr. Rieback's innovative business model and leadership have garnered significant recognition. She has been named one of the 50 most innovative SMEs in the Netherlands by the Dutch Chamber of Commerce, the most innovative IT leader of the Netherlands by CIO magazine, and one of the nine most innovative women in the European Union by the European Commission. She is also a proponent of "post-growth business" and has spoken on this topic at TEDx events. Her work at Radically Open Security, including the development and open-sourcing of their ChatOps tools, reflects her commitment to making security knowledge and tools accessible for the benefit of all.

Reviews

Dr. Zero (Offensive Security Researcher) — SOLID

This is an operational workflow talk, not vulnerability research. Rieback is outlining a non-profit consultancy model built on Rocket.Chat and Hubot wrappers for standard tools. It works because the traditional pentest consultancy model is completely broken, but nobody is dropping zero-days here.

Heather Calloway (CISO) — SOLID

Rieback pitches a model where pen-testers invite your internal team into their ChatOps environment to watch the engagement unfold in real time. It's a practical antidote to the standard consultancy model that just throws a PDF over the wall at the end of the month. Worth considering if you want to extract actual engineering value from your compliance-driven pen tests.

→ Top-rated talks at Swiss Cyber Storm 2019

All talks from Swiss Cyber Storm 2019