SSH Honeypots and Walkthrough Workshops: A History

SSH Honeypots and Walkthrough Workshops: A History

Speakers: Ryan Mitchell, Software Engineer

Conference: DEF CON

YouTube: https://www.youtube.com/watch?v=TXUSC5YEDPo

Overview

In this engaging DEF CON talk, software engineer Ryan Mitchell, known as "clugist" at the Packet Hacking Village (PHP), delves into his dual passions: SSH honeypots and the evolution of interactive walkthrough workshops. Mitchell chronicles his journey from a curious DEF CON attendee to a key contributor at the PHP, demonstrating how security tools like honeypots can transcend their traditional roles to become platforms for creative, immersive, and educational experiences. The talk explores the technical underpinnings of honeypots, the challenges and successes of building large-scale security workshops, and the speaker’s philosophy of integrating "surprise, absurdity, and play" into the often-serious world of cybersecurity.

Mitchell’s presentation is more than just a technical overview; it's a testament to the power of community-driven education and the enduring spirit of hacker culture. He illustrates how a simple security tool can be transformed into elaborate, story-driven challenges that engage participants of all skill levels, fostering both learning and a sense of wonder. By sharing the detailed history of the PHP’s walkthrough workshops and his personal honeypot "quests," Mitchell provides invaluable insights into designing effective, hands-on learning environments and pushing the boundaries of what security tools can achieve.

The talk is particularly relevant for security professionals interested in practical applications of honeypots, educators looking for innovative teaching methodologies, and anyone who believes that learning in cybersecurity should be as fun and imaginative as the challenges it presents. Mitchell's journey highlights the importance of contributing to the community, even outside one's primary professional domain, and showcases how passion projects can significantly impact the broader security landscape.

Background

▶ Watch: Introduction, speaker background, and book promotion (0:00)

Ryan Mitchell, a software engineer with no formal networking or IT background, stumbled into the world of cybersecurity education at DEF CON. His initial experience with the conference was typical: exploring villages, meeting people, and attending parties. This changed dramatically in 2016 when a conversation with Davin Pototts (BSE) of the PHP led him to give a talk at the village. This engagement permanently "trapped" him within the PHP, where he has since contributed significantly to its educational offerings.

A pivotal moment occurred in 2016 when Mitchell encountered a peculiar entry on the Wall of Sheep – an SSH connection for root. Knowing SSH is a secure protocol, its presence on the Wall of Sheep was immediately suspicious. Upon logging in, he found a server with unusual behavior: common Linux commands like vi, Emacs, or Nano were missing, and files listed in directories would report "no such file" when attempting to cat them. This baffling experience eventually led him to the realization that he was interacting with a honeypot – a system designed to mimic a real server to deceive attackers and collect information about their activities. This particular honeypot was a multi-stage challenge, starting with a base64-encoded file leading to steganography puzzles and subsequent servers, culminating in Mitchell being the sole winner that year. This initial encounter ignited his fascination with SSH honeypots.

Mitchell clarifies the common classifications of honeypots: low-interaction honeypots, like Dionaea, support many protocols but offer minimal functionality beyond banners and handshakes. High-interaction honeypots typically involve real operating systems, often within disposable containers, providing a fully functional environment. In between are medium-interaction honeypots, such as Cowrie, which became Mitchell's tool of choice. Cowrie specifically supports SSH and Telnet, simulating a significant subset of Bash commands, maintaining a virtual file structure, and logging attacker interactions. While more robust than low-interaction systems, Cowrie doesn't withstand intense scrutiny, as Mitchell's own experience highlighted. Its ease of configuration and excellent documentation made it ideal for the PHP's purposes.

In 2018, Riverside, the leader of the PHP, tasked Mitchell with developing walkthrough workshops focusing on Cowrie honeypots. Mitchell, along with Warlord (an IT director) and Catalyst (an experienced network engineer), formed a team to create these hands-on sessions. Their initial setup was rudimentary, with limited seating and janky equipment. The first attempt to run the workshop was hampered by the notorious unreliability of the DEF CON network, forcing participants to watch sudo apt get update commands fail. This crucial early failure underscored the need for resilient, offline-capable infrastructure, leading to the development of Docker images with pre-installed Cowrie and a complete overhaul of their workshop materials to eliminate external dependencies.

Key Findings

▶ Watch: Speaker's journey to the Packet Hacking Village (3:05)

Ryan Mitchell's extensive experience with SSH honeypots and the Packet Hacking Village workshops has yielded several significant findings, challenging conventional notions of security tools and education.

Firstly, Mitchell demonstrated that SSH honeypots, particularly Cowrie, can be transformed from passive data collectors into dynamic, interactive, and engaging platforms for security challenges and educational experiences. His initial encounter with a honeypot at DEF CON 2016, which was a multi-stage puzzle, opened his eyes to the potential of these systems beyond simply gathering malware or log data. He realized that by creatively manipulating the simulated environment, honeypots could host complex, story-driven challenges, allowing participants to "break the rules of Bash" and explore imaginative scenarios.

Secondly, the evolution of the PHP's walkthrough workshops highlighted the critical elements for effective, large-scale security education. Starting from a rudimentary setup in 2018, the workshops adapted to the realities of a conference environment (e.g., unreliable internet) by moving to Docker images and meticulously crafted, self-contained HTML guides. The key to their success was a focus on step-by-step instructions, including copy-paste commands, and even dynamically generated, unique credentials to minimize user error and maximize accessibility for participants with diverse skill sets, from Linux novices to seasoned security professionals.

Thirdly, Mitchell's approach revealed that in low-stakes educational environments, prioritizing fun and engagement over excessive security paranoia can lead to better learning outcomes. While the team initially considered elaborate physical security and monitoring systems for their workshop machines, they found that participants were primarily motivated by learning and rarely attempted malicious actions. This insight allowed them to focus on creating a welcoming and enjoyable learning atmosphere. Conversely, a critical lesson learned was never to rely on the DEF CON internet for any workshop components, a practical tip for any conference organizer.

Fourthly, Mitchell's honeypot challenges showcased the power of narrative and thematic integration in making technical puzzles more compelling. From the "evil corporation stealing water" storyline in 2017 to the "Hacker Homecoming" theme in 2022, which involved a Seattle-grid-based adventure, a Game of Life dungeon, and a Hackers-inspired "Gibson cube," these challenges demonstrated how rich backstories can elevate abstract technical problems into memorable experiences.

Finally, the 2025 "Existential Dread" challenge, incorporating generative AI (LLMs), provided a nuanced finding regarding the capabilities and limitations of AI in creating specific types of content. While LLMs excel at producing plausible and meaningful language, Mitchell discovered they are terrible at strictly adhering to finite state grammars or generating text based on precise biogram/n-gram rules (like a Markov chain from a specific text). This necessitated a manual approach for one server, illustrating that even advanced AI has specific constraints when precise, rule-based text generation is required. This highlights the importance of understanding AI's strengths and weaknesses when designing complex interactive systems.

Technical Deep Dive

▶ Watch: First strange SSH entry on the Wall of Sheep (4:10)

The technical foundation of Ryan Mitchell's work at the Packet Hacking Village primarily revolves around Cowrie, a medium-interaction SSH and Telnet honeypot. Cowrie serves as the core engine for simulating a Linux server environment, allowing for the creation of believable yet controlled interactive experiences. It intercepts inbound SSH connections, presents a seemingly authentic shell prompt, and logs all user interactions, including commands executed, files accessed, and even attempts to upload or download files. Crucially, Cowrie maintains a virtual file system, enabling administrators to populate the honeypot with custom files and directories, which can be modified or created dynamically. This capability is central to Mitchell's challenges, where specific files or directory structures become integral parts of puzzles. While Cowrie can simulate many common Bash commands, Mitchell notes that it "doesn't stand up to that much scrutiny," meaning advanced users might quickly discern its true nature. This limitation, however, is intentionally exploited in his challenges, where unexpected server behavior is part of the puzzle.

For the walkthrough workshops, the technical infrastructure evolved significantly to address the challenges of a live conference environment. Initially, participants would manually install and configure Cowrie. However, the unreliability of the DEF CON network quickly led to a paradigm shift. The team developed Docker images that came with Cowrie pre-installed and pre-configured. This approach ensured that every participant had a consistent, functional environment that required no external internet access, drastically improving the workshop's reliability and user experience.

Further enhancing the workshops' accessibility and engagement, Mitchell and his team implemented dynamic content generation. This included generating unique usernames and passwords for each participant's Cowrie server, embedded directly into personalized HTML instruction documents. This eliminated common user errors, such as mistyping generic placeholders like "YOUR PASSWORD HERE," by providing exact copy-paste commands. This level of customization ensured a smoother learning curve for individuals with varying levels of Linux or command-line experience.

Mitchell's honeypot challenges demonstrate increasingly sophisticated technical design:

• 2017 Challenge: Introduced a networked series of honeypots with dynamic elements. One server’s IP address changed with the clock time (e.g., 11.30 at 11:30 AM), requiring rapid interaction. Another innovation was the concept of a "router" to expand the available IP address space, allowing for more virtual servers within the challenge. This challenge also featured JWT token decoding and steganography.
• 2022 "Hacker Homecoming" Challenge:
• Seattle Grid Server: This server simulated a geographical grid based on downtown Seattle's streets. LS commands would reveal directional directories (north, south, east, west), and PWD would display cross-streets and landmarks. A notable technical feat was the transmission of a low-resolution ASCII video of Nirvana's "Smells Like Teen Spirit" over SSH upon completion, demonstrating creative limits of SSH interaction.
• Dungeon Server (Conway's Game of Life): This challenge featured a grid-based file system representing a dungeon. A custom light command allowed users to toggle cells in John Conway's Game of Life. The objective was to build a specific pattern, a glider, which was then validated by a custom cellular automaton validator written within the honeypot itself.
• **Gibson Cube (Homage to Hackers and Pokémon): This server presented a cube-like file system, where access commands revealed accessible "floors." Directories were populated with random, distracting corporate documents generated by a Markov generator. The most interactive element was the random appearance of ASCII guards** (inspired by Pokémon battles) upon command execution. These guards would pose networking questions; correct answers would grant "key cards" to advance floors, while incorrect answers would result in being kicked off the server. The final boss required answering a specific security question.
• 2025 "Existential Dread" Challenge (Current Year): This challenge leverages generative AI through the OpenAI Assistants API for three of its four servers. A "command line assistant" named M, backed by a large AI prompt, interacts with players. The prompt defines M's personality (cryptic, poetic), what information she can reveal (not game solutions), and safeguards against prompt injection (e.g., in-game directives from a secret government organization). The fourth server, however, deliberately avoids LLMs. Mitchell designed it to generate text using a Markov chain from a specific text, aiming for meaningful yet rule-bound output. He discovered that LLMs are ill-suited for generating text that strictly adheres to finite state grammars or precise biogram/n-gram rules, ultimately requiring him to manually construct the biograms and sentences for this segment.

These technical implementations showcase how honeypots can be extended far beyond their basic logging capabilities to create rich, interactive, and intelligent environments for education and entertainment.

Demo / Proof of Concept

▶ Watch: The 'Aha!' moment: realizing it was a honeypot (5:58)

The talk itself serves as a retrospective and ongoing demonstration of Ryan Mitchell's honeypot challenges and the Packet Hacking Village's walkthrough workshops. Each year's challenge, meticulously described by Mitchell, acts as a proof of concept for the innovative use of SSH honeypots as interactive puzzles.

Mitchell vividly recounts his personal "demo" in 2016 when he unknowingly became the first and only winner of the initial honeypot challenge. This experience, which involved decoding a B64 file, solving a steganography puzzle, and navigating multiple servers, proved the viability of creating engaging, multi-stage challenges within a honeypot environment.

The walkthrough workshops are the most direct and continuous demonstration. From their humble beginnings in 2018 with "janky" equipment and network issues, they evolved into a highly streamlined operation. By 2024, the workshops boast over 50 machines, continuously cycling through various topics like password cracking (Kraken's lab), network OS walkthroughs (Catalyst), and botnet building (Bonus). Mitchell's Cowrie workshop, in particular, demonstrates how to install, configure, and add fake files and users to a honeypot system. The success is evident in the constant demand, with all seats filled every hour and a ticketing system implemented for crowd control, proving the high value and effectiveness of hands-on, interactive learning.

The detailed descriptions of the honeypot quests from 2017, 2019, and 2022 further illustrate the capabilities. The 2017 challenge, with its dynamic IP address changes and "router" concept, showcased how to build a complex network of virtual machines for a timed puzzle. The 2019 "Somewhere Over the Rainbow" challenge involved a physical scavenger hunt at DEF CON, proving the integration of digital and physical elements for a unique experience.

The 2022 "Hacker Homecoming" challenges were particularly rich demonstrations:

• The Seattle Grid Server effectively demonstrated a navigable, themed file system, culminating in the playing of a low-resolution ASCII video over SSH – a clear proof of concept for pushing visual boundaries within a text-based interface.
• The Dungeon Server for Conway's Game of Life showcased the integration of a custom game logic (cellular automaton validator) directly within the honeypot, reacting to user input to solve a mathematical puzzle.
• The Gibson Cube demonstrated dynamic enemy encounters and interactive questioning, mimicking a role-playing game within an SSH shell, complete with "key cards" and a "boss fight."

Finally, the 2025 "Existential Dread" challenge is an active, live demonstration. Mitchell explicitly states he was running it before his talk and encourages attendees to participate (ssh [email protected]). This challenge, with its AI-driven command-line assistant M and the non-LLM Markov chain server, is the latest proof of concept for creating immersive, narrative-rich experiences that explore the philosophical implications of AI while also highlighting its technical limitations. Mitchell even shares the names of the first winners, Chris and Lucas, indicating its ongoing success.

Defensive Implications

▶ Watch: Definition and explanation of SSH honeypots and Cowrie (7:00)

Ryan Mitchell’s work, while focused on creative engagement, carries significant defensive implications for organizations and security professionals. His use of SSH honeypots and the design of interactive workshops offer valuable lessons for understanding, mitigating, and educating against cyber threats.

Firstly, threat intelligence and attacker profiling are core defensive applications of honeypots like Cowrie. By deploying such systems, defenders can capture detailed logs of attacker interactions, including commands executed, files uploaded, attempts to exploit vulnerabilities, and enumeration techniques. This information is crucial for understanding the Tactics, Techniques, and Procedures (TTPs) of various threat actors. Mitchell's honeypots, even though designed for puzzles, still record these interactions, providing a rich dataset for security researchers to analyze attacker behavior in a controlled environment. This intelligence can then be used to inform the development of better detection rules, improve incident response playbooks, and strengthen overall defenses.

Secondly, the success of the Packet Hacking Village’s walkthrough workshops highlights an effective model for security awareness and technical training. Mitchell's emphasis on step-by-step, hands-on learning, coupled with dynamically generated personalized content, demonstrates how to make complex security topics accessible to a broad audience, including those with minimal technical background. Organizations can adopt similar methodologies to train their employees on topics like secure coding practices, network forensics, or even basic Linux command-line usage. By creating low-stakes, interactive environments (like the Docker-based workshops), employees can gain practical experience without risking production systems, fostering a culture of continuous learning and skill development.

Thirdly, Mitchell's journey underscores the importance of resilient and self-contained training infrastructure. The early failure of the workshops due to unreliable DEF CON internet led to the adoption of Docker images and offline content. This is a critical lesson for any organization planning internal training or simulation exercises: dependency on external networks or services can derail educational efforts. Designing self-sufficient training modules ensures consistency and reliability, allowing participants to focus solely on learning.

Fourthly, the creative and narrative-driven honeypot challenges offer a blueprint for engaging security exercises and simulations. Instead of dry, theoretical exercises, Mitchell's "Hacker Homecoming" and "Existential Dread" challenges demonstrate how to turn defensive training into an immersive experience. Gamified scenarios, like navigating a "Gibson cube" or solving a Conway's Game of Life puzzle, can enhance problem-solving skills, critical thinking, and teamwork among security teams. Such exercises can be particularly valuable for incident response team training, allowing them to practice identifying and responding to simulated threats in a dynamic and unpredictable environment.

Finally, Mitchell's observation about LLMs being "terrible" at adhering to strict grammatical rules for Markov chains provides a cautionary note for defenders considering AI for specific, rule-based security tasks. While generative AI is powerful for broad language generation, its probabilistic nature may make it unsuitable for applications requiring absolute precision or adherence to predefined syntax, such as generating highly specific malicious payloads or validating complex protocol messages. Defenders need to understand these limitations when integrating AI into their tools and processes, ensuring that the AI's capabilities align with the task's requirements.

Key Takeaways

• SSH honeypots, particularly Cowrie, offer versatile platforms for security research and creative educational challenges, moving beyond simple malware collection to enable interactive, story-driven experiences.
• Interactive, hands-on workshops are highly effective for diverse skill levels, especially when carefully designed with step-by-step instructions, dynamically generated content (like unique credentials), and self-contained infrastructure (e.g., Docker images).
• Lessons from running large-scale security workshops include preparing for unreliable network conditions (never rely on conference internet!) and prioritizing engagement and fun over excessive security paranoia in low-stakes learning environments.
• Honeypots can be transformed into immersive, narrative-rich puzzles that encourage creative problem-solving, integrate diverse technical challenges (steganography, JWT, cellular automata, AI), and foster a sense of "play" in hacker culture.
• Generative AI (LLMs) can create dynamic and engaging challenge elements, as seen with the "M" command-line assistant, but they have limitations, particularly with strictly adhering to finite state grammars or precise rule-based text generation.
• The talk strongly advocates for injecting "surprise, absurdity, and play" into security tools and education to keep hacker culture vibrant, demonstrating that learning can be both profound and enjoyable.

About the Speaker(s)

Ryan Mitchell, known by his handle clugist at the Packet Hacking Village, is a software engineer by profession. He explicitly states he is "not a networking or an IT person" and has spent his entire career in software engineering. Mitchell is a dedicated enthusiast who contributes to DEF CON out of passion, buying his own tickets and taking vacation time from his full-time job. He is a published author of two books: Unlocking Python, which teaches Python from foundational computer science principles, and Web Scraping with Python, which recently celebrated its 10-year anniversary and is now in its third edition. Additionally, he develops and teaches courses for LinkedIn Learning, including the popular "Python Essential Training." His involvement at DEF CON's Packet Hacking Village showcases his commitment to fostering community, education, and the creative application of technical skills in cybersecurity.