Lessons Learned Hunting IoT Malware
Olivier Bilodeau (Cybersecurity Research Lead · GoSecure)
SecTor 2016 · Day 2 · Tech 1 (701A)
Overview
In this insightful talk from SecTor, Olivier Bilodeau, Cybersecurity Research Lead at GoSecure, delves into the escalating threat of Internet of Things (IoT) malware and shares critical lessons learned from years of active hunting. The presentation comes at a particularly salient time, following high-profile distributed denial-of-service (DDoS) attacks reaching unprecedented scales, such as the 1.5 terabit per second assault on OVH and the attack on Brian Krebs's website, both largely attributed to compromised IoT devices like cameras and DVRs. Bilodeau highlights that these attacks, leveraging botnets of hundreds of thousands of devices, are pushing the limits of internet infrastructure and defense capabilities.

Key moments
- 0:00 Introduction: The scale of IoT DDoS attacks
- 4:00 Core vulnerabilities of insecure IoT devices
- 12:00 Honeypot architecture for IoT malware collection
- 25:00 Reverse engineering challenges: stripped ELF binaries
- 34:00 Case Study: MIPS dropper bypassing honeypots
- 44:00 Case Study: Linux.Moose, social media botnet
- 51:00 Proposed solutions for the fundamental IoT security problem
Lessons Learned Hunting IoT Malware
Speakers: Olivier Bilodeau, Cybersecurity Research Lead, GoSecure
Conference: SecTor
YouTube: https://www.youtube.com/watch?v=zh3iyAotFuI
Overview
In this insightful talk from SecTor, Olivier Bilodeau, Cybersecurity Research Lead at GoSecure, delves into the escalating threat of Internet of Things (IoT) malware and shares critical lessons learned from years of active hunting. The presentation comes at a particularly salient time, following high-profile distributed denial-of-service (DDoS) attacks reaching unprecedented scales, such as the 1.5 terabit per second assault on OVH and the attack on Brian Krebs's website, both largely attributed to compromised IoT devices like cameras and DVRs. Bilodeau highlights that these attacks, leveraging botnets of hundreds of thousands of devices, are pushing the limits of internet infrastructure and defense capabilities.
Bilodeau, with a background spanning embedded Linux development and cybersecurity research at ESET and GoSecure, brings a unique perspective to the challenges of understanding and combating IoT threats. Unlike traditional malware, IoT variants operate on resource-constrained devices lacking conventional user interfaces and endpoint security, making detection and analysis inherently difficult. This talk outlines a proactive, honeypot-centric approach to collecting live IoT malware samples, dissecting their technical intricacies, and deriving actionable intelligence for the security community.
The core of Bilodeau's research focuses on overcoming the significant hurdles in acquiring and reverse-engineering IoT malware. By detailing a custom-built, low-interaction honeypot architecture and innovative data analysis techniques, he provides a roadmap for researchers to confront this growing threat. The presentation culminates in three real-world case studies – a sophisticated MIPS dropper, the notorious Lizard Squad botnet, and the stealthy Linux Moose – illustrating the diverse tactics employed by adversaries and offering crucial defensive implications for vendors, internet service providers, and end-users alike.
Background
▶ Watch: Introduction: The scale of IoT DDoS attacks (0:00)
The proliferation of IoT devices has introduced a vast and vulnerable attack surface to the internet. These devices, ranging from smart cameras and DVRs to home routers and even industrial sensors, are characterized by their low-cost design, minimal CPU and memory, and often cheap software development practices. A critical factor contributing to their insecurity is the lack of a general-purpose user interface (UI). While many might offer a web or mobile interface, they rarely provide a desktop-like environment where users can install third-party software, including crucial endpoint security solutions. This absence of traditional security agents means that there's no widespread collection mechanism for IoT malware samples, leaving researchers largely "in the dark."
The problematic IoT devices are typically those that expose administrative channels like Telnet or SSH directly to the internet, often running a full BusyBox userland. Compounding this issue is the pervasive use of default global credentials (e.g., admin/admin), which are rarely changed by end-users. Many users also inadvertently expose these devices by plugging them directly into the internet without proper router-based network segmentation, driven by a desire for remote access without understanding the security implications.
Traditional malware collection and analysis methods fall short for IoT. Techniques like soldering a serial UART port for direct hardware access are not scalable for large-scale studies. Asking users for samples is ineffective because most don't realize their devices are compromised, and extraction requires specialized knowledge. While hunting for ELF binaries on platforms like VirusTotal can yield results, it often surfaces an "echo chamber" of already known or slightly modified samples, primarily compiled with different timestamps, rather than novel threats. The sheer volume of undifferentiated ELF binaries, including benign system tools like ls, further complicates this approach, making it difficult to discern actual malware without extensive manual analysis. This critical gap in sample collection necessitates a more proactive and automated approach, leading to the adoption of honeypots as a primary strategy.
Key Findings
▶ Watch: Honeypot architecture for IoT malware collection (12:00)
Bilodeau's research reveals several key findings and contributions to the understanding and mitigation of IoT malware:
- Honeypots as a Scalable Collection Solution: Recognizing the limitations of traditional malware collection for IoT, the team successfully shifted to a proactive, honeypot-based approach. They developed a low-interaction, software-based honeypot architecture capable of luring attackers and collecting live samples and interaction logs at scale, deployed across various geographic locations. This addresses the critical need for diverse, real-world IoT malware samples.
- Challenges of IoT Malware Reverse Engineering: The talk highlights the significant difficulties in reversing statically linked, stripped ELF binaries, which are common in IoT malware. These binaries lack debugging symbols and import tables, making functions indistinguishable from vast blocks of C library code. Identifying the actual malware logic amidst hundreds of undifferentiated functions (e.g., distinguishing
printffamily calls) is an enormous and unscalable task using traditional methods.
- Innovative Reverse Engineering Technique: To overcome the reverse engineering challenge, Bilodeau proposes and demonstrates a novel technique:
- Reproducing the target environment (architecture, C library, compiler version) using toolchains like OpenWrt.
- Building the same libraries with debugging symbols.
- Using BinDiff (now free from Google) to automatically map known library functions to the unknown functions in the stripped malware binary.
This method allows researchers to filter out the vast amount of library code, focusing only on the truly malicious functions, significantly accelerating analysis. In one instance, it allowed the import of over 225 functions with high confidence.
- Evolution of Attack Tactics: The case studies illustrate the evolving sophistication of IoT malware. The MIPS dropper demonstrates attackers' awareness of honeypot limitations, bypassing
wgetdetection by performing in-line data transfer directly throughecho -Ecommands. This indicates a move towards more advanced staging techniques, akin to those seen in Windows malware.
- Diverse Monetization and Operational Models:
- Lizard Squad (Gasket), while infamous for DDoS attacks, is shown to be operated by numerous, often uncoordinated, actors leveraging leaked source code. Their C2 infrastructure via IRC and plain-text password storage for their "stresser" service reveal a lack of operational security, despite their widespread impact.
- Linux Moose represents a distinct, stealthy botnet focusing on fraudulent social media engagement (selling fake followers/likes). Its use of a SOCKS proxy and HTTPS for C2 traffic, combined with a lack of persistence, demonstrates an effort to remain under the radar while monetizing illicit activities directly on the clear web.
- Effective HTTPS Man-in-the-Middle for IoT: For the Linux Moose analysis, the team successfully implemented an HTTPS man-in-the-middle (MITM) proxy by mimicking security appliance certificates. This allowed them to decrypt and analyze the clear-text social media interaction traffic, providing unprecedented insight into the botnet's monetization mechanisms and target demographics (e.g., models, singers).
Technical Deep Dive
▶ Watch: Reverse engineering challenges: stripped ELF binaries (25:00)
The core of Bilodeau's methodology relies on a robust and flexible honeypot architecture designed to collect IoT malware samples and attacker interactions. This setup is entirely comprised of open-source and free software components, emphasizing scalability and resilience.
The honeypot environment is deployed on cloud instances and features a standalone design—meaning no push-back to a "mothership" during operation. Logs and pcaps are pulled daily via SSH keys, ensuring that if a honeypot is compromised, it can be discarded and recreated without losing valuable data or exposing the research infrastructure.
Key components of the honeypot include:
- QEMU with Debian-based ARM Image: To achieve high-interaction capabilities for executing collected binaries, a QEMU virtual machine running a Debian-based ARM image (specifically, a "Debian armel" image) is used. This provides a full Linux userland environment where actual ELF binaries can be executed and observed without risking the host system.
- Kippo (now Kojin) Honeypot: The primary low-interaction honeypot is Kippo, a Python-based SSH honeypot built on the Twisted framework for asynchronous I/O. Kippo features an emulated file system and easily configurable command outputs, allowing it to mimic various device types (e.g., routers, insulin pumps). A significant contribution by Bilodeau was the implementation of Telnet support for Kippo, which was crucial given the prevalence of Telnet as an infection vector for IoT malware. This feature was subsequently merged upstream, benefiting the wider community. Kippo also offers a valuable log replay mechanism, allowing researchers to visualize attacker interactions in real-time, discerning between automated scripts and human operators.
- Man-in-the-Middle (MITM) Proxy: For specific investigations, such as the Linux Moose botnet, an HTTP/S MITM proxy was integrated. After evaluating alternatives like Bettercap (which proved unstable and lacked structured output), the team settled on a custom solution based on a stable MITM proxy, despite its requirement for manual
iptablesmanagement. This proxy was essential for intercepting and decrypting encrypted traffic (e.g., HTTPS over SOCKS tunnels) to understand botnet C2 communications. - IPTables: Used to strategically forward incoming traffic (e.g., port 23 for Telnet, port 22 for SSH) to the respective honeypot components, while also ensuring the actual SSH management interface remains accessible on a non-standard port and filtered from packet captures.
- Dumpcap: From the Wireshark project, Dumpcap is employed for full packet capture (pcap) of the entire honeypot environment. This provides raw network data, serving as a forensic fallback in case log data is incomplete or for detecting honeypot fingerprinting attempts. Dumpcap's privilege separation enhances security.
Data Analysis Pipeline:
Given the immense volume of logs and pcaps generated, a centralized data analysis approach is adopted:
- A single server pulls all collected data.
- IPython Notebooks serve as a collaborative IDE, allowing researchers and interns to interactively run Python code blocks for analysis.
- Scalable Python libraries like Pandas, NumPy, and SciPy are heavily utilized for efficient data manipulation and analysis, especially for reading and writing large CSV files.
- GNU Parallel is a crucial tool for parallelizing CPU-bound or I/O-intensive tasks. It allows researchers to run
tsharkone-liners across multiple pcaps simultaneously, leveraging all available CPU cores for efficient log reordering and data extraction. - Specific parsers, such as the
man-in-the-middle proxylibrary, are used for structured log data.
Reverse Engineering Stripped Binaries:
The most challenging aspect of IoT malware analysis is dealing with statically linked, stripped ELF binaries. These binaries contain the entire C library (e.g., uClibc, glibc, musl) embedded within them, leading to large files with no debugging symbols or external function imports. To address this, Bilodeau advocates for a specific workflow:
- Environment Reproduction: Identify the target device's architecture (MIPS, ARM), C library, and compiler version.
- Toolchain Utilization: Use toolchains like those provided by OpenWrt, which are commonly targeted by attackers, to build the identified C library and other common libraries (e.g., OpenSSL if present) with full debugging symbols.
- BinDiff for Symbol Mapping: Leverage BinDiff (now freely available from Google) to compare the stripped malware binary against the symbol-rich library builds. BinDiff automatically identifies and maps functions with high similarity, allowing researchers to mass-import symbols for common library calls (e.g.,
VFprintf_internal). This effectively filters out the noise, leaving only the truly malicious, custom-written code for focused analysis. This technique was successfully demonstrated, importing over 225 functions with 99% confidence for auClibc 0.9.33-based malware.
Demo / Proof of Concept
▶ Watch: Case Study: Linux.Moose, social media botnet (44:00)
Bilodeau illustrated the honeypot's effectiveness and the evolving tactics of attackers through a vivid log replay of an infection involving a MIPS dropper. This demonstration highlighted how adversaries are adapting to honeypot environments.
During the replay, an attacker successfully gained a shell on the emulated system. Instead of using traditional methods like wget to download the next stage of malware—a common technique that honeypots like Kippo are designed to intercept and log—the attacker employed a more evasive maneuver. They utilized echo -E commands to perform in-line data transfer of the binary directly into a file on the emulated filesystem. The echo -E command, combined with backslash escapes, allowed the attacker to stream the raw binary data, including the ELF header, character by character, and redirect it to a newly created file.
The sequence observed was:
- Attacker gains shell access.
echo welcome(part of their fingerprinting).- Attempts to
cd /tmp(which might fail depending on honeypot emulation). echo -E 'PXF...<binary_data>...' > /tmp/draw(the in-line data transfer).chmod +x /tmp/drawto make the file executable.- Attempt to execute
/tmp/draw.
In this specific low-interaction honeypot setup, while the file transfer was successful, the execution of the native ELF binary failed with a "command not found" error, precisely because the low-interaction honeypot does not execute native binaries directly. This outcome, however, is a success for the researcher, as the full binary sample is captured for later analysis. This demo underscored the need for honeypots to evolve and capture stage-two payloads, prompting Bilodeau's future plans to integrate QEMU in bin-for-map mode for automated execution and collection of these subsequent stages.
Another significant "proof of concept" was the successful HTTPS Man-in-the-Middle (MITM) attack against the Linux Moose botnet. This wasn't a live demo during the talk, but a detailed explanation of a method critical for their research. Because Linux Moose communicated with social media platforms (Instagram, Twitter, YouTube) via HTTPS over a SOCKS proxy, the traffic was encrypted and obscured. To understand the botnet's monetization (selling fake followers/likes), the researchers deployed their custom MITM proxy. By configuring the proxy to present certificates that mimicked those of common security appliances (e.g., Blue Coat, Checkpoint), they found that the botnet did not flag the certificate warnings. This allowed them to successfully terminate the SSL connection, inspect the clear-text HTTP traffic, and then re-encrypt it for forwarding to the legitimate social media sites. This technique provided invaluable insights into the botnet's operations, including the specific APIs used, the nature of the fraudulent interactions, and even the types of customers purchasing the services (e.g., models, musicians). The findings from this research were embargoed at the time of the talk but were slated for release at Black Hat Europe.
Defensive Implications
▶ Watch: Proposed solutions for the fundamental IoT security problem (51:00)
The insights gleaned from hunting IoT malware offer crucial defensive implications for various stakeholders:
- IoT Device Vendors:
- Per-Device Credentials: The most critical recommendation is for vendors to abandon default global credentials. Instead, each device should ship with unique, strong credentials, ideally generated during manufacturing and perhaps printed on the device itself (e.g., leveraging the MAC address). This simple change would significantly raise the bar for automated brute-force attacks.
- Secure Default Configurations: Vendors must be extremely cautious about exposing management services like Telnet and SSH to the internet by default. These services should be disabled by default or, if necessary for remote access, require explicit user configuration and strong authentication.
- No Persistence: While not a complete fix, malware like Linux Moose that lacks persistence (i.e., is removed after a reboot) highlights that a simple reboot can temporarily clean a device. Vendors could explore ways to make device reboots more effective at clearing transient malware.
- Internet Service Providers (ISPs):
- Telnet Blocking: Given the escalating scale of IoT DDoS attacks, Bilodeau suggests that ISPs might eventually be forced to seal off Telnet from the internet entirely. While this is a drastic measure that might be unpopular with some technologists, the current situation with terabit-scale DDoS attacks driven by compromised IoT devices may necessitate such a blunt intervention to protect the broader internet infrastructure. ISPs could also actively scan their networks for exposed Telnet/SSH services and notify customers.
- End-Users/Consumers:
- Change Default Passwords: The most immediate and impactful action for consumers is to change all default passwords on their IoT devices and home routers. Even writing the new password down and sticking it to the device is better than leaving easily guessable defaults.
- Network Segmentation: Users should understand the importance of not directly exposing IoT devices to the internet. Using a router with proper NAT (Network Address Translation) and firewall rules, or even segmenting IoT devices onto a separate VLAN, can significantly reduce their exposure to direct attacks.
- Regular Reboots: For malware without persistence, regular reboots of devices can help clear transient infections.
- Security Researchers and Industry:
- Continued Sample Collection: The talk emphasizes the ongoing need for active collection and analysis of IoT malware samples to stay ahead of evolving threats. Honeypots remain a vital tool for this.
- Improved Reverse Engineering Tools: The development of more robust FLIRT signatures and automated symbol mapping for embedded architectures and C libraries is crucial to scale reverse engineering efforts.
- Sharing Intelligence: Collaborative efforts, including sharing honeypot configs, analysis techniques, and threat intelligence, are essential to combat this widespread problem effectively.
Key Takeaways
- IoT Insecurity is a Critical Problem: The prevalence of default credentials, exposed management interfaces, and lack of endpoint security makes IoT devices highly vulnerable, leading to massive DDoS attacks.
- Honeypots are Essential for IoT Malware Collection: Traditional methods for acquiring malware samples are ineffective for IoT. Low-interaction, software-based honeypots provide a scalable and safe solution for proactive collection.
- Reverse Engineering IoT Malware Requires Specialized Techniques: Stripped, statically linked ELF binaries common in IoT malware necessitate advanced methods like
BinDiffwithOpenWrttoolchains to map library functions and focus on malicious code. - Attackers are Adapting: IoT malware authors are evolving their tactics, using methods like in-line data transfer to bypass honeypot detection and employing sophisticated C2 mechanisms like SOCKS proxies with HTTPS.
- IoT Botnets Have Diverse Monetization: Beyond DDoS, botnets like Linux Moose demonstrate profitable models based on fraudulent social media engagement, highlighting the varied motivations of attackers.
- Simple Fixes Can Have a Huge Impact: Implementing per-device credentials, restricting default internet exposure of management interfaces, and encouraging users to change passwords are low-hanging fruit that can significantly improve IoT security.
About the Speaker(s)
Olivier Bilodeau is a Cybersecurity Research Lead at GoSecure, bringing extensive experience in embedded systems and network security. Prior to his role at GoSecure, he conducted cybersecurity research at ESET for three years, where he gained significant experience in analyzing various forms of malware, including early IoT threats. His background as a developer on Linux systems and a network administrator provides him with a deep understanding of the technical intricacies of IoT devices, which he finds more intuitive to reverse engineer than traditional Windows malware.
Beyond his professional roles, Olivier is deeply involved in the cybersecurity community. He co-founded Montreal Hack, a hands-on security workshop that provides training through real CTF (Capture The Flag) exercises. He also serves on the council of the NorthSec security conference in Montreal, which combines conference talks with a competitive CTF event. Olivier is a seasoned speaker, having presented at prominent conferences such as 44CON, Virus Bulletin, Black Hat Europe, and Def Con, sharing his research and insights with a global audience. His work on IoT malware, including his contribution of Telnet support to the Kippo honeypot project, demonstrates his commitment to enhancing cybersecurity defenses through practical research and community engagement.
Reviews
Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT
Bilodeau didn't just deploy a honeypot and read the logs; he actually did the grueling work of reversing stripped uClibc binaries. Building OpenWrt toolchains to generate symbols and slamming them into BinDiff to filter out the noise is an elegant, massive time-saver. This is highly practical, no-bullshit research that reverse engineers can actually use.
Heather Calloway (CISO) — SOLID
Bilodeau provides a highly competent breakdown of how researchers hunt and reverse IoT malware using low-interaction honeypots. While the bulk of the presentation is a technical tutorial on overcoming stripped ELF binaries, the strategic takeaway for security leaders is the unpatchable, unmonitored risk these embedded devices pose when placed on corporate networks.