Smart Devices, Dumb Resets? Testing Firmware Persistence in Commercial IoT

Smart Devices, Dumb Resets? Testing Firmware Persistence in Commercial IoT

Speakers: Matei Josephs, Senior Security Researcher, Happening

Conference: DefCamp

YouTube: https://www.youtube.com/watch?v=tRnwqe5ffFc

Overview

In his compelling DefCamp talk, "Smart Devices, Dumb Resets? Testing Firmware Persistence in Commercial IoT," Matei Josephs, a Senior Security Researcher at Happening, unveiled a critical, yet often overlooked, vulnerability in the supply chain of pre-owned smart devices. Josephs demonstrated how returned IoT devices, particularly those categorized by retailers with terms like "pre-owned" or even "pre-loved," are often resold without adequate security checks, allowing for the persistence of malicious firmware. This research, also presented at Defcon's IoT village, highlights a significant risk to consumers and organizations, as seemingly innocent second-hand gadgets could harbor persistent backdoors, enabling surveillance, data exfiltration, or participation in botnets.

Josephs' talk serves as a stark warning about the potential dangers lurking in the burgeoning market of refurbished and returned electronics. His experiments, focusing on readily available and easily hackable devices, expose a systemic failure in how retailers manage the security integrity of items re-entering the market. The implications extend far beyond individual privacy, suggesting avenues for targeted attacks against businesses and individuals, making this a crucial topic for anyone involved in cybersecurity, consumer electronics, or supply chain management.

The core of Josephs' investigation revolves around proving the hypothesis that retailers' inspection processes for returned IoT devices are insufficient to detect firmware modifications. By successfully implanting persistent, custom firmware on common internet routers and having them re-listed for sale, he illustrated how a threat actor could weaponize the secondary market. This talk underscores the urgent need for industry-wide changes in how pre-owned smart devices are handled, from manufacturer design to retailer processing, to safeguard against sophisticated, yet surprisingly simple, supply chain attacks.

Background

▶ Watch: Introduction: Hacking pre-owned IoT device firmware (0:00)

The genesis of this research stems from Matei Josephs' personal experience seven years prior, as a "broke student" returning a used, broken inflatable kayak. To his surprise, the retailer accepted the return without question, refunding his $150, and later resold the kayak at a 25% discount. This incident sparked an idea: if physical items could be returned and resold with minimal scrutiny, what about smart devices with modifiable firmware? Could the return policy be leveraged as a "deposit" for borrowing items, or worse, for planting persistent malicious code?

Josephs, while not claiming to be an IoT hacking expert, credits the vibrant cybersecurity community for inspiring and supporting his research. He specifically acknowledged Andrew Bellini for an introductory hardware hacking talk at Defcon, which empowered him to pursue his own projects. Other notable mentions included Michael Copola's Defcon 20 presentation on the Router Post Exploitation Framework (RPF), Dennis Giza's work on robot vacuum cleaner cameras, and Matt Brown's extensive research videos on YouTube. Josephs emphasized that his goal was not to showcase unparalleled expertise, but to encourage others to delve into IoT hacking, highlighting the fascinating aspect of cyber actions having tangible physical consequences, as exemplified by his previous experience escaping a self-check-in kiosk mode to print a custom conference badge.

The underlying problem this research addresses is the lack of robust security practices throughout the lifecycle of consumer IoT devices, particularly when they transition from one owner to another. Many devices are designed without strong secure boot mechanisms or tamper-detection features, making them susceptible to firmware modification. When these devices are returned to retailers, the focus is often on basic functionality checks (e.g., "does it turn on?") rather than deep integrity assessments. This creates a fertile ground for adversaries who can exploit this gap to inject persistent backdoors, transforming everyday gadgets into tools for espionage, data theft, or distributed denial-of-service (DDoS) attacks. The economic incentive for retailers to quickly process and resell returned items, coupled with a presumed lack of advanced technical security expertise for firmware analysis, further exacerbates this vulnerability.

Key Findings

▶ Watch: Research Hypothesis: Retailers fail to check returned devices (8:30)

Matei Josephs' research definitively confirmed his central hypothesis: "Retailers do not implement enough checks to prevent reselling backdoor devices." Through his methodical approach, he demonstrated that the security posture of the secondary IoT market is alarmingly weak, creating a significant attack vector.

The primary findings of his experiments were:

1. Insufficient Retailer Checks: Josephs found that retailers' inspection processes for returned IoT devices were superficial at best. Devices were merely checked to see if they powered on and if their basic LEDs illuminated. There was no evidence of any deeper functional testing, let alone firmware integrity verification. This meant that his modified firmware, which maintained all original device functionalities while adding a hidden backdoor, went completely undetected.
2. Persistence Despite Resets: A critical finding was the persistence of the modified firmware even after users (or retailers) performed a factory reset. Josephs specifically engineered his backdoor to reside in a part of the firmware that was not affected by the standard reset button, which typically only clears configuration files. This ensured that once the malicious firmware was flashed, it would remain active unless a full re-flash of the original firmware was performed by a knowledgeable user.
3. Ignoring Physical Tampering: Even a device that Josephs had physically opened to pull firmware off the chip using SPI (breaking a seal in the process) was accepted for return and subsequently resold. This indicates a complete disregard for physical tamper evidence, further highlighting the lax security protocols in place.
4. Successful Backdoor Deployment and Callback: Out of 15 TP-Link routers returned to five different retailers, Josephs received callbacks from all of them. This included the 13 devices he successfully bought back and, critically, the two devices that were sold to other unsuspecting customers. These callbacks, in the form of curl requests to his controlled IP address, served as undeniable proof that the backdoor had persisted and was active in the wild.
5. Scalability and Impact: While Josephs' experiment involved a limited number of devices and a benign callback, he underscored the potential for full-scale backdoors, botnets leveraging residential IPs, or targeted surveillance using devices like robot vacuum cleaners and IoT cameras. He noted that the "resealed" tape and generic "personalized security" labels on returned devices did not correspond to any actual security verification. The self-destruct mechanism, designed to revert devices to their original state after 3,000 requests (approximately 21 days), eventually stopped the callbacks from the two devices sold to others, demonstrating both the potential for temporary persistence and a responsible approach to research.

In essence, Josephs' findings reveal a gaping hole in the supply chain security of pre-owned IoT devices, where retailers inadvertently act as vectors for distributing compromised hardware.

Technical Deep Dive

▶ Watch: Technical Breakthrough: Achieving firmware persistence via TFTP (15:30)

Josephs' research followed a structured methodology designed to prove his hypothesis. This involved identifying suitable devices, acquiring them, modifying their firmware, returning them, and then attempting to re-acquire them to confirm the persistence of the modifications.

1. Device Identification and Acquisition:

The first step was to select devices with easily overwritable firmware. Josephs targeted devices compatible with OpenWRT, a Linux distribution for embedded devices, as the process for flashing custom firmware is well-documented within this ecosystem. The rationale for choosing such devices was twofold:

• Vendor Trust & Customization: Users often flash OpenWRT to enhance customization, improve security, or simply because they "don't trust the vendors with our data," especially for devices like robot vacuums that map out homes.
• Cost-Effectiveness: Josephs emphasized the need for cheap devices, anticipating potential damage during hardware hacking attempts. He also posited that cheaper devices might be less secured due to wider adoption and real-world testing by users, or conversely, less scrutiny from manufacturers.

He ultimately chose TP-Link routers, describing them as "everybody's favorite IoT hacking beginner friendly devices." He ordered a total of 15 devices from five different retailers (three from each) to observe potential variations in return processing.

2. Firmware Modification:

The core of the technical deep dive involved modifying the device firmware to include a persistent backdoor. Josephs explored several approaches:

• Initial Extraction (SPI): His first attempt involved physically opening one device and pulling the firmware directly from the chip using SPI (Serial Peripheral Interface). This process, while technically sound, was fraught with challenges due to the need for "very very steady hands," which Josephs admitted he lacked. Crucially, opening the device also broke its seal, a detail that later proved irrelevant to the retailers.
• Firmware Analysis (Binwalk): With the firmware extracted, Josephs used Binwalk, a popular firmware analysis tool, to examine the file system. His goal was to inject a simple service that would execute a curl request to a controlled IP address every 10 minutes, acting as a beacon or "callback." Each request included a unique identifier for the specific device.
• Self-Destruct/Self-Restore Mechanism: To mitigate the ethical implications of leaving backdoored devices in the wild, Josephs integrated a "self-destruct" script. This script would disable the callback service and delete itself after 3,000 requests, effectively restoring the device to its original state after approximately 21 days of continuous operation. He clarified that "self-restore" was a more accurate term.
• Firmware Recreation (make squashfs): After modifying the file system, Josephs used make squashfs to recreate the firmware binary. He encountered an incompatibility issue where the initial firmware (rootfs.bin) was xz compressed, while his recreated one was zlib compressed. This was resolved by specifying the correct compression type (xz) with the make squashfs command, ensuring the modified firmware was as close as possible to the original to avoid detection or functional issues.

3. Firmware Flashing Challenges and Breakthrough (TFTP):

Flashing the modified firmware proved to be the most challenging part:

• Web Interface (Failed): Josephs initially attempted to upload the new firmware via the device's web interface. This failed, likely due to "signature checks" implemented by the manufacturer to prevent unauthorized firmware installations.
• SPI Flashing (Failed): Given his "rudimentary" hardware hacking setup and shaky hands, directly flashing the chip via SPI was also unsuccessful.
• TFTP Workaround (Success): The breakthrough came with the discovery that many devices, including the TP-Link routers, use TFTP (Trivial File Transfer Protocol) for recovery purposes. These devices, when in a specific recovery mode (often activated by holding the reset button during boot), query a TFTP server on a predefined local IP address, looking for a specific firmware file. By setting up a TFTP server with his modified firmware, Josephs was able to successfully flash the devices.

4. Persistence Mechanism:

Crucially, Josephs confirmed that even after applying the modified firmware via TFTP and then performing a standard factory reset (pressing and holding the reset button), the backdoor persisted. The reset button, in this context, only removed configuration files, leaving the underlying firmware—and thus the malicious code—intact. This finding cemented the core vulnerability: a factory reset, commonly perceived as a security measure, was ineffective against this type of firmware persistence.

Demo / Proof of Concept

▶ Watch: Broader Impact: Scalable, targeted attacks via fake stores (22:20)

The demonstration of Josephs' research wasn't a live, on-stage hack, but rather a meticulously documented and verifiable proof-of-concept spanning weeks of real-world interaction with retailers and devices. The entire process served as the "demo," illustrating the shocking ease with which a sophisticated attack could be deployed through the secondary market.

What was demonstrated:

1. Successful Firmware Implantation: Josephs demonstrated that he could successfully modify the firmware of commercial IoT devices (TP-Link routers) to include a persistent, hidden backdoor. This backdoor was designed to make a simple curl request to a controlled IP address, serving as a "beacon" or "callback."
2. Bypassing Retailer Checks: The primary demonstration was the complete failure of retailers to detect any tampering. All 15 modified devices were returned, accepted without question (even the one with a broken physical seal), and money was refunded. This proved that retailers' checks were, as Josephs stated, "superficial," merely ensuring the devices powered on and their LEDs functioned.
3. Persistence in the Wild: The most compelling part of the demonstration was the receipt of callbacks from all 15 devices after they had been re-listed as "pre-owned." This included the 13 devices Josephs managed to buy back and, critically, the two devices that were purchased by other, unsuspecting customers. These callbacks, logging the unique identifier of each device, provided irrefutable evidence that the malicious firmware had persisted through the return-and-resale cycle and was actively beaconing from new locations.
4. Resilience to Factory Resets: The callbacks from the devices, even after they were likely reset by retailers or new owners, demonstrated that the modified firmware was resistant to standard factory resets, which only cleared configuration data, not the underlying operating system.
5. Ethical Mitigation: While not a "hack" per se, the inclusion and successful execution of the "self-destruct" (or "self-restore") script within the modified firmware was an important part of the proof-of-concept. After approximately a month, Josephs observed that the callbacks from the two devices sold to other customers ceased, confirming that the script had successfully reverted the devices to their original, clean state after 3,000 requests (roughly 21 days), thus minimizing harm.

The visual evidence, such as the "resealed" tape on the boxes and generic "personalized security" labels, juxtaposed with the actual lack of security checks, further underscored the demonstration's core message: consumer confidence in the security of pre-owned devices is misplaced. The entire talk, therefore, served as a comprehensive, real-world proof-of-concept of a significant supply chain vulnerability.

Defensive Implications

▶ Watch: Q&A: Firmware persistence on Android devices and bootloaders (29:00)

The findings from Matei Josephs' research present severe defensive implications for multiple stakeholders: retailers, manufacturers, and end-users. Addressing this vulnerability requires a multi-faceted approach, moving beyond the current, inadequate practices.

For Retailers:

• Enhanced Inspection Protocols: Retailers must implement rigorous technical checks for all returned IoT devices. This goes beyond simply verifying if a device powers on. Inspections should include:
• Firmware Integrity Verification: Using cryptographic hashes or digital signatures to compare the installed firmware against a known-good version.
• Physical Tamper Detection: Thoroughly checking for broken seals, altered screws, or any signs of physical manipulation. Josephs' finding that a device with a broken seal was still resold is a critical failure point.
• Full Firmware Re-flashing: For any returned IoT device, a mandatory process should be to re-flash the official, cryptographically signed firmware from the manufacturer, ensuring any malicious modifications are overwritten.
• Security Training: Retail staff handling returned electronics need training on identifying potential security risks and understanding the importance of thorough inspections.
• Fraud Prevention Integration: Fraud prevention teams, typically well-funded, should be made aware of this attack vector and integrate it into their risk models, especially for large volumes of returns.

For Manufacturers:

• Implement Secure Boot: Devices should incorporate secure boot mechanisms that verify the integrity of the bootloader and firmware before execution, preventing unauthorized code from running.
• Locked Bootloaders: By default, bootloaders should be locked, requiring specific authentication or physical access to unlock them, making it harder for attackers to flash custom firmware.
• Encrypted Firmware: Encrypting firmware makes it more difficult for attackers to analyze, modify, and re-flash.
• Tamper-Evident Design: Incorporate tamper-evident seals (e.g., void stickers that visibly break if removed) and tamper-proof screws that require specialized tools, making physical access harder to conceal.
• Vulnerability Disclosure Programs (VDPs): Manufacturers should expand the scope of their VDPs and bug bounty programs to include vulnerabilities that require physical access, acknowledging that such attacks are feasible and impactful. Josephs directly argues that excluding physical access attacks from VDPs is a "bad idea."

For Consumers:

• Exercise Caution with Pre-Owned Devices: The most direct advice is to be extremely wary of buying "pre-owned," "refurbished," or "resealed" IoT devices, especially from unofficial channels like Facebook Marketplace. Josephs highlights that end-users, unlike retailers, are even less likely to detect tampering.
• Prioritize Reputable Vendors: When purchasing IoT devices, stick to reputable vendors who are known for their security practices and have active VDPs.
• Understand Factory Resets: Consumers should be aware that a factory reset might not be sufficient to remove malicious firmware, as demonstrated by this research.
• Engage with Security Programs: For technically inclined users, participating in bug bounty and VDPs for IoT devices can help improve overall device security.
• Consider Open-Source Firmware: Josephs provocatively suggests normalizing the purchase of devices without firmware, allowing users to flash their own open-source firmware. While he acknowledges this is not feasible for "90% of us" due to the loss of "plug-and-play" simplicity and low user adoption, it represents an ideal scenario for transparency and control.

Broader Impact and Threat Model:

The implications extend to creating botnets using compromised residential IPs for attacks, facilitating targeted surveillance (e.g., modifying robot vacuums to stream live video, or IoT cameras to publicly expose feeds regardless of settings), and even supply chain attacks at scale by malicious actors setting up their own online stores to distribute backdoored devices. While Josephs didn't achieve network access into the retailers' systems, he demonstrated the potential for deploying backdoors that could have such capabilities, or be used for subsequent attacks. The alleged actions of groups like the NSA's Tailored Access Operations, intercepting deliveries to implant devices, underscore that physical access attacks are a real-world threat, not just an academic exercise.

Ultimately, the defensive posture must evolve to recognize that the physical security and firmware integrity of IoT devices are intertwined with their cyber security, especially in a market increasingly reliant on reuse and resale.

Key Takeaways

Here are 4-6 main points from Matei Josephs' talk:

• Retailer Security Gaps: Retailers' processes for handling returned IoT devices are grossly inadequate, often limited to basic power-on checks, allowing physically tampered and firmware-modified devices to be resold undetected.
• Persistent Firmware Vulnerability: Malicious firmware can be persistently installed on IoT devices, often via recovery mechanisms like TFTP, and will survive standard factory resets, which typically only clear user configurations.
• Risks of Pre-Owned IoT: Buying "pre-owned" or "refurbished" IoT devices carries significant security and privacy risks, as they may harbor hidden backdoors enabling surveillance, data exfiltration, or botnet participation without the new owner's knowledge.
• Scalable Attack Vector: This vulnerability can be scaled by threat actors to create large botnets using residential IPs, conduct targeted surveillance (e.g., via robot vacuums or cameras), or even establish malicious supply chains through fake online stores.
• Call for Industry-Wide Security Enhancements: Manufacturers must implement stronger security features like secure boot, locked bootloaders, encrypted firmware, and tamper-evident designs. Retailers need to adopt rigorous technical checks and mandatory firmware re-flashing for all returned smart devices.
• Consumer Vigilance: Consumers should exercise extreme caution when purchasing second-hand IoT devices, prioritize reputable vendors, and understand that factory resets are not a panacea for firmware-level compromises.

About the Speaker(s)

Matei Josephs is a Senior Security Researcher at Happening, a role that allows him to delve into various aspects of cybersecurity. Beyond his corporate position, Matei is an entrepreneur, having founded Hifhack, a cybersecurity company he co-manages with his wife, Alexa. He also serves as the General Director of Concept Electronics, which he proudly notes as the oldest active IT company in Romania, balancing the agility of a startup with the established processes of a seasoned enterprise.

Matei describes himself as someone who simply "loves hacking stuff," a passion that extends beyond devices to web applications, people, and even buildings. He is particularly fascinated by IoT hacking due to the unique interaction where "changes or actions you make in a cyber in a virtual environment have consequences in a physical environment." This fascination was vividly illustrated by his anecdote of exploiting a conference self-check-in kiosk to print a custom badge, showcasing a full exploitation chain involving people, processes, and technology. Despite his impressive work, Matei humbly states he is "not an IoT hacker" expert, emphasizing his desire to encourage others to explore the field. He also jokingly refers to himself as a "junior husband," celebrating three years of marriage and hinting at a well-deserved promotion.