Emulating Embedded Linux Devices at Scale w LightTouch Firmware Rehosting

Emulating Embedded Linux Devices at Scale with Light-Touch Firmware Rehosting

Speakers: Sigusr Polke

Conference: DEF CON 33 (August 2025)

YouTube: https://www.youtube.com/watch?v=f-LTMUFQzjQ

Slides: https://media.defcon.org/DEF%20CON%2033/DEF%20CON%2033%20presentations/Sigusr%20Polke%20-%20Emulating%20Embedded%20Linux%20Devices%20at%20Scale%20with%20Light-Touch%20Firmware%20Rehosting.pdf

Overview

Vulnerability research on embedded Linux devices — the routers, access points, and network appliances that constitute much of the internet's physical layer — has historically required either physical hardware or a substantial investment in device-specific emulation. Sigusr Polke, from the advanced research team at CrowdStrike, presents a middle path: a "light-touch rehosting" methodology that emulates devices at the SoC level rather than the device level, enabling a single rehosting environment to run firmware from dozens of vendor products. The talk is grounded in a concrete worked example using the Broadcom BCM4709 SoC, producing a working emulated environment for ASUS, Netgear, TP-Link, D-Link, and other routers — all from one kernel build, with no physical hardware required.

Background

▶ Watch: Problem: traditional firmware emulation fails on complex embedded SoCs (2:13)

The embedded Linux security research ecosystem has converged on a standard approach: unpack a firmware image, chroot into the extracted filesystem on a stock ARM or MIPS QEMU machine, and use LD_PRELOAD to intercept and stub out calls that fail because the expected hardware is absent. Tools like Firmwalker, FIRMAE, and various derivatives of the Debian userspace approach have made this tractable for single-binary analysis.

The limitation of this approach becomes visible when the target software is designed to operate holistically — when a router's web management interface depends on an NVRAM subsystem, a GPIO layer, network interface detection, and JFFS2 persistent storage all cooperating correctly. Patching each incompatibility via LD_PRELOAD becomes a progressively more fragile exercise, and the resulting environment often cannot run more than isolated binaries. Full services don't start. The web interface returns errors. Honeypots can't be constructed.

A fundamentally different approach is full-system emulation of a specific device: build a QEMU machine that mimics the hardware exactly and boot the vendor's kernel image directly. This works but requires hardware documentation, data sheets, and months of effort per device. The payoff — one working emulated device — rarely justifies the investment.

Polke's rehosting concept occupies the space between these extremes. The key insight is that SoC supply chains create natural groupings: Broadcom, Qualcomm, and MediaTek each supply SoCs to dozens of vendors. Netgear, ASUS, and TP-Link don't build their own chips. They buy a SoC and build userspace on top of the manufacturer's SDK, which provides the kernel, drivers, and low-level libraries. Userspace code is vendor-specific; everything below it is largely manufacturer-provided and consistent across the device family.

If you can build a kernel that userspace cannot distinguish from the real one — without running on the actual hardware — you can run firmware from any device in that SoC family.

Key Findings

▶ Watch: LightTouch approach: all adaptation work confined to the kernel layer (6:48)

• A single rehosting environment targeting the Broadcom BCM4709 SoC supports firmware from multiple vendors (ASUS, Netgear, TP-Link, D-Link, and others) with minimal per-device adaptation.
• The first vendor emulation for a new SoC requires roughly one day of effort; subsequent vendors in the same family typically require about one hour.
• The primary worked example — an ASUS AC87U router on Broadcom BCM4709 — reached approximately 95% functionality, with SSH access, running processes and services, and a fully operational web management interface, all without physical hardware.
• No LD_PRELOAD hooks and no userspace modifications are required. The firmware image used is the same binary that would be flashed to the real device.
• The methodology is portable: the team began porting the approach to additional SoC families after the BCM4709 prototype.

Technical Deep Dive

▶ Watch: Kernel patching: configuring flash layout and hardware abstraction (20:19)

The Rehosting Concept

The core principle is to do all adaptation work in the kernel, not in userspace. Userspace can only interact with what the kernel exposes. If the kernel environment is indistinguishable from the real device's kernel — from userspace's perspective — then userspace will behave as it would on real hardware.

The practical implementation:

1. Build a mainline Linux kernel at a version compatible with the firmware's expected kernel (for BCM4709 firmware, this is around 2.6.36.4).
2. Compile it for the closest QEMU machine with a compatible architecture (VersatileExpress A9 for BCM4709's Cortex A9 core).
3. Load the firmware filesystem and observe what breaks.
4. Add kernel patches, dummy drivers, and QEMU machine modifications reactively as each failure is diagnosed.

Resources available without vendor cooperation: QEMU machines, mainline kernel source, firmware images, reverse engineering tools, compiled kernels and kernel modules extracted from firmware, GPL source code releases, and open-source alternative firmware projects (OpenWRT, DD-WRT) that target the same SoCs.

Primary Target: Broadcom BCM4709

The BCM4709 (Cortex A9 core, integrated Ethernet stack) is deployed across a broad range of consumer routers from ASUS, Netgear, TP-Link, D-Link, and smaller OEMs. The ASUS AC87U (2014, end-of-life 2021) served as the primary specimen.

Firmware acquisition: from the ASUS support page in TRX format (Broadcom's packaging format). Unpacking with Unblob (preferred over binwalk for precise carving without post-processing artefacts) yields an LZMA-compressed kernel and a SquashFS filesystem. The vendor kernel version is 2.6.36.4b for BRCM ARM, compiled with GCC 4.3/4.5.3 using Buildroot 2010.2/2012.

Boot process inspection reveals the init target is /sbin/pre-init — a symlink to the rc binary, which dispatches behavior based on its invocation name and PID (it must run as PID 1 to execute the full initialization routine). The filesystem layout uses many symlinks into /tmp, which the init process populates at boot.

QEMU Configuration

The VersatileExpress A9 machine provides a Cortex A9 core with NOR flash support, matching the BCM4709's CPU family. Key configuration requirements:

• SquashFS with XZ compression: Mainline 2.6.36 does not include XZ decompression for SquashFS (added in 2.6.38). Fix: backport XZ support from 2.6.38 or extract it from ASUS's GPL code release.
• Flash layout: 64 MB NOR flash with a precisely defined mtdparts layout. The SquashFS filesystem lives at mtdblock2. Additional named partitions (BRCM, NAND, ASUS) are required for JFFS2 mounts that the init process expects.
• Kernel compilation: All required drivers compiled in (no module loading), SquashFS and MTD enabled.

Kernel Patches and Dummy Drivers

Each failure mode during iterative boot testing required a targeted fix:

NVRAM (/dev/nvram): The most critical peripheral. The Broadcom userspace NVRAM library (libnvram) expects a character device at /dev/nvram supporting nvm_init, nvm_get, nvm_set, nvm_commit, and nvm_getall operations. The storage format is a contiguous buffer of null-separated key=value pairs, backed by a flash partition. Polke implemented a kernel character device replicating this interface exactly. Defaults are populated by the init process itself (no physical device dump required; reset keys exist in the init logic).

Network Interfaces: The BCM4709's Ethernet stack expects multiple physical interfaces (eth0, eth1, etc.). The VersatileExpress machine uses an SMSC 911x NIC. A custom QEMU machine ("BCM479") was created by cloning VExpress and adding additional Ethernet interfaces with proper IRQ assignments. The SMSC driver was patched to intercept the get_if_status ioctl and return "physically up" (all bits high) regardless of actual link state, satisfying the router software's network readiness checks.

GPIO (/dev/gpio): Many router services interact with GPIO pins for LED control and hardware signaling. A dummy kernel device was implemented that accepts ioctl calls and returns a neutral state without crashing. Active-low/active-high behavior from device tree entries was preserved.

JFFS2 Partitions: The named partitions (BRCM, NAND, ASUS) in the mtdparts layout are mounted by pre-init via /proc/mtd. Adding them to the flash layout and mtdparts boot argument was sufficient.

Bridge/LAN Access: A host tap interface in the 192.168.1.0/24 subnet, with eth1 on the QEMU guest taking 192.168.1.1, established bidirectional connectivity without requiring a full bridge emulation.

Cross-Vendor Results

Once the BCM4709 environment was functional for the AC87U, other vendors' firmware was loaded with minimal modification:

• Other ASUS BCM4709 devices: worked immediately, sharing the same SDK base.
• Netgear: required identifying the specific device in the firmware; web interface functional after network configuration.
• TP-Link: NVRAM stored at /dev/cfm with a different format (CRC32 header followed by key-value store). Required a new NVRAM driver variant, approximately one hour of work.
• D-Link: Web interface functional with no additional changes.
• FCOM (Chinese vendor): Functional out of the box.

Demo / PoC

▶ Watch: Debugging: suppressing NVRAM errors that halt emulation boot (24:47)

The demo showed the ASUS AC87U emulated environment:

• SSH access to the emulated router shell, with running processes visible via ps.
• The router's web management interface loading in a browser, with full login, configuration pages, and settings accessible.
• Network services active and reachable from the host.

The ~5% non-functional portion relates to the Quantenna SoC that handles the AC87U's 802.11ac radio — a separate hardware component that is outside the BCM4709 SoC family and was not emulated.

Subsequent demos showed the same environment booting D-Link and FCOM firmware with no significant modification, demonstrating the cross-vendor applicability of the single rehosting build.

Defensive Implications

▶ Watch: Success: full userspace running with network stack accessible (38:15)

For vulnerability researchers: The methodology substantially lowers the barrier to embedded device security research. A researcher can now test firmware across an entire product family without acquiring physical units. This is particularly valuable for EOL devices (such as the AC87U, which left ASUS support in 2021) that remain deployed in the field but are unavailable for purchase.

For threat intelligence and honeypots: Running emulated router firmware as a honeypot captures realistic attacker behavior against actual router software — not simplified simulations. Attack telemetry from emulated Netgear, ASUS, and TP-Link devices running genuine firmware is substantially more valuable than generic Linux honeypots.

For vendors: The fact that a team can achieve ~95% functionality in a rehosting environment without source code or hardware access underscores the importance of security review during the SDK and firmware development phase, rather than post-release. Manufacturer-provided SDK components shared across vendor products represent a single fix that benefits the entire device ecosystem.

For defenders: Embedded routers and access points running EOL firmware represent a persistent attack surface. The rehosting methodology makes it tractable to run large-scale automated vulnerability scanning across entire firmware families — which means both researchers and adversaries can now do this. Firmware update policies and EOL device replacement timelines should account for the expanding attack surface of firmware-level analysis tools.

Key Takeaways

• SoC supply chains create natural device families: one rehosting environment per SoC supports firmware from dozens of vendors.
• All adaptation work is done in the kernel. Userspace firmware runs unmodified, including the same binary that ships on the physical device.
• The BCM4709 rehosting environment achieved ~95% functionality for the ASUS AC87U and ran firmware from Netgear, TP-Link, D-Link, and others with one day of initial effort and roughly one hour per subsequent vendor.
• NVRAM emulation via a kernel character device is the most critical enabler: the /dev/nvram interface must match Broadcom's libnvram API exactly.
• The methodology is portable to other SoC families, multiplying the potential scope of the work.
• This lowers the cost of embedded device vulnerability research and honeypot deployment significantly, with implications for both defensive research and offensive capability.

About the Speaker

Sigusr Polke is a security researcher on the advanced research team at CrowdStrike. Their work focuses on embedded systems security, with an emphasis on firmware emulation and vulnerability research at scale. The rehosting methodology presented here emerged from the team's need to conduct vulnerability research and deploy honeypots across large numbers of embedded Linux devices without physical hardware access.