Reverse engineering RTL8761B* Bluetooth chips, to make better Bluetooth security tools & classes

Reverse engineering RTL8761B* Bluetooth chips, to make better Bluetooth security tools & classes

Speakers: Xeno Kovah, Founder, Dark Mentor LLC

Conference: DistrictCon

YouTube: https://www.youtube.com/watch?v=lgpkDE6dzvU

Overview

In a revealing talk at DistrictCon, Xeno Kovah, founder of Dark Mentor LLC and Open Security Training, unveiled the intricate journey of reverse engineering Realtek RTL8761B* series Bluetooth chips. This deep dive into the silicon's inner workings aims to unlock unprecedented control over Bluetooth communications, enabling researchers and educators to craft arbitrary packets, spoof device addresses, and conduct advanced machine-in-the-middle (MiTM) attacks. The presentation meticulously detailed the challenges faced, from undocumented MIPS 16E architecture to the absence of public SDKs and debug interfaces, culminating in the ability to run custom firmware on ubiquitous, low-cost USB Bluetooth dongles.

Kovah's motivation stems from a critical need within the Bluetooth security research and education community. Existing tools and hardware are often outdated, expensive, or lack the granular control required for modern Bluetooth specifications (e.g., Bluetooth 5.4). By reverse engineering these widely available Realtek chips, Kovah and his team at Open Security Training are paving the way for more effective, accessible, and up-to-date Bluetooth security classes. This research not only empowers future vulnerability hunters but also highlights significant implications for device security, supply chain integrity, and the fundamental architectural limitations of the Bluetooth Host Controller Interface (HCI).

Background

▶ Watch: Custom firmware goal for Bluetooth packet control (0:00)

The landscape of Bluetooth security research has long been constrained by hardware limitations. Many researchers, including Kovah's team, have been forced to rely on CSR 4.0 dongles, which support only the Bluetooth 4.0 specification (circa 2010), severely limiting the types of packets that can be sent and received. This antiquated hardware is a major roadblock for teaching and researching modern Bluetooth protocols, such as Bluetooth 5.4, which is critical for understanding current device vulnerabilities.

A core challenge in Bluetooth security is the inability to spoof Bluetooth Low Energy (BLE) public addresses or Bluetooth Classic (BC) addresses using standard, architecturally defined Host Controller Interface (HCI) commands. While other BLE address types can be easily changed, public and classic addresses are hardcoded, making MiTM attacks against devices using these address types difficult without non-standard methods. This limitation directly impacts the ability to conduct realistic MiTM demonstrations for educational purposes and advanced security research.

Kovah's research was driven by multiple motivations:

1. Enhancing Open Security Training (OST) Classes: The goal was to update the upcoming Bluetooth 1501 class to demonstrate machine-in-the-middle attacks using modern Bluetooth 5.4 capabilities, which requires the ability to spoof public addresses and send custom packets.
2. Improving Bluetooth Printing: Building upon Veronica Kovah's Black Hat 2020 research, which uncovered over-the-air arbitrary code execution vulnerabilities in Bluetooth chips from Texas Instruments and Silicon Labs, Xeno Kovah sought to improve "Bluetooth printing" – active fingerprinting of Bluetooth hardware. Existing tools like Cracktooth (an academic tool for sending low-level Link Manager Protocol (LMP) packets) were found to be suboptimal: proprietary, x86-64 only (high power, expensive hardware), difficult to obtain (using obsolete Espressif dev boards), and costly ($55 for the dev kit, plus an x86 box). The aim was to replace this with small, cheap USB dongles ($4-$15) capable of sending arbitrary LMP packets.
3. Leading by Example and Mentorship: Kovah intends this research to serve as a guide for a "crew" of Bluetooth security enthusiasts, demonstrating a practical reverse engineering methodology for a specific chip, and encouraging them to apply similar techniques to other Bluetooth silicon vendors.

The chosen target hardware—ubiquitous, inexpensive USB Bluetooth dongles—primarily featured **Realtek RTL8761B* chips, specifically the BUE and BUV variants for USB interfaces, and BTE and BTV for UART. These dongles are often falsely advertised (e.g., claiming Bluetooth 5.3/5.4 support when only providing 5.1), highlighting a supply chain issue. The reverse engineering effort faced significant hurdles: the chips use a MIPS 16E architecture, which is not natively well-supported by tools like Ghidra; there are no public or leaked SDKs or documentation; and the ROM contains almost no useful strings, making initial analysis extremely challenging. Furthermore, the absence of JTAG** or other standard debug interfaces meant a purely black-box approach was necessary.

Key Findings

▶ Watch: Practical motivations: RCE, Bluetooth printing, replacing Cracktooth (6:00)

The extensive reverse engineering effort yielded several critical findings that collectively enabled full control over the Realtek RTL8761B* chips:

• MIPS 16E Architecture: The Realtek RTL8761B* chips were confirmed to use a MIPS 16E instruction set, a 16-bit encoding of 32-bit MIPS instructions. This discovery was crucial for configuring the disassembler and understanding the chip's core logic.
• Firmware Location and Structure: The necessary firmware for these chips is distributed within common Linux distributions, found at /lib/firmware/rtlwbt/ (e.g., rtlwbt_fw_b.bin, rtlwbt_fw_bu.bin). This firmware, along with an appended configuration file, is loaded into the chip's RAM.
• Vendor-Specific Commands (VSCs) are Key: The standard Bluetooth HCI architecture provides an abstraction layer, but limits direct control over low-level protocols like Link Layer Control Protocol and Link Manager Protocol (LMP). Realtek, like other vendors, implements vendor-specific commands that bypass these limitations. These VSCs are identifiable by an Opcode Group Field of 0x3F, which translates to specific leading bytes (FC, FD, FE, FF) in the 16-bit opcode.
• Firmware Loading Mechanism: The chip's firmware is loaded in 253-byte fragments using a specific vendor-specific command with opcode 0x20. The configuration file, which includes critical settings like the Bluetooth Device Address (BD_ADDR), is concatenated at the end of the firmware image.
• Memory Read Capability: A crucial vendor-specific command, FC61 (sent as 0x61FC over UART), was identified. This command, despite being named "read flash" (the chip lacks actual flash memory), allows arbitrary memory reads from the device. This became the primary tool for memory mapping and dumping the chip's internal state.
• Memory Map Discovery: Through systematic scanning using the FC61 command, a partial memory map was constructed. Key regions identified include a ROM at the 0x80000000 address range and RAM at 0x80100000, along with various aliased regions and likely memory-mapped I/O. Invalid memory accesses returned a distinctive DEADBEEF pattern, aiding in boundary identification.
• Critical String and Function Identification: Despite the scarcity of strings in the dumped ROM, a manual search revealed a handful of highly relevant strings like "HCI command," "events," "LMP," and "LC" (Link Controller). Cross-referencing these strings led to the discovery of a central function responsible for processing incoming HCI commands. Further analysis of this function, specifically its handling of the 0x3F Opcode Group Field, confirmed it as the vendor-specific command handling function. This function acts as a dispatcher, calling specific sub-functions for different VSCs.
• Enabling Custom Firmware: By understanding the memory layout, the firmware loading process, and the vendor-specific command handling, Kovah's team successfully developed and deployed custom firmware. This custom firmware enables the direct crafting and sending of arbitrary Bluetooth packets, including those required for spoofing public BD_ADDRs and sending low-level LMP packets.

These findings collectively provided the necessary insight to bypass architectural limitations, gain full control over the Bluetooth controller, and achieve the project's goals of advanced packet manipulation.

Technical Deep Dive

▶ Watch: MIPS 16E discovery and RISC-V connection for analysis (16:00)

The technical journey of reverse engineering the Realtek RTL8761B* chips was a testament to persistent, methodical analysis in the face of significant obstacles. The core of the chip's processing unit was identified as a MIPS 16E processor. This is a compressed instruction set architecture, analogous to ARM's Thumb mode, where 16-bit instructions are used to encode a subset of the full 32-bit MIPS instruction set. This presented an immediate challenge for disassembly, as standard Ghidra MIPS analysis initially struggled, producing "instruction daddy" errors. The workaround involved manually selecting MIPS 32-bit little endian and specifically enabling MIPS 16E disassembly. However, a recurring problem was Ghidra's inability to correctly disassemble code segments immediately followed by data, a common pattern in embedded firmware. This was overcome by a custom Ghidra script developed by Veronica Kovah, which intelligently navigates and disassembles these interleaved sections.

The starting point for code analysis was the firmware files found in /lib/firmware/rtlwbt/ on Linux systems, such as rtlwbt_fw_b.bin and rtlwbt_fw_bu.bin. These files are the binaries loaded into the chip's RAM at boot. Initial Ghidra analysis of these raw binaries at offset zero revealed header information, followed by a large block of zeros, and then the actual code. Key to understanding the chip's interaction mechanism was the Host Controller Interface (HCI). HCI defines a standard interface between the host (e.g., Linux) and the Bluetooth controller (the Realtek chip). While HCI provides standard commands (e.g., Set Random Address, Inquiry), it explicitly abstracts away the lowest layers of the Bluetooth stack, such as the Link Layer Control Protocol (LLCP) and Link Manager Protocol (LMP). To gain full control over these low-level packets, direct manipulation at the firmware level of the controller is necessary.

This is where vendor-specific commands (VSCs) become paramount. The Bluetooth specification reserves an Opcode Group Field of 0x3F for vendor-specific commands. When this 6-bit field is shifted and combined with the 10-bit Opcode Command Field to form a 16-bit HCI command, it results in opcodes starting with 0xFC, 0xFD, 0xFE, or 0xFF. These specific byte patterns became crucial markers during reverse engineering, indicating potential VSC handlers.

A significant breakthrough came from interacting with a Realtek development board (specifically a BTV variant via UART) using the vendor's own RF test tool on Windows. By enabling a hidden "BT menu" within this tool, access was gained to functions like "BT RW reg" for reading and writing registers. While direct register access proved unhelpful initially, observing the tool's communication with the dev board via a Saleae logic analyzer was transformative. This revealed that a vendor-specific command, FC61 (transmitted as 0x61FC due to little-endian byte order), was being used to read memory. This command typically took an address and a length (e.g., 0x20 for 4 bytes) and returned the corresponding data.

Armed with the FC61 memory read primitive, a script was developed to systematically scan the chip's entire 32-bit address space. This process involved sending FC61 commands with varying addresses and lengths over the UART interface and recording the responses. Invalid memory regions consistently returned a DEADBEEF pattern, allowing for precise mapping of accessible memory. This revealed a ROM region starting around 0x80000000 and a RAM region starting around 0x80100000, along with various aliased memory blocks and suspected memory-mapped I/O. The entire ROM contents were then dumped and loaded into Ghidra for static analysis.

Despite the scarcity of embedded strings in the ROM, a manual search eventually uncovered a few crucial ones: "HCI command," "events," "LMP," and "LC." Tracing cross-references to these strings led to a central function. Further analysis of this function showed it performing checks against input values, notably the 0x3F Opcode Group Field. This confirmed it as the vendor-specific command handler, responsible for dispatching VSCs to specific sub-functions. This critical function serves as the gateway to understanding and manipulating the chip's deepest functionalities. By identifying these handlers, researchers can now craft their own VSCs to trigger desired low-level operations, such as spoofing BD_ADDRs or sending arbitrary LMP packets, capabilities previously unattainable through standard HCI.

Demo / Proof of Concept

▶ Watch: Reverse engineering 'Read Flash FC61' vendor command (32:00)

While the talk itself did not feature a live, real-time demonstration, Xeno Kovah explicitly stated the achieved outcome and provided the necessary steps for others to replicate the results. The core proof of concept derived from this extensive reverse engineering is the ability to load custom firmware onto the Realtek RTL8761B* USB dongles, thereby gaining full, unarchitected control over Bluetooth packet transmission.

Kovah described the end result: "alakazam, boom, all of a sudden these dongles can send packets that architecturally in Bluetooth specification you are not supposed to be able to have full control of these packets. They don't want you to send these packets, but I want you to. So, that's what I made possible." This directly refers to overcoming the architectural limitations of standard HCI commands, particularly the inability to spoof Bluetooth Low Energy public addresses or Bluetooth Classic addresses, and the inability to send arbitrary Link Manager Protocol (LMP) packets.

The practical demonstration of this capability involves:

1. Installing Custom Firmware: Users can run install_dark_firmware_real.sh or a series of sudo copy commands on a Linux system.
2. Plugging in a USB Dongle: Once the custom firmware is in place, plugging in a target USB dongle (such as an RTL8761BUE or RTL8761BUV based device) results in it running the custom firmware.

With this custom firmware, the dongle can now perform actions previously impossible:

• Spoofing Public BD_ADDRs: Essential for conducting realistic machine-in-the-middle attacks against devices that use these fixed address types.
• Crafting Arbitrary LMP Packets: This enables pre-authentication vulnerability research, allowing interaction at the lowest layer of the Bluetooth stack, independent of pairing or higher-level authentication mechanisms. This is a critical capability for advanced "Bluetooth printing" and vulnerability hunting, building on previous research into over-the-air code execution.

Essentially, the research provides the "magic's biggest secrets revealed" behind how to transform a commodity Bluetooth dongle into a powerful, fully controllable security research tool, capable of manipulating Bluetooth communications at a depth previously restricted to vendor-specific, often proprietary, and inaccessible hardware.

Defensive Implications

▶ Watch: Finding critical strings and vendor command handler in ROM (37:00)

The detailed reverse engineering of Realtek RTL8761B* Bluetooth chips carries several significant defensive implications for manufacturers, developers, and end-users of Bluetooth-enabled devices.

1. Firmware Integrity and Signing: The ability to load custom firmware onto these chips underscores the critical importance of robust firmware signing and verification mechanisms. If an attacker can replace the legitimate firmware with a malicious version, they gain complete control over the device's Bluetooth stack, enabling sophisticated attacks. Manufacturers should ensure that their Bluetooth controllers only execute cryptographically signed firmware, ideally with secure boot processes that verify integrity from the very first instruction.
2. Supply Chain Security and Trust: The prevalence of cheap, white-label USB dongles that falsely advertise Bluetooth specification support (e.g., claiming 5.4 but delivering 5.1) highlights a broader supply chain security issue. Users and enterprises often purchase these devices without understanding their true capabilities or underlying firmware. This research demonstrates that these widely available, low-cost components can be compromised and repurposed for malicious activities. Organizations should exercise caution in procuring Bluetooth hardware and, where possible, opt for trusted vendors with verifiable security practices.
3. Bluetooth Address Spoofing and Tracking: The ease with which public Bluetooth Device Addresses (BD_ADDRs) can now be spoofed has direct implications for privacy and tracking. While BLE introduced randomized addresses to mitigate tracking, the continued use of public addresses in many devices, or the Bluetooth Classic address, makes them vulnerable to identification and MiTM attacks. Defenders should be aware that BD_ADDRs can no longer be trusted as unique or fixed identifiers. Device tracking and authentication mechanisms relying solely on these addresses are inherently insecure.
4. Pre-Authentication Attack Surface: The capability to craft and send arbitrary Link Manager Protocol (LMP) packets at the lowest layer of the Bluetooth stack opens a vast pre-authentication attack surface. This means that devices can be attacked before any pairing or authentication takes place, potentially leading to denial-of-service, information leakage, or even remote code execution, as demonstrated by previous research. Manufacturers must ensure that their Bluetooth stacks are highly resilient to malformed or unexpected LMP packets, implementing robust input validation and error handling at the earliest possible stage.
5. Importance of Deep Device Analysis: This research serves as a stark reminder that even commodity, seemingly benign hardware components can hide deep vulnerabilities and offer avenues for attack when thoroughly reverse engineered. Defenders, especially those developing or deploying critical infrastructure with Bluetooth components, should invest in or commission deep technical analysis of their chosen hardware's firmware and underlying silicon.
6. Education and Awareness: The work by Open Security Training aims to educate a new generation of security researchers. As more individuals gain the skills to perform this type of low-level analysis, the volume of discovered Bluetooth vulnerabilities is likely to increase. Defenders need to stay abreast of the latest research and attack techniques to adequately protect their systems.

In essence, the findings necessitate a shift from relying on architectural abstractions to a more granular, firmware-level understanding of Bluetooth security, urging proactive measures across the entire device lifecycle.

Key Takeaways

• Unprecedented Bluetooth Control: Reverse engineering Realtek RTL8761B* chips allows loading custom firmware onto commodity USB dongles, granting full control over Bluetooth packet crafting, including the ability to spoof public BD_ADDRs and send arbitrary Link Manager Protocol (LMP) packets.
• Modernizing Bluetooth Security Research: This research overcomes the limitations of outdated hardware (e.g., CSR 4.0 dongles) and suboptimal tools (e.g., Cracktooth), providing a path to conduct advanced Bluetooth security research and education using modern Bluetooth specifications (e.g., 5.4) with affordable and widely available hardware.
• Vendor-Specific Commands are Critical: Deep interaction with Bluetooth controllers beyond standard HCI specifications requires understanding and exploiting vendor-specific commands, identifiable by specific Opcode Group Field values (0x3F) and their corresponding byte patterns (FCxx, FDxx, FExx, FFxx).
• Practical Reverse Engineering Methodology: The talk demonstrates a robust methodology for embedded device reverse engineering, including effective Ghidra workarounds for MIPS 16E, leveraging open-source intelligence, using logic analyzers for protocol analysis, and systematic memory mapping via custom scripts, even in the absence of documentation or debug interfaces.
• Significant Defensive Implications: The ability to manipulate low-level Bluetooth packets facilitates advanced MiTM attacks and pre-authentication vulnerability research. This emphasizes the need for strong firmware signing, scrutiny of the Bluetooth supply chain, and robust defenses against malformed low-level packets to mitigate new attack vectors.
• Empowering Future Researchers: This work directly supports Open Security Training's mission to provide high-quality, free security education, serving as a blueprint and mentorship guide for a new generation of Bluetooth security researchers to explore and secure the vast ecosystem of Bluetooth devices.

About the Speaker(s)

Xeno Kovah is the founder of Dark Mentor LLC. The majority of his time is dedicated to his non-profit organization, Open Security Training (OST), where he and his team create and provide free, high-quality security classes that are comparable to those costing thousands of dollars. Approximately 25% of his time is spent on consulting and research, which he describes as being "for funsies" and also serves as a "Trojan horse" to promote the mission of Open Security Training, which is to make everyone better at security.

Veronica Kovah, Xeno's wife and collaborator, was mentioned for her significant contributions, including a Black Hat 2020 talk where she discovered over-the-air arbitrary code execution vulnerabilities in Bluetooth silicon from Texas Instruments and Silicon Labs. She also developed the crucial Ghidra script that enabled proper disassembly of the MIPS 16E code with interleaved data segments, which was essential for this research.