From Pentest to Hacktivism: A Journey into 6 Zero Days Hidden in widely used Routers

From Pentest to Hacktivism: A Journey into 6 Zero Days Hidden in widely used Routers

Speakers: Danilo Erazo, Automotive Security Researcher, PCAutomotive

Conference: Ekoparty Miami

YouTube: https://www.youtube.com/watch?v=q_NLs_8swYM

Overview

This talk by Danilo Erazo, an Automotive Security Researcher at PCAutomotive, delves into a comprehensive penetration test and subsequent responsible disclosure process for a widely used series of Calix ONT (Optical Network Terminal) routers. Erazo's presentation, titled "From Pentest to Hacktivism: A Journey into 6 Zero Days Hidden in widely used Routers," highlights the critical security vulnerabilities found in these devices, which are prevalent across the Americas, particularly in South America. The "hacktivism" in the title refers to the speaker's persistent and ultimately successful efforts to compel the manufacturer to patch these severe flaws, even when initially dismissed due to the devices being considered "end-of-life."

The significance of this talk lies in its exposure of multiple zero-day vulnerabilities in end-of-life (EOL) routers that are still extensively deployed by Internet Service Providers (ISPs). Erazo uncovered six critical vulnerabilities, including root shells without passwords, hidden web servers, hardcoded credentials, and remote code execution (RCE) flaws. The talk not only details the intricate technical process of discovering these vulnerabilities—requiring hardware-level analysis and firmware extraction—but also chronicles the challenging journey of responsible disclosure, emphasizing the importance of persistence and the role of Computer Emergency Response Teams (CERTs) or National CSIRTs (CNA) in ensuring vendor accountability. The findings underscore a widespread security risk affecting potentially hundreds of thousands of users due to unpatched, insecure infrastructure.

Background

▶ Watch: Introduction: Six zero-days in widely used routers (0:00)

The focus of Danilo Erazo's research is on ONT routers, specifically those manufactured by Calix, which are commonly deployed in homes to convert fiber optic signals into usable internet connections (LAN, Wi-Fi). A key characteristic of these devices, as highlighted by Erazo, is that many are considered "offline" or end-of-life (EOL), meaning they no longer receive official support or security updates from the manufacturer. Despite this, they remain in active use across various countries, creating a significant attack surface.

The routers are complex, featuring two distinct internal systems or "computers": a Broadcom System-on-Chip (SoC) responsible for fiber optic connectivity, and a Quantenna System-on-Chip (SoC) handling Wi-Fi, LAN, and other internal network communications. This dual-system architecture introduces potential avenues for lateral movement and complex interactions that can be exploited.

Erazo embarked on this research after being contracted by an entity in Ecuador to analyze anomalies in user connections. His initial approach, typical of a penetration test, quickly escalated into a deep dive due to an unusual finding: the router models had no publicly recorded CVEs (Common Vulnerabilities and Exposures) despite being on the market for over a decade. This absence of reported vulnerabilities, coupled with the manufacturer not providing public firmware for analysis, necessitated a hardware-centric approach. The only way to analyze the firmware was to physically extract it from the device, a challenging task that ultimately revealed the extent of the hidden insecurities. This context sets the stage for a journey beyond conventional penetration testing, pushing into advanced hardware hacking and a protracted disclosure battle.

Key Findings

▶ Watch: Achieving root shell via UART using FTDI adapter (14:00)

Danilo Erazo's research uncovered a staggering six zero-day vulnerabilities in widely used Calix ONT routers, revealing critical security gaps that had persisted for over a decade. These findings demonstrate a severe lack of security hygiene and responsible product lifecycle management by the manufacturer.

The six key findings are:

1. UART Root Shell without Password: By accessing the UART (Universal Asynchronous Receiver/Transmitter) debug port on the Broadcom system, Erazo discovered a root shell that required no password for access. This vulnerability grants an attacker complete control over the Broadcom SoC simply by physically connecting to the device's debug pins.
2. Hidden Web Server with Hardcoded Credentials: A separate, undocumented web server was found running on the Quantenna system, accessible via a link-local IP address (169.254.x.x) that facilitated internal communication between the two SoCs. This web server was protected by hardcoded default credentials such as admin:admin, user:user, and s:s, providing unauthorized access to device configurations.
3. Hardcoded Wi-Fi Passwords: The firmware on the Quantenna system contained hardcoded Wi-Fi network names (SSIDs) and their corresponding passwords in plain text. This meant that anyone with access to the device's firmware or the hidden web interface could easily retrieve the Wi-Fi credentials, compromising the local network.
4. Command Injection (RCE1) on Quantenna System: Erazo identified a Remote Code Execution (RCE) vulnerability within a PHP script (tocand.php) on the Quantenna system. This script processed a user-supplied text variable from a POST request and directly passed it to the shell_exec() function without proper sanitization. This allowed for arbitrary command injection, giving an attacker remote control over the Quantenna SoC.
5. Lateral Movement between SoCs: The two distinct systems (Broadcom and Quantenna) communicated via a link-local IP interface. Exploiting the root shell on the Broadcom SoC allowed Erazo to perform lateral movement to the Quantenna SoC by leveraging this internal communication channel, effectively bridging the security domains of the two components. This also revealed that the Quantenna system's boot process was dependent on the Broadcom system pinging its link-local interface.
6. Second Command Injection (RCE2) on Port 6998: Later in the disclosure process, an anonymous contact revealed a separate RCE vulnerability on TCP port 6998. This command injection flaw, present in a service running on that port, could be exploited without physical access and was discovered to be circulating in black market circles. This second RCE further exacerbated the security posture of the routers.

These findings collectively painted a grim picture of the Calix ONT router series, highlighting multiple avenues for attackers to gain full control over the devices, access sensitive network information, and potentially pivot to other devices on the local or even wider ISP network.

Technical Deep Dive

▶ Watch: Command injection RCE in PHP script (28:00)

The technical journey to uncover these vulnerabilities was a testament to Erazo's persistence, blending hardware hacking with reverse engineering.

Firmware Extraction:

The initial hurdle was obtaining the firmware. Since Calix did not provide public firmware, physical extraction was necessary.

• Broadcom SoC Firmware: This proved to be the more challenging of the two. The firmware resided on a 48-pin flash memory chip. Extracting it required specialized hardware: an XGecu TL866II Plus programmer. Erazo noted the difficulty, stating that "duping this was super difficult, and we managed it." The process involved connecting the programmer to the flash chip's logical gate, counting addresses, and then dumping the contents.
• Quantenna SoC Firmware: This was comparatively easier. The firmware was stored on an 8-pin SPI Flash memory chip. A more common programmer, the CH341A, was sufficient for extraction.

Once extracted, initial analysis of the firmware revealed hardcoded IP addresses, kernel versions, and other details, but the real gems were hidden deeper.

UART Shell Discovery and Challenges:

Accessing the UART debug port on the Broadcom SoC was the first step towards a shell. However, initial attempts using a standard USB-to-UART adapter (often called a "bus pirate") resulted in garbage characters. This indicated a mismatch in baud rate or other communication parameters.

• Blind Shell with Logic Analyzer: To diagnose the issue, Erazo employed a logic analyzer. By connecting it only to the RX (receive) line of the UART, he could send commands blindly (e.g., cat /etc/passwd) via the bus pirate and observe the raw data received by the router. Processing these raw signals manually confirmed that a shell was indeed present and responding, albeit with garbled output to the bus pirate.
• FTDI Adapter Solution: The breakthrough came with the use of an FTDI adapter featuring the FD232RL chip. This specific chip provided a more robust and reliable serial communication interface, resolving the data loss and corruption issues. With the FTDI adapter, a clear root shell was achieved, revealing that the root user had no password (root::0:0 in /etc/passwd). This immediate root access was the first major zero-day.

Lateral Movement and Hidden Services:

With root access on the Broadcom SoC, Erazo observed the boot process. Crucially, the Broadcom system performed a ping to a specific link-local IP address (169.254.x.x) belonging to the Quantenna SoC. This revealed a critical inter-system dependency: the Quantenna system would only boot if it responded to this ping.

• Quantenna Link-Local Interface: Further investigation showed that this link-local interface hosted a hidden web server on port 80 and a Telnet server. The Telnet service, like the UART shell, allowed login with admin:admin, user:user, and s:s credentials, all without passwords. These credentials were found hardcoded within the Quantenna firmware, specifically in files like /etc/md_admin.conf, which also contained the plain-text Wi-Fi passwords and SSIDs. This constituted multiple severe vulnerabilities: a hidden service, hardcoded weak credentials, and exposed sensitive information.

Remote Code Execution (RCE1) on Quantenna:

Analyzing the Quantenna firmware, Erazo looked for common web vulnerabilities.

• Architecture Challenges: The Quantenna system ran on an ARM Compact architecture, which was less common than MIPS or ARM. This presented a challenge for reverse engineering tools. Erazo initially used Ghidra with a custom-developed fork for ARM Compact support, and also noted that Binary Ninja (with Jordan's help) could handle this architecture, and later acquired IDA Pro.
• Command Injection: The RCE was found in a PHP script named tocand.php. This script received a text variable via a POST request and directly passed it to the shell_exec() function without any input sanitization. This allowed for trivial command injection, enabling an attacker to execute arbitrary commands as the web server user, effectively achieving RCE on the Quantenna SoC.

Second Remote Code Execution (RCE2) on Port 6998:

A later, independent discovery, revealed through an anonymous contact, identified another critical RCE.

• Port 6998 Vulnerability: This RCE was a command injection vulnerability within a service listening on TCP port 6998. Unlike the first RCE, this flaw could be found and exploited without disassembling the router, making it even more accessible to attackers. This vulnerability was reportedly circulating in black market forums, indicating its prior knowledge and exploitation by malicious actors.

These detailed technical findings underscore the severity of the security posture of these Calix routers, with multiple layers of easily exploitable flaws.

Demo / Proof of Concept

▶ Watch: Manufacturer resistance: Disclosure timeline and EOL argument (42:00)

Danilo Erazo's presentation included several compelling demonstrations of the discovered vulnerabilities, showcasing their real-world impact and potential for exploitation. The proofs of concept illustrated not only individual flaws but also how they could be chained together for greater effect.

The demonstrations included:

1. Gaining a Root Shell via UART: Although not explicitly shown live, the process of achieving a root shell via the FTDI adapter on the Broadcom SoC was a foundational step. Erazo explained how this provided immediate, unauthenticated control over the device.
2. Exploiting Command Injection (RCE1) for Remote Access: Using a Python script, Erazo demonstrated exploiting the command injection vulnerability in tocand.php on the Quantenna system. This allowed him to establish a reverse shell back to his attacker machine (e.g., via Netcat on port 4444). From this shell, he could execute arbitrary commands on the Quantenna SoC.
3. Lateral Movement and Wi-Fi Password Extraction: Once a shell was obtained on the Quantenna system, Erazo showcased lateral movement. He detailed how, starting from the Broadcom root shell, he could telnet to the Quantenna's link-local IP (e.g., 169.254.x.x) using the hardcoded credentials. From the Quantenna shell (or RCE), he could then access files like /etc/md_admin.conf to retrieve the plain-text Wi-Fi SSID and password, effectively gaining free Wi-Fi access and further compromising the local network.
4. Cross-Router Exploitation via WAN: This was a particularly impactful demonstration. Erazo simulated an attack where he exploited a Calix router from another location, leveraging the ISP's WAN (Wide Area Network). By having knowledge of the victim router's private IP address within the ISP's network (which an attacker could potentially enumerate or guess), he was able to connect to the vulnerable services (e.g., the RCE on port 6998 or the Telnet service if exposed) and gain access to a router in a different "house" or network segment. This highlighted the potential for wormable attacks across an ISP's customer base.
5. Accessing Internal Network Resources: As a final illustration, Erazo showed how, once a router was compromised, he could interact with other devices on the local network. In one example, he demonstrated accessing a simple web server running on a connected Windows computer, proving that the compromised router could serve as a pivot point for further internal network attacks. The web server logs clearly showed the connection originating from the compromised router.

These demonstrations underscored the ease and breadth of exploitation, ranging from physical access leading to root, to remote code execution, lateral movement, and even cross-customer network compromise.

Defensive Implications

▶ Watch: Discovery of 6th zero-day RCE via anonymous collaboration (52:00)

The findings presented by Danilo Erazo carry profound defensive implications for Internet Service Providers (ISPs), network administrators, and end-users. The prevalence of these vulnerable Calix ONT routers across multiple countries, coupled with their "end-of-life" status, creates a significant and persistent security challenge.

Here are the key defensive implications:

1. ISPs Must Prioritize Patching and Replacement: The most critical implication is the responsibility of ISPs. The patches for these vulnerabilities are not publicly available to end-users; they must be deployed by the ISP. ISPs using these Calix models (e.g., those listed by Erazo, including those in the US, Ecuador, Argentina, Brazil, Mexico, etc.) must immediately apply the provided firmware updates or, for truly EOL devices that Calix refuses to patch, replace them with secure alternatives. Erazo's research revealed that over 35,000 such routers were exposed on Shodan at the time of the talk, making them easy targets for attackers.
2. End-Users Cannot Self-Patch: Consumers who own or rent these routers from their ISPs are largely helpless. They do not have access to the necessary firmware updates or the technical means to apply them even if they did. This emphasizes the burden on ISPs to protect their customers. If an ISP fails to patch, users should strongly consider demanding a router replacement.
3. Audit and Secure Link-Local Interfaces: The discovery of a hidden web server and Telnet service on a link-local IP interface (169.254.x.x) is a critical lesson. Network administrators should be aware that devices can host services on such interfaces, which are typically used for internal communication but can become exposed if not properly secured. Regular auditing of all network interfaces, including internal ones, for unexpected open ports and services is crucial.
4. Eliminate Hardcoded and Default Credentials: The presence of hardcoded root passwords, admin credentials (admin:admin, user:user, s:s), and plain-text Wi-Fi passwords is a fundamental security failure. Manufacturers must cease this practice, and ISPs should ensure that any devices they deploy have strong, unique default credentials that are forced to change upon first use.
5. Strict Lifecycle Management for EOL Devices: The case of these Calix routers highlights the dangers of EOL devices remaining in active deployment. Manufacturers and ISPs must establish and adhere to strict product lifecycle management (PLM) policies, including clear timelines for security support and mandatory replacement programs for devices reaching EOL. Continuing to use unsupported hardware is an open invitation for attacks.
6. Importance of Responsible Disclosure and CNA Involvement: Erazo's challenging journey to get Calix to issue patches underscores the need for robust responsible disclosure frameworks. When manufacturers are unresponsive or attempt to suppress CVEs, security researchers should leverage Computer Emergency Response Teams (CERTs) or National CSIRTs (CNAs). As Erazo demonstrated with Fluida Tax (a CNA in Colombia), these entities have the authority to compel manufacturers to act responsibly, ensuring vulnerabilities are acknowledged, patched, and publicly disclosed via CVEs.
7. Awareness of Black Market Exploitation: The discovery that one of the RCEs was already circulating in black market forums is a stark reminder that vulnerabilities in widely deployed devices are quickly weaponized. This necessitates proactive security measures and rapid response to disclosed flaws.

In summary, the defensive posture against these types of vulnerabilities requires a multi-faceted approach, with ISPs bearing the primary responsibility for ensuring the security of the devices they provide to their customers, and a strong emphasis on proper disclosure channels when vendors are uncooperative.

Key Takeaways

• End-of-Life (EOL) devices pose a significant and widespread security risk: Many EOL routers, like the Calix ONT series, remain widely deployed by ISPs, creating a massive attack surface due to unpatched vulnerabilities.
• Hardware-level analysis is crucial for deeply embedded systems: When firmware is unavailable, physical access to the device and techniques like firmware extraction (e.g., from 48-pin flash or SPI flash) and UART debugging (e.g., with FTDI adapters) are essential for discovering hidden flaws.
• Fundamental security failures persist in widely used devices: The discovery of unauthenticated root shells, hidden services with hardcoded default credentials, plain-text Wi-Fi passwords, and multiple command injection (RCE) vulnerabilities highlights a severe lack of basic security practices by manufacturers.
• Lateral movement between internal components is a critical attack vector: The dual-SoC architecture of the Calix routers (Broadcom and Quantenna) allowed for lateral movement between systems via a link-local interface, demonstrating how compromise of one component can lead to full device control.
• Responsible disclosure requires persistence and leveraging CNAs: Manufacturers may initially dismiss vulnerabilities or attempt to suppress CVEs, especially for EOL products. Researchers must be persistent and, if necessary, involve national CERTs or CNAs (e.g., Fluida Tax) to ensure ethical disclosure, patching, and public acknowledgment of vulnerabilities.
• ISPs bear the primary responsibility for customer security: End-users cannot patch these complex devices themselves. ISPs must actively apply firmware updates, replace vulnerable EOL routers, and implement robust security policies to protect their customer base from widespread exploitation.

About the Speaker(s)

Danilo Erazo is an Automotive Security Researcher at PCAutomotive. He identifies himself as someone deeply passionate about hacking, engaging in "24/7 hacking" in his free time. Originally from Ecuador, Erazo has made significant contributions to the cybersecurity community, including founding the Power Dive conference in Ecuador, which aims to expand its reach to the Galapagos Islands in future editions. He is also known by his YouTube channel, "Rever," where he shares his work. His talk at Ekoparty Miami showcased his expertise in hardware hacking, reverse engineering, and his commitment to responsible vulnerability disclosure.