Invisible thieves in the front yard - from an advanced evasive edge-device attack to potential mitigation methods

Invisible thieves in the front yard - from an advanced evasive edge-device attack to potential mitigation methods

Speakers: Ting-Wei Hsieh, Malware Analyst, CHT Security Co

Conference: Virus Bulletin

YouTube: https://www.youtube.com/watch?v=Y35uKsHpTpo

Overview

In this compelling presentation at Virus Bulletin, Ting-Wei Hsieh, a Malware Analyst from CHT Security Co, unveiled the intricate details of a sophisticated espionage campaign that compromised a cloud service provider in Taiwan. The talk, titled "Invisible thieves in the front yard - from an advanced evasive edge-device attack to potential mitigation methods," meticulously dissected the threat actor's tactics, techniques, and procedures (TTPs), with a particular focus on their exploitation of edge devices like SSLVPNs, load balancers, and routers. Hsieh highlighted how these often-overlooked network perimeters become lucrative targets due to their direct internet exposure, lack of robust security controls, and high network throughput, which collectively create an ideal environment for stealthy operations.

The core of Hsieh's findings revolved around the discovery of a highly advanced and evasive malware family dubbed Magic Locksmith. This malware distinguished itself through its use of a fully dynamic magic packet pattern for command and control (C2) communication, coupled with a custom ECDH-based key exchange and a bespoke AES-like cryptographic implementation. These advanced evasion techniques render traditional network detection methods, such as static Suricata rules or even computationally intensive Lua scripts, largely ineffective. The talk not only provided a deep technical dive into these sophisticated mechanisms but also offered critical defensive implications and recommendations for both organizations and device manufacturers to bolster their security posture against such advanced threats.

The significance of this research cannot be overstated. It sheds light on the evolving landscape of cyber espionage, where threat actors are increasingly targeting critical infrastructure components like edge devices with highly customized and stealthy toolsets. By detailing the exploitation of a known vulnerability, CVE-2019-19778, and linking the campaign to the notorious BlackTech APT group, Hsieh's presentation serves as a stark reminder of the persistent and sophisticated threats faced by organizations globally. The insights provided are invaluable for security professionals seeking to understand, detect, and mitigate advanced persistent threats that operate with a high degree of technical sophistication and evasion.

Background

▶ Watch: Why attackers target vulnerable edge devices (1:28)

The persistent targeting of edge devices by advanced threat actors is a critical and growing concern in cybersecurity. As Ting-Wei Hsieh explained, these devices — including SSLVPNs, load balancers, and routers — are often the first line of defense, connecting internal networks to the vast and often hostile internet. Their inherent characteristics make them exceptionally attractive targets. Firstly, they are typically directly accessible from the internet, providing attackers with a clear attack vector from anywhere in the world. Secondly, and perhaps most critically, they often operate without the benefit of traditional security software such as antivirus or Endpoint Detection and Response (EDR) solutions. This absence of automated security monitoring means that once compromised, attackers can operate largely unimpeded. Thirdly, their role in handling high network throughput makes it incredibly difficult to detect anomalous connections or subtle malicious traffic patterns amidst legitimate data flows.

Hsieh underscored that the problem is exacerbated by operational realities, such as operators neglecting to update these devices for various reasons, including concerns about downtime or lack of established patching protocols. This negligence leaves known vulnerabilities unaddressed, creating persistent windows of opportunity for attackers. The talk referenced numerous prior incidents that validate this trend, including Mandiant/Google Cloud reports from 2023 detailing campaigns exploiting Fortinet vulnerabilities, followed by similar campaigns targeting Ivanti Connect appliances and Juniper routers in early 2024. Recorded Future also published a report on a campaign targeting Cisco devices, further illustrating the widespread nature of this attack vector. In the specific incident discussed, the threat actor exploited CVE-2019-19778, a vulnerability affecting Citrix Application Delivery Controller (ADC) and Gateway devices, which was already an n-day vulnerability at the time of the exploit.

A common tool in the arsenal of these threat actors, as observed in this campaign and others, is the Tiny Shell backdoor. Hsieh described Tiny Shell as an old, open-source project favored by attackers for its simplicity and versatility. Its code structure is straightforward, offering three core functionalities: downloading files (get file), uploading files (put file), and executing arbitrary shell commands (run shell). Its open-source nature makes it easy to customize, and its ability to run on various stripped-down or highly customized Linux and BSD distributions makes it ideal for diverse edge device environments. While it offers basic encryption to obscure traffic from rudimentary analysis tools, its default cryptographic scheme is acknowledged by its original author as insecure, primarily serving as a basic obfuscation layer rather than robust security.

To maintain stealth, attackers often employ a technique known as magic packet or port knocking. Instead of binding to an open TCP or UDP port, which can be easily detected via tools like netstat, or constantly connecting to a C2 server, which would show up as anomalous traffic, the malware remains dormant. It uses PCAP or BPF filters to passively inspect all inbound network packets, searching for a specific, predefined pattern. Upon detecting this "magic packet," the backdoor activates, enabling C2 communication. This technique allows the malware to remain hidden from network analysis tools for extended periods, making detection exceptionally challenging. While an old technique, dating back to the late 1990s or early 2000s, it remains remarkably effective, especially when combined with sophisticated, dynamic patterns, as seen in the Magic Locksmith malware. Previous intrusions often used static magic packet patterns, which allowed security researchers to develop Suricata rules for detection. However, the campaign Hsieh detailed introduced a significantly more advanced form of this evasion.

Key Findings

▶ Watch: Magic Locksmith's custom AES-like cryptographic implementation (12:00)

The investigation conducted by CHT Security Co. uncovered a highly sophisticated espionage campaign primarily targeting edge devices within a Taiwanese cloud service provider. The key findings illuminate the advanced capabilities and evasive tactics employed by the threat actor:

• Espionage Campaign Targeting Edge Devices: The core discovery was an ongoing espionage operation that successfully compromised critical edge devices, including SSLVPNs and load balancers, within the target environment. The actor's strategic focus on these devices provided direct access to the internet-facing infrastructure and high-value network traffic.

• Exploitation of CVE-2019-19778: The initial compromise vector was identified as the exploitation of CVE-2019-19778, an n-day vulnerability affecting Citrix Application Delivery Controller (ADC) and Gateway. This highlights the critical importance of timely patching, especially for internet-facing infrastructure.

• Discovery of Magic Locksmith Malware: A novel and highly evasive malware family, dubbed Magic Locksmith, was a central discovery. This backdoor employs advanced techniques to maintain stealth and hinder analysis, marking a significant evolution in threat actor capabilities.

• Dynamic Magic Packet for C2 Evasion: Magic Locksmith utilizes a fully dynamic magic packet pattern for C2 communication. Unlike static patterns, this dynamic approach generates up to 2^128 different potential patterns, making it virtually impossible to detect with traditional signature-based network intrusion detection systems (NIDS) like Suricata rules or even computationally intensive Lua scripts on high-throughput devices.

• Custom Cryptographic Implementations: The malware incorporates highly customized cryptographic schemes for its C2 communications. This includes an ECDH-based key exchange using hardcoded C2 public keys and a unique, AES-like symmetric encryption algorithm that deviates from standard AES by employing custom Rijndael constants (s-box, affine matrix, mixed columns coefficients). These customizations render standard cryptographic analysis tools ineffective.

• Sophisticated Data Exfiltration Methods: The threat actor employed a range of methods for credential and data exfiltration, including the widespread use of TCP dump to capture network traffic. Custom tools written in C and Golang were found on C2 servers, designed to parse PCAP files, extract specific HTTP headers, and dump credentials. Additionally, custom Python scripts were used to traverse and automatically download confidential documents from WebDAV servers based on specific keywords.

• Attribution to BlackTech APT: Based on strong similarities in magic packet constants and a unique modified RC4 implementation found in newly discovered Bifrost samples, the campaign was attributed with high confidence to the BlackTech APT group. This attribution underscores the advanced and persistent nature of the threat actor.

Technical Deep Dive

▶ Watch: Attacker's sophisticated data exfiltration techniques (14:00)

The technical sophistication of the threat actor, particularly in the design of the Magic Locksmith malware, represents a significant challenge to conventional security defenses. Ting-Wei Hsieh meticulously detailed two primary areas of innovation: the dynamic magic packet for C2 activation and the custom cryptographic implementations.

Dynamic Magic Packet Evasion

The magic packet technique, while not new, was significantly advanced in Magic Locksmith. Instead of relying on static, easily identifiable patterns, the malware implements a fully dynamic mechanism for C2 activation. This process involves:

1. Packet Data Acquisition: The malware intercepts the first 16 bytes of incoming packet data (whether TCP, UDP, or ICMP) and then the subsequent 16 bytes, totaling 32 bytes of raw packet data.
2. XOR Operation: These two 16-byte segments are then XORed together. This operation produces a 16-byte stream.
3. Pattern Matching: The resulting 16-byte stream must precisely match a hardcoded value embedded within the Magic Locksmith sample. This serves as the initial trigger for the backdoor's activity.
4. RC4 Key Derivation and Decryption: Crucially, the original first 32 bytes of the packet data (before the XOR operation) are used as an RC4 key. This RC4 key is then used to decrypt the subsequent data within the magic packet itself, which contains the C2 configuration details.

The advantage of this dynamic approach is profound. By generating a 16-byte stream through XORing two variable 16-byte segments, the number of possible patterns is astronomically high—up to 2^128. This makes it practically impossible to craft effective, static Suricata rules for detection. While Lua scripts could potentially implement such complex logic, running them on high-throughput edge devices is often not feasible due to performance overhead, leading to significant slowdowns or dropped packets. This mechanism allows the malware to remain dormant and undetectable by standard network analysis tools until the precisely crafted, dynamic magic packet is received.

Custom Cryptographic Implementations

Magic Locksmith deviates significantly from standard cryptographic practices, employing a custom ECDH-based key exchange and a bespoke AES-like symmetric encryption algorithm. This stands in stark contrast to the simpler, less secure encryption used by the original Tiny Shell backdoor.

Original Tiny Shell Encryption (for context)

The original Tiny Shell backdoor uses a basic, time-based seed and hardcoded secret/password for AES key derivation.

1. Upon establishing a TCP connection, the C2 sends a time-based seed to the implant.
2. The implant calculates a hash using a hardcoded secret and the received seed to derive an AES key.
3. The C2 performs a similar calculation using a user-provided password and the same seed to derive its AES key.
4. The C2 sends a hardcoded challenge, encrypted with its derived key, to the backdoor.
5. If the backdoor successfully decrypts the challenge using its derived key and the challenge matches, secure communication is established.

This scheme is explicitly noted as insecure by its author, primarily for hiding traffic from basic tools. Since the secret is hardcoded and each command uses a new session with new keys, an analyst with packet captures and the secret could potentially decrypt traffic.

Magic Locksmith ECDH Key Exchange

Magic Locksmith implements a more robust, albeit custom, Elliptic Curve Diffie-Hellman (ECDH) key exchange. A critical security feature is the hardcoded public keys from the C2 server within the malware sample, which effectively prevents man-in-the-middle (MITM) attacks by ensuring the backdoor only communicates with the legitimate C2. The process unfolds as follows:

1. Backdoor Key Pair Generation: The backdoor generates two independent random key pairs: KR (private key kr, public key PR) and KQ (private key kq, public key PQ).
2. Point Derivation (PS): The backdoor calculates a public point PS by multiplying its second random private key kq with the C2 server's hardcoded public key PC (i.e., PS = kq * PC).
3. Point Addition (PA): The backdoor then adds the public point PS to its first random public key PR to produce PA (i.e., PA = PS + PR).
4. Transmission to C2: The backdoor transmits two public points, PQ and PA, to the C2 server.
5. C2 Shared Secret Derivation: The C2, possessing its private key pc, can re-derive the shared secret. It can also recover the backdoor's first random public key PR from the received PQ and PA points.
6. Symmetric Key Seed: The recovered PR (the first random public key from the backdoor) is then used as the symmetric key seed for subsequent encrypted communication.

A key aspect of this scheme is that the traffic only contains PQ and PA. Without the C2's private key, it is computationally infeasible to derive PR and PS from the intercepted traffic, making passive decryption extremely difficult.

Magic Locksmith Custom AES-like Cryptography

Beyond the custom key exchange, Magic Locksmith employs a symmetric encryption algorithm that strikingly resembles AES (Rijndael) but uses custom, non-standard constants. This customization is why Hsieh named it "Magic Locksmith."

• Custom Rijndael Constants: The malware replaces the standard Rijndael constants with its own unique values. These include:
• S-box: A custom one-to-one byte mapping lookup table. While conceptually similar to AES's s-box, the specific byte transformations are different.
• Affine Matrix and Shift Vector: In standard AES, these are derived from a single line representing a shifted matrix. Magic Locksmith uses its own distinct parameters. Hsieh highlighted that the bit count for each custom parameter closely matches that of the standard AES constants, suggesting an intentional mimicry rather than a random deviation.
• Mixed Columns Coefficients: The coefficients used in the MixColumns transformation, a crucial step in AES for diffusion, are also custom. Hsieh presented a direct comparison, showing the green lines for standard AES coefficients versus the red lines for Magic Locksmith's coefficients, visibly demonstrating the alteration.

These deliberate modifications mean that standard AES decryption libraries and tools will fail to decrypt Magic Locksmith's traffic, even if the key is known. Analysts would need to reverse-engineer the custom constants and implement a bespoke decryption routine, significantly increasing the complexity and time required for analysis. This level of cryptographic customization is a hallmark of advanced threat actors aiming for maximum stealth and longevity of their implants.

Data Exfiltration Mechanisms

The talk also detailed the actor's robust data exfiltration pipeline:

• TCP Dump: The primary method for data collection was the pervasive use of tcpdump on compromised devices. Hsieh noted finding numerous PCAP files on both compromised devices and the C2 server's open directories.
• Custom Parsing Tools:
• C-based Binary: A custom C binary was discovered, specifically designed to target and filter packets from Citrix appliances within packet dumps, indicating a focused effort to extract data from specific victim systems.
• Golang Tools: Other Golang-based tools were used to parse collected PCAP files, specifically extracting HTTP headers to dump credentials. These credentials were then fed into other Golang tools designed to interact with the victim's WebDAV server.
• Python Automation: Custom Python scripts were found, configured to traverse the entire WebDAV server, searching for files containing specific keywords (redacted for confidentiality) and automatically downloading them. This demonstrates an automated and targeted approach to exfiltrate high-value, confidential documents.

The combined use of these highly evasive C2 techniques and sophisticated data exfiltration tools underscores the advanced capabilities of the threat actor.

Demo / Proof of Concept

▶ Watch: Key security lessons for organizations from the incident (19:00)

While the talk did not feature a live demonstration or proof-of-concept, the speaker presented detailed evidence and code snippets from the discovered malware samples and C2 server artifacts. This included images of the hardcoded cryptographic constants, visual comparisons of custom Rijndael components, and screenshots of attacker-used tools like TCP dump commands and custom C/Golang/Python scripts, providing a strong evidentiary basis for the technical claims.

Defensive Implications

▶ Watch: Recommendations for edge device manufacturers to improve security (22:00)

The sophisticated nature of the "Invisible thieves" campaign, particularly the advanced evasion techniques employed by Magic Locksmith, necessitates a multi-layered and proactive defensive strategy. Ting-Wei Hsieh outlined crucial recommendations for both organizations operating edge devices and the manufacturers of these devices.

Recommendations for Organizations

1. Apply Patches as Soon as Possible: The exploitation of CVE-2019-19778, an n-day vulnerability, was a critical enabler for this attack. Organizations must prioritize patching edge devices immediately upon vulnerability disclosure. Establishing clear protocols for updates, even for devices handling crucial services that cannot afford downtime, is paramount. This may involve implementing redundant systems or maintenance windows to minimize service disruption.
2. Avoid Weak or Reused Passwords: The discovery of brute-forcing tools on the C2 servers highlights the attacker's reliance on credential compromise. Enforcing strong, unique passwords and implementing multi-factor authentication (MFA) across all critical systems, especially those exposed to the internet, remains a fundamental security control.
3. Deploy Internal Honeypots: The client in this incident was alerted to anomalous internal traffic, triggering the investigation. Deploying honeypots or honeynets within the internal network can act as early warning systems, detecting lateral movement or unusual connection attempts that might otherwise go unnoticed by perimeter defenses. These early alerts can significantly reduce dwell time and mitigate potential damage.
4. Encrypt and Sign Confidential Data Transfer, Even Within LAN: The attacker's use of tcpdump to harvest credentials and sensitive data from unencrypted internal traffic underscores a critical vulnerability. All confidential data transfers, even within the local area network (LAN), should be securely encrypted and digitally signed. Encryption prevents passive eavesdropping, while digital signatures prevent man-in-the-middle (MITM) attacks where attackers might replace legitimate TLS certificates with fraudulent ones to decrypt traffic. Hsieh acknowledged that this can pose challenges for Web Application Firewalls (WAFs) that need to decrypt and inspect traffic. A potential workaround is to use dedicated WAF devices for specific services, separating them from other bundled services like SSLVPNs, thereby reducing the attack surface.

Recommendations for Manufacturers

Manufacturers of edge devices bear a significant responsibility in securing their products against advanced threats. Hsieh provided several key recommendations:

1. Privilege Separation Between Different Services: Edge devices often bundle multiple services (SSLVPN, load balancing, routing) onto a single platform. Manufacturers should implement robust privilege separation, ideally by isolating these services into different OS privilege containers or virtualized environments. This ensures that a compromise of one service (e.g., an SSLVPN vulnerability) does not grant the attacker full root access to the entire device or other critical components like the load balancer, effectively limiting the blast radius.
2. Provide On-Demand Integrity Validation Tools: To aid in post-exploitation scenarios, manufacturers should provide operators with on-demand integrity validation tools. These tools would allow administrators to verify the integrity of the device's operating system, firmware, and critical files, checking for unauthorized modifications or the presence of malware after a suspected intrusion.
3. Provide Low-Level Access for Forensic Tools to Licensed Users: In the event of a sophisticated intrusion, detailed forensic analysis is crucial. Manufacturers should offer low-level access (e.g., shell access, memory dumps, file system access) to licensed users or authorized incident response teams. This access is vital for recovering malware samples, identifying vulnerabilities, and determining the full scope of an attack, which is often impossible with standard administrative interfaces.
4. Log All Privileged Operations and Commands: Comprehensive logging of all privileged operations and executed commands on the target edge device is essential. This includes system-level calls, configuration changes, and execution of utilities like tcpdump. Even better, devices should be capable of alerting operators when unexpected or unauthorized commands are executed, allowing for immediate detection of post-exploitation activity and rapid incident response.

By adopting these defensive postures and implementing these manufacturer recommendations, organizations can significantly enhance their resilience against advanced, evasive attacks targeting their critical edge infrastructure.

Key Takeaways

• Edge devices are prime targets: Their direct internet exposure, lack of traditional security controls (AV/EDR), and high throughput make them highly attractive and vulnerable targets for sophisticated threat actors.
• Advanced evasion techniques are prevalent: Threat actors, particularly the BlackTech APT group, are employing highly dynamic magic packets for C2 activation and custom cryptographic implementations (ECDH, AES-like algorithms with custom constants) to bypass traditional network and security analysis tools.
• Patching is non-negotiable: Proactive and timely patching of known vulnerabilities, such as CVE-2019-19778, on internet-facing edge devices is paramount to prevent initial compromise.
• Defense must extend to the LAN: Internal network traffic should be encrypted and signed to prevent credential harvesting and data exfiltration via tools like tcpdump, even within seemingly trusted environments. Internal monitoring with honeypots can provide crucial early warnings.
• Manufacturers play a critical role: Device manufacturers must implement privilege separation, provide integrity validation tools, allow low-level forensic access, and ensure comprehensive logging and alerting for privileged operations to enhance the security posture of their products.
• BlackTech APT remains a significant threat: The attribution of this campaign to the BlackTech APT group highlights their continued sophistication, persistence, and ability to develop highly customized and evasive malware.

About the Speaker(s)

Ting-Wei Hsieh is a Malware Analyst at CHT Security Co. His professional interests span a wide range of malware analysis, including x86 and x64 Windows and Linux malware, .NET and Java-based malware, and various packer technologies. His work involves responding to sophisticated cyber incidents and dissecting advanced threat actor techniques, as demonstrated in his presentation on the Magic Locksmith malware.