Unveiling Malicious Behavior in Unknown Binaries
Tim Blazytko (Founder · mof)
Swiss Cyber Storm 2023 · Day 1 · Scenario
Overview
In an era increasingly plagued by sophisticated supply chain attacks, the ability to discern malicious intent within opaque software binaries has become paramount. Tim Blazytko, founder of mof, delivered a critical talk at Swiss Cyber Storm, addressing the pervasive challenge of trusting unknown executables. His presentation, "Unveiling Malicious Behavior in Unknown Binaries," provided a comprehensive guide for reverse engineers and security analysts, outlining both foundational and advanced strategies to identify hidden threats.

Key moments
- 0:00 Introduction: Analyzing unknown binaries for malicious behavior
- 3:25 What constitutes malicious content in an unknown binary?
- 7:30 Identifying malicious intent via suspicious API function calls
- 12:00 Live demo: Basic static analysis with Binary Ninja
- 20:10 RC4 crypto algorithm: A key malware detection heuristic
- 23:45 Live demo: Plugin for RC4 implementation detection
- 28:30 Q&A: Strategies for evading binary detection
Unveiling Malicious Behavior in Unknown Binaries
Speakers: Tim Blazytko, Founder, mof
Conference: Swiss Cyber Storm
YouTube: https://www.youtube.com/watch?v=uhMP8wXw0Tk
Overview
In an era increasingly plagued by sophisticated supply chain attacks, the ability to discern malicious intent within opaque software binaries has become paramount. Tim Blazytko, founder of mof, delivered a critical talk at Swiss Cyber Storm, addressing the pervasive challenge of trusting unknown executables. His presentation, "Unveiling Malicious Behavior in Unknown Binaries," provided a comprehensive guide for reverse engineers and security analysts, outlining both foundational and advanced strategies to identify hidden threats.
The talk underscored the inherent difficulty in assessing binaries compared to open-source code, where transparency allows for direct inspection. Blazytko articulated a methodical approach, starting with accessible static analysis techniques and progressing to sophisticated heuristics designed to accelerate and guide manual reverse engineering efforts. This presentation is vital for anyone involved in software supply chain security, incident response, or embedded device protection, offering actionable insights into safeguarding systems against concealed malicious functionalities.
Background
▶ Watch: Introduction: Analyzing unknown binaries for malicious behavior (0:00)
The landscape of modern computing is rife with unknown binaries. These could originate from third-party vendors, customer submissions, internet downloads, or commercial software purchases. The fundamental problem, as highlighted by Blazytko, is a lack of inherent trust: how can one be certain that an executable performs only its intended functions and does not harbor unexpected, detrimental capabilities? This challenge is amplified in scenarios like the "explain hack" mentioned by the conference host, a supply chain attack where malicious code was injected into widely distributed software components. While open-source components offer the theoretical advantage of source code review, binaries present a black box, making direct inspection impossible without specialized techniques.
Blazytko defined malicious content not necessarily as outright malware, but as any "additional features inside that you don't expect." This broad definition encompasses a spectrum of undesirable behaviors, from ransomware-like encryption of user files and exfiltration of data, to keylogging functionality or unauthorized network communication. The core issue revolves around discovering these hidden features within a binary – a task that requires a deep understanding of reverse engineering principles and the common patterns employed by threat actors to conceal their tracks. The talk aimed to equip analysts with the tools and methodologies to systematically unravel the complexities of unknown binaries and expose their true nature.
Key Findings
▶ Watch: Identifying malicious intent via suspicious API function calls (7:30)
The central contribution of Tim Blazytko's talk lies in its structured presentation of a tiered approach to binary analysis, combining accessible static methods with advanced, targeted heuristics. The key findings demonstrate that a strategic application of these techniques can significantly enhance an analyst's ability to uncover malicious behavior, even within highly obfuscated or packed executables.
Firstly, the talk established that several basic static analysis techniques provide crucial initial indicators. These include:
- Signature checks: Utilizing tools like YARA to match known malicious code patterns.
- Suspicious string analysis: Identifying human-readable text that hints at unintended functionality (e.g., keyloggers, network communication).
- API function import analysis: Examining the operating system functions a binary requests, which can reveal its operational intent (e.g., file manipulation, process injection).
- Entropy analysis: Quantifying the randomness of code sections to detect packing or encryption, a common malware tactic.
Secondly, Blazytko presented a suite of advanced heuristics designed to guide and accelerate manual reverse engineering when basic methods are insufficient. These heuristics target specific code constructs frequently employed by malware authors:
- State machine identification: Pinpointing complex, loop-based dispatching routines often used for network protocols or Command and Control (C2) communication.
- Frequently called functions analysis: Identifying functions invoked numerous times, which can reveal core functionalities or hidden import mechanisms.
- XOR decryption loop detection: Recognizing simple, insecure decryption patterns commonly used for obfuscating strings or unpacking code.
- RC4 cryptographic algorithm identification: Detecting the presence of the RC4 algorithm, a broken but frequently used encryption scheme in malware due to its ease of implementation.
Collectively, these findings underscore that a systematic combination of broad-spectrum static analysis and targeted, malware-specific heuristics provides a powerful framework for efficiently pinpointing and understanding malicious behavior in unknown binaries, transforming a daunting reverse engineering task into a more manageable, guided process.
Technical Deep Dive
▶ Watch: Live demo: Basic static analysis with Binary Ninja (12:00)
Blazytko meticulously detailed a progression of analysis techniques, starting with easily implementable static checks and advancing to sophisticated heuristics. The initial phase of analysis focuses on readily extractable information from the binary without execution.
Signature Checks form the first line of defense. These involve applying pre-defined rules, often expressed in formats like YARA, against a binary. These rules encapsulate patterns of known malware or specific malicious features. For instance, a YARA rule might detect the presence of screenshot functionality, specific cryptographic constants like SHA-1 or encoding schemes like CRC32, or indicators of code packing. While YARA rules provide quick insights, their effectiveness is limited by the completeness of the signature database and their susceptibility to obfuscation.
Next, Suspicious String Analysis involves extracting all human-readable strings from the binary. Malicious actors often inadvertently leave clues. Blazytko showcased an example of a Chinese malware sample containing strings like X-plug keylogger.cpp or screen.cpp, directly indicating keylogging or screen capturing capabilities. Other suspicious strings might include HTTP, POST, or partial URLs like /update, hinting at network communication or C2 server interaction. The absence of expected strings can also be a red flag, suggesting string obfuscation.
API Function Import Analysis scrutinizes the functions a binary requests from the operating system's libraries (e.g., user32.dll, kernel32.dll on Windows). Certain API calls are inherently more suspicious than others. Functions like GetKeyStates or GetAsyncKeyState are legitimate but are also frequently used in keylogging. Calls related to file system access, such as FindFirstFile, FindNextFile, WriteFile, or CreateFile, can indicate unauthorized data manipulation. However, the presence of WriteProcessMemory is a particularly strong indicator of malicious intent, as it signifies an attempt to inject code into another process, a common technique for privilege escalation or evasion.
Entropy Analysis provides a mathematical measure of randomness within data. A binary's entropy plot, which maps entropy values across its byte positions, can reveal packed or encrypted sections. A "normal" unpacked file typically exhibits entropy values between 5 and 6. In contrast, sections with very high entropy, approaching a value of 8, strongly suggest the presence of packed or encrypted code, a common technique used by malware to conceal its true functionality and evade signature-based detection.
When these basic methods yield insufficient information or require deeper investigation, manual reverse engineering becomes necessary. To guide this often arduous process, Blazytko introduced several heuristics:
- State Machine Identification: This heuristic targets common code constructs found in complex applications and, crucially, in malware. A state machine typically involves a large loop that dispatches execution to various "states" or basic blocks based on external input (e.g., network commands). Identifying such loop-based dispatching routines often pinpoints network protocol implementations or C2 communication logic within malware.
- Frequently Called Functions: The premise here is that functions called by a large number of other code locations are likely important. In the context of malware, this heuristic can reveal critical helper functions or even hidden import mechanisms. For instance, in an XorDos Trojan sample, frequently called functions included memory and string manipulation routines. In the PLX malware family, where imports were hidden, the most called functions were
CRC32andLsaLookupSids. These were used in a hash-based import hiding scheme, where the malware computed hashes of potential API functions and then loaded those matching the pre-computed hashes. Identifying these frequently called functions exposed a core evasion technique.
- XOR Decryption Loops: Malware frequently employs simple XOR operations with a fixed constant (e.g.,
0xDE) within loops to decrypt strings or unpack code sections. While not cryptographically secure, this method is easy to implement and serves to obfuscate static analysis. Detecting these specific loop patterns is a strong indicator of malicious activity, as legitimate applications rarely use such simplistic decryption schemes for core functionalities.
- RC4 Crypto Algorithm Identification: The RC4 algorithm is notoriously insecure and largely deprecated in modern, legitimate applications. However, its simplicity makes it exceptionally popular in malware, where it is often the most widely used cryptographic algorithm. Blazytko explained that RC4 consists of two main functions: the Key-scheduling algorithm (KSA) and the actual encryption/decryption routine. The KSA, in particular, is identifiable by two distinct loops that consistently operate with the constant
256(or0x100in hexadecimal). Identifying functions with this specific structural pattern (two loops, constant256) provides a high-confidence heuristic for RC4, which in malware is commonly used for string decryption or Domain Generation Algorithms (DGAs) to dynamically determine C2 server addresses.
These heuristics, when integrated into tools like Binary Ninja (a disassembler and decompiler framework), significantly streamline the reverse engineering process, allowing analysts to quickly focus on potentially malicious code segments rather than sifting through thousands of functions manually.
Demo / Proof of Concept
▶ Watch: Live demo: Plugin for RC4 implementation detection (23:45)
Tim Blazytko provided a concise yet effective demonstration of how these advanced heuristics can be practically applied using a custom plugin within the Binary Ninja analysis framework. The goal was to illustrate how an analyst, faced with a binary containing hundreds or thousands of functions, could quickly identify areas of interest.
The demonstration focused specifically on the RC4 crypto algorithm identification heuristic. Blazytko opened a binary in Binary Ninja and, without prior knowledge of its contents, invoked his custom plugin. The plugin was configured to "look for potential RC4 implementations." In the output window, the plugin promptly listed several functions that matched the heuristic's criteria.
Blazytko then navigated to one of the identified functions. He highlighted how the decompiler output clearly showed the tell-tale signs of an RC4 KSA implementation: two distinct loops and the repeated use of the constant 0x100 (which is 256 in decimal). He acknowledged that heuristics are not infallible, pointing out that in this specific run, one of the identified functions was a false positive. However, he emphasized that the value of the heuristic lies in its ability to narrow down the search space from potentially hundreds or thousands of functions to just a handful of candidates. This dramatically reduces the manual effort required, allowing the analyst to quickly confirm whether a suspected function is indeed RC4 and then delve deeper into its context and usage within the binary. The demo effectively showcased how these automated guides empower analysts to pinpoint malicious functionality with greater efficiency.
Defensive Implications
▶ Watch: Q&A: Strategies for evading binary detection (28:30)
The insights presented by Tim Blazytko offer critical guidance for security defenders operating across various domains, from enterprise security to embedded device protection. Understanding the techniques for unveiling malicious behavior in binaries is not just an academic exercise but a strategic imperative in the ongoing "arms race" against threat actors.
Firstly, defenders should integrate static analysis tools into their security pipelines. Tools like YARA, coupled with regularly updated rule sets, can provide immediate alerts for known malicious patterns or suspicious features. Automated string extraction and analysis can help identify unusual keywords, URLs, or file paths that warrant further investigation. Similarly, monitoring API function imports can flag binaries attempting highly suspicious actions, such as WriteProcessMemory, which should trigger high-priority alerts and deeper analysis.
Secondly, the talk highlights the importance of entropy analysis as a first-pass check for obfuscation or packing. Defenders should employ tools that can generate entropy plots for incoming binaries, immediately flagging executables with high-entropy sections as potentially malicious and requiring thorough unpacking and reverse engineering.
For security teams engaged in manual reverse engineering or threat hunting, Blazytko's heuristics provide invaluable guidance. Developing or adopting tools (like the Binary Ninja plugin demonstrated) that can identify state machines, frequently called functions, XOR decryption loops, and RC4 implementations can drastically reduce the time and expertise required to understand complex malware. This allows defenders to quickly home in on critical functionalities like C2 communication, data exfiltration, or anti-analysis techniques.
Furthermore, these insights inform proactive defense strategies. Software developers, particularly those working on embedded systems or critical infrastructure, should be educated on these common malware patterns. Avoiding insecure cryptographic algorithms like RC4, implementing robust string obfuscation techniques that don't rely on simple XOR loops, and carefully managing API calls can make legitimate software less susceptible to being mistaken for malware and more resilient against actual attacks. The overarching implication is that by understanding how attackers hide, defenders can better equip themselves to uncover and neutralize those hidden threats, turning the black box of a binary into a transparent window into its true intentions.
Key Takeaways
- Trust No Binary by Default: Binaries from unknown or untrusted sources should always be treated with suspicion, as they can harbor hidden, malicious functionalities.
- Layered Analysis is Key: Combine basic static analysis techniques (signatures, strings, API imports, entropy) for initial insights with advanced heuristics to guide deeper manual reverse engineering.
- Entropy Reveals Obfuscation: High entropy values in binary sections are strong indicators of packing or encryption, common malware evasion tactics.
- Heuristics Accelerate Deep Dives: Specialized heuristics, such as identifying state machines, frequently called functions, XOR decryption loops, and RC4 implementations, significantly streamline the process of uncovering malicious code patterns.
- RC4 is a Malware Red Flag: The presence of the RC4 cryptographic algorithm, especially its Key-scheduling algorithm (KSA) characterized by two loops and the constant
256(0x100), is a strong indicator of potential malware. - Empower Analysts with Tools: Tools and plugins that automate the detection of these heuristics, like the Binary Ninja plugin demonstrated, empower security analysts to efficiently pinpoint and investigate suspicious behavior.
About the Speaker(s)
Tim Blazytko is a seasoned expert in reverse engineering, with over 15 years of experience in the field. He is the founder of mof, a company dedicated to building embedded security solutions for embedded devices. In addition to his entrepreneurial role, Blazytko is a recognized trainer, specializing in code obfuscation and advanced reverse engineering techniques. His extensive background makes him a leading authority on analyzing and securing complex software binaries.
Reviews
Dr. Zero (Offensive Security Researcher) — SOLID
Blazytko knows reverse engineering better than almost anyone, which makes this basic RE 101 presentation a bit of a letdown. If you've never analyzed malware, the RC4 and IAT hashing heuristics will save you hours, but there is zero novel research here.
Heather Calloway (CISO) — SOLID
This is a strictly operational talk for your reverse engineers, not your leadership team. It won't change your vendor risk policies, but it outlines the exact automated tradecraft your incident response team needs to adopt to speed up triage during a supply chain crisis.