Analyzing And Reverse Engineering Antivirus Signatures

Cracked Shield: Analyzing and Reverse Engineering Antivirus Signatures

Speakers: Dobin Rutishauser, Red Team Lead, Raiffeisen Schweiz

Conference: HITB Phuket

YouTube: https://www.youtube.com/watch?v=Sxw8K6gFSbg

Overview

In "Cracked Shield: Analyzing and Reverse Engineering Antivirus Signatures," Dobin Rutishauser, a Red Team Lead at Raiffeisen Schweiz, delves into the often-overlooked mechanics of how antivirus (AV) software detects malicious files through static signatures. The talk, originally conceived with the provocative title "Our signatures are bad and we should feel bad," explores the fundamental question of whether modern AV signatures are as robust as they appear, especially in the face of sophisticated evasion techniques employed by red teamers and malware authors. Rutishauser introduces AV rat, a custom-built tool designed to identify the exact byte sequences within a file that trigger AV detection, effectively reverse engineering the signature itself.

The core focus of this presentation is on the "antivirus" component of AV software – specifically, the static scanning of files for known malicious patterns, distinct from more advanced capabilities like Endpoint Detection and Response (EDR), heuristics, or sandbox execution. Rutishauser highlights the constant cat-and-mouse game between red teamers seeking to deploy tools like Cobalt Strike, Mimikatz, or Sliver, and blue teams striving to block them. By dissecting how AV signatures are structured and triggered, the talk provides invaluable insights for both offensive and defensive security practitioners, offering a deeper understanding of AV limitations and potential bypasses. The publicly available tool, AV rat, allows anyone to replicate the research and test their own files at avred.root.ch.

Background

▶ Watch: Introduction to talk and AV signature problem (0:00)

The landscape of cybersecurity is perpetually shaped by the escalating arms race between attackers and defenders. For red teamers, the primary objective often involves deploying custom tools or well-known offensive frameworks to achieve their objectives. However, modern antivirus solutions are designed to thwart these efforts, frequently detecting and quarantining such tools upon execution or even before they hit the disk. This presents a significant challenge for red teamers, as a detected payload not only halts their progress but also alerts the Security Operations Center (SOC), potentially compromising the entire operation.

Traditional evasion techniques have included recompiling tools from source to alter their binaries, using obfuscators to scramble code and data, or employing packers like UPX (though easily identified). More advanced methods involve loaders, which encrypt a malicious payload (shellcode or executable) and then use a separate loader component to decrypt and execute it in memory. The sophistication of these loaders has grown exponentially, incorporating advanced anti-EDR techniques such as process hollowing, API unhooking via indirect system calls, Return-Oriented Programming (ROP), faking backtraces on spawned threads, strategic sleeping to evade timing-based detections, and process herpaderping. The trend of DLL side-loading, where malicious DLLs are dropped onto disk and loaded by legitimate applications, also gains traction, but still leaves the payload vulnerable to static AV scanning while on disk.

Rutishauser traces the evolution of antivirus technology back to the "floppy disk era" of the 1980s and 90s, when viruses spread by copying themselves to executables on MS-DOS systems. He draws an analogy between biological viruses (strands of DNA that hijack cells) and computer viruses. Just as vaccines expose the immune system to parts of a virus, AV signatures expose the scanner to known malicious byte patterns. Early viruses evolved polymorphism (changing code while retaining functionality, e.g., ADD 1 vs. ADD 100 SUB 99) and later metamorphism (altering the core virus code itself) to evade simple signatures. AVs responded with code emulators and heuristics to execute and analyze code dynamically.

A key motivation for AV rat stemmed from a talk by Vladimir Mauer at Insomni'hack, who presented AV Debugger (a Python-based, PE section-aware version of Threat Check) and AV Cleaner. Mauer's research suggested that antivirus solutions primarily focused on data strings within executables, not the code itself. This contradicted Rutishauser's understanding of older AV mechanisms, prompting him to investigate whether this assumption held true for modern AV engines. The primary goals for AV rat were twofold: first, to precisely identify which parts of a file trigger AV detection, and second, to simplify the process of modifying these detected parts to achieve undetectable payloads.

Key Findings

▶ Watch: Live demo: Bypassing AV by changing a string in SharpUp (15:00)

Rutishauser's research with AV rat yielded several critical insights into the nature and effectiveness of modern antivirus signatures:

1. Challenging the Data-Only Assumption: While Vladimir Mauer's prior work suggested AVs primarily scan data sections, AV rat's findings indicate a more nuanced reality. For PE files (native executables like those compiled from C/C++), approximately 50% of detected matches were found in code sections, and 50% in data sections. However, for .NET files, the vast majority of detections were indeed linked to data sections (strings, metadata) rather than the Common Intermediate Language (CIL) bytecode itself. This suggests that while encrypting strings can be effective for .NET, native binaries require consideration of code modifications as well.

1. Variability in Signature Robustness: AV rat categorizes signatures into three types based on the complexity required for bypass:

• Weak (Green): Characterized by a "dominant match," where altering a single, specific byte sequence is sufficient to evade detection. These are relatively easy to bypass.
• Robust (Gray): Requires modifying a combination of several matches. Bypassing these demands more effort and analysis to determine the critical combinations.
• Very Robust (Red): No single dominant match; detection relies on a complex interplay of many matches. Tools like Cobalt Strike and Mimikatz frequently fall into this category, indicating significant investment from AV vendors in developing strong, resilient signatures for high-profile threats.

1. Impact of Cloud-Delivered Protection: A significant finding was the behavior of Windows Defender's cloud-delivered protection. While AV rat could successfully bypass static signatures when this feature was disabled or when accessing Defender via the AMSI interface, enabling cloud protection (especially when downloading files via Chrome or Edge) resulted in immediate re-detection, even for files modified to bypass local signatures. This highlights a multi-layered defense strategy where dynamic, cloud-based analysis supplements static signatures, rendering local bypasses less effective. Interestingly, using Firefox to download the same modified files often bypassed this cloud protection, suggesting potential differences in integration or scanning triggers.

1. Signature Quality and Source: The quality and robustness of signatures vary widely, even within the same AV product. Rutishauser speculates this could be due to a mix of automated signature generation, manually crafted rules, legacy signatures, and potentially AI-driven detection. This variance means that while some signatures are trivial to bypass, others are highly resilient.

1. Headers Are Often Ignored: Both PE file headers and .NET headers generally do not contribute to AV detection. Similarly, detailed parsing of complex file formats like OLE2 (used in Word documents for macros) appears unnecessary for signature detection; AVs likely scan the raw macro bytes directly, rather than reconstructing the fragmented OLE2 file system. This simplifies the attacker's job of identifying relevant detection points.

1. Challenges in Reverse Engineering: The process of reliably identifying signatures is far from trivial. Challenges include:

• Multiple Matches: A file might be detected by several independent signatures.
• Hash-Based Detection: Sometimes, files or even sections appear to be detected by their cryptographic hash, making byte-level modification ineffective without recompilation.
• Long Scan Times: AV scans can take minutes or even hours, significantly slowing down the iterative process of signature analysis.
• Incomplete Reduction: The algorithm occasionally finishes, but the file remains detected even after all identified "matches" are overwritten, suggesting unknown or complex detection mechanisms.

These findings underscore the evolving nature of AV defenses and the continuous need for both offensive and defensive teams to understand the underlying mechanisms of detection and evasion.

Technical Deep Dive

▶ Watch: Augmenting matches with PE/ .NET disassembly for context (37:00)

AV rat operates as a sophisticated framework for reverse engineering antivirus signatures, built around an iterative process of file reduction, AV querying, and detailed match analysis.

Antivirus Oracle via AMSI

The foundational component of AV rat is its Antivirus Oracle. To reliably query an antivirus product, AV rat leverages the Antimalware Scan Interface (AMSI) in Windows. AMSI is a generic interface that allows any application (e.g., PowerShell, JScript, or custom code) to submit a content stream to the installed AV product for scanning. This is crucial because it abstracts away the specifics of different AV vendors' command-line tools or APIs.

Rutishauser implemented an AV rat server on a Windows machine. This server exposes an HTTP interface, allowing a Linux analysis workstation (or any other system) to send file blobs and receive detection verdicts. A key advantage is the ability to install various AV products on different Windows machines, each running an AV rat server, and then query them all from a central analysis system. This setup prevents potential conflicts or performance issues that might arise from installing multiple AVs on a single machine. The AMSI interface provides a consistent and reliable way to ask: "Is this blob malicious or not?"

The Reducer Algorithm

With the AV Oracle in place, the next challenge is to pinpoint the exact byte sequences responsible for detection. AV rat employs a divide and conquer algorithm for this purpose:

1. Initial State: Start with a known malicious file.
2. Divide: Override approximately half of the file's content (e.g., with null bytes or random data).
3. Conquer: Query the AV Oracle.

• If the file is still detected, the malicious signature must reside in the unmodified half.
• If the file is no longer detected, the malicious signature was in the overwritten half.

1. Iterate: Repeat the process on the identified malicious half, progressively narrowing down the region until a sufficiently small "match" (an offset and size) is found.

While conceptually simple, Rutishauser emphasizes the complexity of making this algorithm robust in practice. Challenges include:

• Multiple Matches: If a file contains multiple independent signatures, the simple divide-and-conquer might only find one. The algorithm needs to handle scenarios where overwriting one part makes the file undetected, but another part still triggers a separate signature.
• Hash-Based Detections: Some AVs might detect files based on their cryptographic hash. If a file is detected by hash, no amount of byte modification will bypass it (short of recompilation), making the reducer appear to fail or run indefinitely.
• Performance: Each query to the AV Oracle can take seconds, minutes, or even hours. An efficient implementation is critical to complete the reduction process in a reasonable timeframe.
• PE Section Awareness: The reducer initially attempts to focus on PE sections (code, data) rather than headers, assuming headers are less relevant for signatures. However, if matches span multiple sections or cannot be isolated, it falls back to reducing the complete file (excluding headers).

The Verifier

The reducer's output is a list of potential "matches." The verifier component ensures the accuracy and relevance of these matches. It performs several tests on each identified match:

1. Dominant Match Test (Test #2): For each match, the verifier creates a copy of the original file, overwrites only that specific match with zeros, and then queries the AV Oracle.

• If the file is no longer detected, that match is considered a dominant match, indicating a weak signature that can be bypassed by altering just that segment. (Green conclusion)
• If the file is still detected, that match alone is not sufficient for detection.

1. Incremental Override Test: For files with multiple matches, the verifier might test combinations. For instance, it could incrementally overwrite matches (e.g., match 7, then matches 6+7, then matches 5+6+7) to identify if a specific combination of changes is required for bypass. This helps categorize signatures as "robust" (gray) or "very robust" (red).

The verifier aggregates these individual match conclusions into an overall "signature conclusion" for the file, indicating whether the signature is weak, robust, or very robust.

YARA Rule Analogies

Rutishauser draws parallels between AV signatures and YARA rules, which are widely used for malware identification. YARA rules define patterns (strings, regular expressions, byte sequences) and conditions (AND/OR logic, counts, wildcards) that must be met for a file to be identified.

• Weak Signatures: Analogous to YARA rules using all of them for multiple strings. Changing just one string breaks the entire rule.
• Strong Signatures: Similar to YARA rules using any of them or some of them (e.g., 2 of 3 strings). These require multiple modifications to bypass.
• Wildcards (XX): YARA rules can use wildcards (e.g., 8B FF XX 33 C0 where XX is any byte). This allows for generic matching (e.g., any register being XORed with itself). However, excessive wildcards can make signatures less precise or lead to individual detection of parts rather than a cohesive pattern.

The talk highlights issues with signature generation tools, referencing YARASignator's potential misidentification of 64-bit code as 32-bit, leading to ineffective or nonsensical YARA rules.

Augmentation: Making Matches Meaningful

Raw byte offsets are not useful for a red teamer. AV rat's augmentation phase enriches the identified matches with contextual information:

1. PE Files (Native Code):

• Challenge: PE files use virtual addresses (VAs) in memory, which differ from file offsets on disk. Disassemblers typically work with VAs.
• Solution: AV rat translates file offsets to VAs using PE header information (section virtual addresses and raw data offsets).
• Tooling: Integrates with Radare2 (via Python API) to disassemble the matched byte sequences. This provides human-readable assembly instructions, allowing red teamers to understand what code is being detected (e.g., a function prologue, specific API call, or unique instruction sequence).

1. .NET Files (Common Intermediate Language):

• Challenge: .NET executables contain Common Intermediate Language (CIL) bytecode, not native CPU instructions. Microsoft's choice to embed all .NET metadata within the PE's .text section complicates parsing.
• Tooling: Initially used IL Spy for understanding, then leveraged Python libraries like enfile and IL-Seal for robust parsing of .NET structures (headers, methods, data streams). These libraries handle the complex mapping between file offsets and virtual addresses within .NET metadata.
• Augmentation: Provides details about detected matches, such as the method header (e.g., MaxStack size), the CIL instructions, or specific strings within user string, metadata, or binary blob streams.

1. Word Documents (VBA Macros):

• Challenge: Word documents with macros (.docm) are essentially ZIP files containing XML and a VBA project.bin file. This .bin file is an OLE2 compound document, which is a complex, file system-like structure with fragmented chunks (512 bytes) and mini chunks (64 bytes). Correlating offsets in the OLE2 structure to actual file offsets is extremely difficult.
• Tooling: Used a patched version of OLEVBA (to obtain P-code disassembly with offsets) and olefile for parsing the OLE2 structure.
• Finding: Despite the complexity, Rutishauser found that AVs likely scan the raw macro bytes directly, rather than fully parsing the OLE2 structure. This implies that the painstaking work of OLE2 defragmentation for signature analysis was largely unnecessary.

By augmenting raw matches with this semantic information, AV rat transforms meaningless byte offsets into actionable intelligence for modifying files.

Demo / Proof of Concept

▶ Watch: Exploring automatic signature modification techniques (Outflank) (51:00)

Rutishauser presented several live demonstrations of AV rat's capabilities, illustrating both its effectiveness in reverse engineering signatures and the challenges posed by advanced AV features.

SharpUp: Simple String Modification

1. Detection: A red teaming tool called SharpUp.exe was uploaded to AV rat and immediately detected by the antivirus.
2. Match Identification: AV rat performed its analysis and identified several matches. One prominent match included the string "Decrypt GPP password.c password" (and "C password").
3. Modification: Rutishauser demonstrated modifying the source code of SharpUp, changing "GPP password.c password" to "GPP test.c test" through a simple refactoring.
4. Bypass: After recompiling and re-uploading the modified SharpUp binary, AV rat reported that the file was no longer detected by the antivirus. This demonstrated a successful and straightforward bypass by altering a single, dominant string-based signature.

Seatbelt: The Cloud Protection Challenge

1. Initial Bypass: The Seatbelt.exe tool was shown to be detected. Rutishauser then downloaded a version of Seatbelt.exe where AV rat had automatically overwritten the first dominant match with zeros. With Windows Defender's cloud-delivered protection disabled, this modified file was no longer detected, confirming the local signature bypass.
2. Cloud Re-detection: When cloud-delivered protection was enabled in Windows Defender (and the file downloaded via Chrome or Edge), the same modified file was immediately re-detected. This highlighted that Defender's cloud capabilities add a layer of detection beyond static, local signatures, capable of catching files even after their initial static signatures have been broken.
3. Firefox Anomaly: Interestingly, downloading the same cloud-bypassed file using Firefox resulted in it not being detected. This suggests that Defender's cloud-delivered protection might be more tightly integrated with specific browsers (Chrome/Edge) or activated under certain conditions (e.g., Mark-of-the-Web for specific download sources) that Firefox did not trigger in this context, or that the AMSI interface itself does not trigger the full cloud scan.

Outflank: Automated Signature Breaking (Limited Success)

Rutishauser also explored "outflanking" – attempts to automatically modify files to break signatures without manual intervention. While conceptually appealing, this approach proved to be less reliably successful:

• NOP/INT3 Swapping: Changing NOP (no operation) instructions to INT3 (breakpoint) and vice versa in assembly code sometimes worked but was not consistently effective, possibly because AVs might wildcard common NOP sequences.
• Assembly Line Swapping: Swapping the order of independent assembly instructions (those not touching the same registers) sometimes broke signatures. An example with sysminidump.exe (part of Seatbelt) showed success, where swapping two lines in a dominant match rendered the file undetectable while ideally preserving functionality.
• .NET Max Stack Size Modification: For .NET files, modifying the MaxStack value in a method header (e.g., increasing it by a small amount) sometimes broke the signature for specific files (e.g., 222.x).

These automated techniques demonstrated potential but were not universally applicable or reliable. This indicates that while AV rat can identify the problem, automatically fixing it in a robust, functional way often requires more sophisticated, context-aware code transformation (e.g., using LLVM for native code or source-level obfuscators for .NET) that goes beyond simple byte manipulation.

Defensive Implications

▶ Watch: Q&A: Using PDB symbols for more precise signature analysis (65:00)

Rutishauser's research provides several critical insights for blue teams and security defenders:

1. Signature Quality Varies Wildly: Defenders should not assume all AV signatures are equally robust. While high-profile red teaming tools like Cobalt Strike and Mimikatz often have very strong, multi-faceted signatures, other less common tools or bespoke malware might be detected by weak, easily bypassable patterns. This variability underscores the need for a multi-layered defense strategy beyond sole reliance on static signatures.

1. Cloud-Delivered Protection is Paramount: Windows Defender's cloud-delivered protection offers a significant uplift in detection capabilities, often catching modified files that bypass local static signatures. Defenders must ensure this feature is enabled, fully functional, and integrated across their endpoints. Relying solely on local scans, or testing via interfaces like AMSI that might not trigger the full cloud analysis, can lead to a false sense of security. The discrepancy in detection when downloading via Chrome/Edge versus Firefox also suggests that browser choice or Mark-of-the-Web attributes can influence whether advanced cloud scanning is triggered.

1. Beyond Static Signatures: The inherent vulnerability of static signatures to modification reinforces the necessity of Endpoint Detection and Response (EDR), behavioral analysis, and sandbox execution. These dynamic detection methods are crucial for catching polymorphic malware, obfuscated payloads, and advanced loaders that successfully evade signature-based detection.

1. Understand Attacker Methods: Blue teams need to understand the techniques red teamers and malware authors use to bypass AV, including recompilation, obfuscation, and various loader techniques. This knowledge helps in anticipating threats and designing more resilient defenses. Tools like AV rat can be invaluable for internal testing, allowing blue teams to assess how their current AV stack performs against modified versions of known malicious tools or custom payloads.

1. Strategic Signature Creation: For organizations that develop their own internal signatures (e.g., via YARA rules), Rutishauser advises a thoughtful approach. For short-term, campaign-specific threats (e.g., a macro-laden Word document used for a few days), a simpler, less robust signature might suffice. However, for persistent threats or common red teaming tools, significant investment should be made in creating robust, polymorphic-aware, and multi-faceted signatures that are difficult to bypass through minor modifications. This involves considering wildcards carefully and ensuring signatures don't rely on easily changed components.

1. AMSI Limitations in Testing: While AMSI is an excellent interface for querying AV, the talk demonstrates that it might not always trigger the full extent of an AV's detection capabilities, particularly advanced cloud-based scanning. Blue teams should be aware of this when designing their testing methodologies and consider real-world execution scenarios rather than solely relying on AMSI queries.

1. Future of Malware Development: The research hints at a future where malware development could integrate tools similar to AV rat into CI/CD pipelines. Attackers could automatically mutate their code, test against various AVs/EDRs, and iterate until the payload is undetected, significantly accelerating the evasion process. Defenders need to prepare for this by evolving their detection mechanisms to be more resilient to automated mutation.

Key Takeaways

• AV signatures are byte patterns, and their quality varies widely. Some are easily bypassed by altering a single "dominant match," while others (e.g., for Cobalt Strike) are highly robust and require complex modifications.
• AV rat can reliably identify detected byte sequences (matches) within malicious files using an AMSI-backed oracle and a refined divide-and-conquer algorithm.
• Windows Defender's cloud-delivered protection significantly enhances detection, often catching modified files that bypass local static signatures or AMSI queries. This highlights the importance of multi-layered, dynamic AV capabilities.
• For PE files, signatures are roughly 50/50 split between code and data sections, challenging the notion that AVs only scan data. For .NET files, detections are predominantly in data sections.
• Automated signature breaking (outflank) is challenging and not consistently reliable. While techniques like line swapping or modifying .NET method headers can sometimes work, robust evasion often requires more sophisticated, context-aware code transformation.
• Headers (PE, .NET) are largely ignored by AV signatures, and complex parsing of formats like OLE2 (for VBA macros) is often unnecessary, as AVs likely scan raw byte streams.
• Defenders should prioritize EDR, behavioral analysis, and ensure cloud protection is active, as static signatures alone are insufficient against evolving threats and targeted evasion.

About the Speaker(s)

Dobin Rutishauser is a seasoned cybersecurity professional currently serving as a Red Team Lead at Raiffeisen Schweiz, which he notes is the second largest bank in Switzerland. Prior to his role in red teaming, Dobin gained experience as a penetration tester. He is an active contributor to the security community, having delivered several talks at conferences and taught at universities, sharing his expertise and research with a broader audience. His work, such as the AV rat project, demonstrates his practical approach to understanding and addressing real-world security challenges.