All You Always Wanted To Know About AntiViruses

All You Always Wanted To Know About AntiViruses

Speakers: Marcus Botacin, Computer Science Professor, Texas A&M University

Conference: HITB Amsterdam

YouTube: https://www.youtube.com/watch?v=fnexx1Ek168

Overview

In his HITB Amsterdam talk, "All You Always Wanted To Know About AntiViruses," Marcus Botacin, a Computer Science Professor at Texas A&M University, offered a comprehensive and demystifying look into the intricate internal workings of modern antivirus (AV) solutions. Drawing from extensive research detailed in a 70-page paper, Botacin's presentation aimed to foster a more accurate understanding of AV capabilities and limitations, moving beyond superficial criticisms to enable constructive dialogue and innovation. The core premise is that informed critique, rooted in a deep technical understanding, is essential for advancing the field of cybersecurity and proposing more robust defensive mechanisms.

The talk systematically deconstructed common misconceptions, such as confusing AV products with their underlying engines and oversimplifying detection mechanisms. Botacin meticulously explored various facets of AV operation, from static signature-based detection and dynamic analysis via hooking to kernel-level monitoring, network interception, and even the often-overlooked aspects of quarantine, caching, and update processes. He highlighted critical trade-offs made by AV vendors regarding performance, detection scope, and resource allocation, emphasizing that many perceived "failures" are, in fact, deliberate design decisions or inherent limitations within their operational threat models.

Ultimately, Botacin's presentation served as a crucial call to action for both researchers and practitioners. By exposing the nuanced realities of AV technology, he provided a foundation for developing fairer evaluation methodologies, designing more resilient security solutions, and fostering transparency within the cybersecurity industry. His work underscores the necessity of moving beyond simplistic benchmarks to embrace a multi-dimensional understanding of how AVs function in complex, real-world environments.

Background

▶ Watch: Introduction and AV engine vs. product distinction (0:00)

The landscape of cybersecurity is often fraught with conflicting narratives, particularly concerning the efficacy of antivirus software. Marcus Botacin observed a pervasive trend of criticism against AVs, much of which he agrees with, but also noted that a significant portion lacks accuracy due to a fundamental misunderstanding of how these complex systems operate. This knowledge gap hinders progress, as uninformed criticism fails to pinpoint genuine areas for improvement. The motivation behind his research and subsequent talk was to bridge this gap, providing a detailed, internals-focused perspective on AVs to enable more informed and constructive discourse.

A primary misconception Botacin addressed is the conflation of antivirus engines with antivirus products. Many commercial AV products license detection engines from a smaller pool of providers. This leads to biased evaluations where simply counting the number of AV products detecting a sample can inflate detection rates by counting the same underlying engine multiple times. Botacin's research, analyzing millions of samples submitted to VirusTotal over years, demonstrated how AV products cluster around specific engines, with some even migrating between engine providers over time (e.g., Avast and AVG merging their engines). This foundational understanding is critical for accurate benchmarking and fair comparisons.

Furthermore, AVs are not monolithic applications but rather sophisticated systems comprising multiple components: graphical user interfaces (GUIs), backend processes, kernel drivers, and libraries. Crashes in one component, such as the GUI, do not necessarily imply a failure of the entire security system, as backend processes and kernel drivers often continue to operate. This multi-component architecture necessitates a holistic understanding to provide meaningful critique. The talk aimed to lay bare these internal structures and operational principles, providing the essential context for why certain "problems" exist or why specific design choices are made, thereby setting the stage for a more mature evaluation of current and future AV technologies.

Key Findings

▶ Watch: Signature prevalence and evasion opportunities (6:00)

Botacin's research unveiled several critical insights into the internal mechanisms and operational philosophies of antivirus software, challenging conventional wisdom and highlighting areas for improved understanding and development.

First, whitelists are a pervasive and often default detection mechanism, not just user-configured exceptions. AVs ship with extensive internal whitelists for legitimate software (e.g., OneDrive processes), often checking these lists before initiating any complex scanning. This design choice prioritizes avoiding false positives on known good software, but it can skew evaluations if compared against models that lack such pre-configured knowledge.

Second, while often debated, signatures remain a significant component of AV detection. Through a novel binary search algorithm, Botacin's team found that 30% to 40% of all detections across various AVs are still performed using simple signatures. These signatures range from a few kilobytes to less than 1 MB, sometimes even revealing shared threat intelligence across different AV products. This finding underscores the continued relevance of traditional evasion techniques like polymorphism, but also points to the growing reliance on more advanced methods for the majority of detections.

Third, AVs exhibit diverse and often strategic support for packers. Despite the security community's general knowledge of unpacking techniques (e.g., UPX), many AVs deliberately choose not to support certain packers or only support their standard versions. This is not due to a lack of technical capability but rather a project decision influenced by perceived threat models or performance trade-offs, leading to varied detection rates for packed malware.

Fourth, dynamic analysis involves significant trade-offs. AVs employ custom hooking mechanisms to collect data from user-land functions and kernel-land activities via drivers and callbacks. The number of hooked functions varies greatly between AVs, reflecting a balance between comprehensive data collection (more hooks) and performance overhead (fewer hooks). This trade-off often leads to unfair comparisons when evaluating new, heavily instrumented detection models against existing AVs with different performance constraints.

Fifth, network interception often involves man-in-the-middle (MITM) techniques. Products like Kaspersky's AVP process redirect all network connections, installing their own certificates to inspect encrypted HTTPS traffic. While users consent to this, the underlying technical approach is identical to a malicious MITM attack, highlighting the deep level of system integration AVs achieve.

Sixth, quarantined files are typically encoded, not encrypted. Botacin demonstrated that files moved to quarantine are often merely encoded (e.g., byte-mapped) to prevent execution, retaining their original size and structure. This allows for straightforward decoding, potentially enabling access to fresh malware samples for analysis without decryption keys.

Seventh, AV caching mechanisms can be exploited for evasion. By strategically introducing components of a multi-stage malware (e.g., a loader followed by a payload), an attacker can trick an AV's emulator into caching a "clean" verdict for the loader when the payload is absent. Once the payload is introduced, the cached verdict allows the loader to execute undetected.

Finally, AV update mechanisms can be vulnerable and deliver entire software components. Some AVs, like Avast, download definition updates over unencrypted HTTP, making them susceptible to MITM attacks. While these updates are signed, their cleartext nature allows for analysis of their structure, revealing that AVs often update not just signatures but also their own DLLs and other software components as part of the update process. This underscores the critical importance of update integrity and the potential for supply chain attacks if these mechanisms are compromised.

Technical Deep Dive

▶ Watch: Exploiting AV caching for evasion (18:00)

Antivirus software, as revealed by Marcus Botacin, is a complex ecosystem operating at multiple layers of the system, employing a diverse array of techniques that go far beyond simple signature matching. A key distinction emphasized is between antivirus products and antivirus engines. Many products, visible to the end-user, license detection capabilities from a smaller pool of core engines. This was empirically demonstrated by clustering millions of VirusTotal labels, showing how products group around engines and even migrate between them over time, such as the observed merging of Avast and AVG engines post-acquisition.

The architecture of an AV is multi-component, encompassing a Graphical User Interface (GUI), backend processes, kernel drivers, and various libraries. A common misconception is that a GUI crash signifies total AV failure, when in reality, the critical backend processes and kernel drivers often continue to operate, maintaining core protection.

Detection mechanisms begin with whitelisting, a surprisingly prevalent first line of defense. AVs often ship with pre-configured whitelists for known legitimate files and processes (e.g., OneDrive.exe). During a scan, the whitelisting function is typically invoked first; if a file matches an entry, it bypasses further, more resource-intensive analysis. This strategy minimizes false positives but can lead to unfair comparisons when evaluating novel detection models that lack such inherent "known good" data.

Signature-based detection, despite claims of its obsolescence, still accounts for a significant portion of AV detections, ranging from 30% to 40% according to Botacin's research. To quantify this, his team developed a binary search algorithm: for a detected file, they iteratively patched halves of the file and resubmitted them, narrowing down to the minimal byte pattern (the signature) responsible for the detection. This method identified signatures ranging from a few kilobytes to nearly 1 MB, with the majority being in the kilobyte range. Analysis also revealed instances of different AVs sharing identical signatures, indicating shared threat intelligence.

The handling of packers, such as UPX, illustrates a critical aspect of AV design: conscious trade-offs. While unpacking UPX is a well-understood technique in the security community, AVs vary widely in their support. Experiments showed some AVs detecting only original samples, others detecting standard UPX-packed samples, and only a few capable of detecting samples packed with custom-compiled UPX versions. This variability stems from project decisions balancing performance, resource allocation, and perceived threat prevalence, rather than a universal inability to unpack.

Dynamic analysis is performed through hooking mechanisms, typically custom-developed by each AV vendor. These hooks intercept function calls at user-land to collect data for classification. The number of hooked functions varies significantly across AVs, reflecting a trade-off: more hooks provide richer telemetry but incur greater performance overhead. This difference in data collection scope makes direct comparison between a research model with extensive hooking and a production AV with performance constraints inherently unfair.

Beyond user-land, AVs operate extensively in kernel-land through multiple drivers. These drivers leverage callbacks to monitor processes, registry access, file system operations, and network activity. While contributing to detection, a primary use case for these kernel components is self-protection, preventing tampering with AV files and directories (e.g., denying access to the AV installation path). Identifying these protected locations can serve as a reverse engineering hint for attackers, revealing critical AV components.

Network inspection involves deep packet analysis, sometimes utilizing a man-in-the-middle (MITM) strategy. Kaspersky's AVP (Antivirus Proxy) process, for instance, redirects all user connections. To inspect HTTPS traffic, AVs install their own root certificates in the browser's trust store, effectively decrypting and re-encrypting secure connections. While consented to by the user, the underlying technical methodology is identical to a malicious MITM. Some AVs, like Viper, even integrate subsets of established intrusion detection systems, such as Snort rules, for network-based threat identification.

Files flagged as malicious are moved to quarantine. Botacin demonstrated that these files are typically encoded, not encrypted. By mapping original bytes to different values, the AV prevents execution while retaining the original file size and structure. This encoding is often reversible, allowing researchers to extract pristine malware samples for analysis without needing decryption keys, a process some AVs unfortunately charge for.

AVs extensively use databases for telemetry and caching. Telemetry databases store vast amounts of system information, including scanned files, timestamps, and installed software, which can be leveraged for internal monitoring or even reverse engineered by skilled practitioners for their own system insights. Caching is critical for performance, preventing redundant scans of unchanged files or frequently accessed URLs. However, this caching mechanism can be exploited. Botacin illustrated a scenario where a multi-stage malware (a loader and a payload) could evade detection: the loader is scanned without its payload, cached as "clean," and then, when the payload is later introduced, the cached "clean" verdict allows the combined malware to execute successfully.

Finally, AV update mechanisms present both critical functionality and potential vulnerabilities. Tracking real AV systems showed frequent updates, sometimes multiple times a day. Disturbingly, some AVs, like Avast, download updates over plain HTTP without encryption. While these updates are digitally signed to ensure integrity, their cleartext transmission allows for man-in-the-middle observation. Analysis of these update files revealed a consistent structure: a header, the actual payload (often a DLL), and a digital signature. This indicates that AVs update not just signature definitions but also their own software components, making the update channel a critical attack vector for supply chain compromise. The integrity of these updates is paramount, as corrupted or tampered AV files can lead to crashes or complete cessation of protection without user notification.

Botacin also highlighted crucial, often unstated, threat model assumptions. AVs generally assume a pristine installation on a clean system, expecting to prevent future infections. This assumption breaks down when users install AVs on already infected machines, leading to race conditions where existing malware can interfere with or prevent AV installation, rendering it ineffective. This was demonstrated by a simple malware creating an infinite loop to delete AV installation files, successfully crashing all tested AVs during installation. The talk also touched upon the inherent limitations of Android AVs, which are significantly simpler than their desktop counterparts. Lacking kernel drivers and relying heavily on external reputation databases and abusing services like Accessibility, they offer limited on-device verification and are prone to large whitelists, often equating popularity with safety—a dangerous trade-off.

Demo / Proof of Concept

▶ Watch: The 'pristine installation' assumption (22:00)

Throughout his talk, Marcus Botacin illustrated complex technical concepts with practical demonstrations and experiments, serving as effective proofs of concept for his key findings.

1. Signature Detection of Text Files: To prove that AVs still rely on signatures, Botacin demonstrated creating a simple text file containing a known shellcode string (an Indicator of Compromise or IOC). When this non-executable file was copied to a system with an AV and forced to scan, the AV detected it as malicious. This clearly showed that the detection was based purely on the file's content signature, not dynamic analysis or emulation, as it wasn't a PE file.

1. Binary Search Algorithm for Signature Extraction: Botacin presented a conceptual demo of his team's algorithm for identifying the exact byte patterns used as signatures. For a file detected by an AV, the algorithm iteratively patches halves of the file and resubmits them for scanning. By repeatedly splitting the detected portion, it quickly converges on the minimal set of bytes that trigger the AV detection, revealing the precise signature. This was visually represented with diagrams showing the splitting process.

1. UPX Packer Detection Experiment: To highlight the variability in packer support, Botacin detailed an experiment involving original malware samples, samples packed with a standard UPX downloaded from the internet, and samples packed with a custom-compiled UPX version. The results, presented graphically, showed that some AVs only detected original samples, others detected standard UPX, and only a few detected the custom UPX, demonstrating that lack of detection isn't always an inability but a deliberate design choice.

1. Cache Exploitation with Loader/Payload: A compelling demonstration involved a two-stage malware: a loader and an encrypted payload. Initially, the loader was placed on the system without the payload. The AV scanned it, found no malicious activity (as the payload couldn't be decrypted), and cached a "clean" verdict. Subsequently, the encrypted payload was introduced. When the loader executed, it successfully decrypted and ran the payload, which then displayed a message box, proving that the AV's cached "clean" state allowed the final malicious execution to proceed undetected.

1. Avast Update Man-in-the-Middle: Botacin showed evidence of Avast downloading its virus definitions via plain HTTP. He demonstrated how this allowed an attacker to intercept the update traffic. By parsing the intercepted cleartext update file, he revealed its structure: a header, the actual payload (which was identified as a DLL), and a digital signature. This proved that AVs update their core software components (like DLLs) through potentially vulnerable channels, and that the update format is discoverable, despite the integrity checks.

1. Malware vs. AV Installer Race Condition: To illustrate the "pristine installation" assumption, Botacin created a simple malware that entered an infinite loop, continuously attempting to remove files from known AV installation directories. He then ran an AV installer on this system. The result was a race condition where the malware successfully interfered with the AV installation, preventing it from completing and effectively crashing the installation process for all tested AVs.

These demonstrations provided concrete, reproducible evidence for Botacin's arguments, moving the discussion beyond theoretical critique to practical, observable phenomena within AV operations.

Defensive Implications

▶ Watch: Adversarial attacks against ML-based AVs (32:00)

The insights provided by Marcus Botacin offer several crucial implications for defenders seeking to enhance their security posture and make informed decisions about antivirus solutions.

First and foremost, defenders must understand the limitations and design choices of their AV products. The notion that AVs are infallible or that a single metric like "detection rate" fully encapsulates their efficacy is misleading. Security teams should investigate whether their chosen AV primarily relies on whitelists for common software, how it handles various packers, and the depth of its dynamic analysis capabilities. This nuanced understanding enables more realistic threat modeling and better tailoring of defenses.

Fair evaluation of AVs is paramount. Rather than simply counting detections, organizations should consider the underlying AV engine, the scope of its dynamic analysis (e.g., number of hooked functions), and its performance overhead. When comparing new security solutions, ensure that the baseline AV is evaluated under comparable conditions, especially regarding data collection and resource utilization. This means asking critical questions about the AV's internal configuration and not just relying on vendor claims or superficial benchmarks.

Defenders should be acutely aware of the trade-offs AV vendors make. A product might prioritize avoiding false positives over comprehensive detection, or performance over deep analysis. These are not necessarily flaws but design decisions. Understanding these trade-offs helps in selecting an AV that aligns with an organization's specific risk tolerance and operational environment. For instance, if an organization frequently encounters custom-packed malware, an AV with robust unpacking capabilities would be a higher priority.

The discovery that quarantined files are often merely encoded, not encrypted, presents an opportunity for threat intelligence gathering. Security teams with appropriate access can potentially decode quarantined files to obtain fresh malware samples for internal analysis, reverse engineering, and improving their own detection rules, rather than relying solely on external feeds or paying for access to such samples.

The potential for cache exploitation highlights a subtle but critical vulnerability. Defenders should be aware that multi-stage attacks might leverage AV caching mechanisms. Implementing additional security layers, such as endpoint detection and response (EDR) solutions that monitor post-execution behavior, can help catch threats that evade initial AV scans due to cached verdicts.

The revelation of unencrypted AV updates (as seen with Avast) underscores the importance of securing the entire software supply chain. Organizations should verify that their AV solutions utilize encrypted and authenticated update channels. Any reliance on plain HTTP for critical security updates introduces a significant risk of man-in-the-middle attacks, potentially leading to compromise of the AV itself.

Finally, the "pristine installation" assumption is a vital, often unstated, threat model consideration. Defenders must recognize that installing an AV on an already infected system may not be effective due to race conditions. For compromised systems, a clean reinstallation followed by AV deployment is often the most secure approach. Furthermore, the significant limitations of Android AVs, which rely heavily on reputation and abuse accessibility services, suggest that mobile security for enterprises requires a multi-layered approach beyond basic AV, focusing on mobile device management (MDM), app vetting, and network-level protections. Physical security also emerges as a critical, often overlooked, aspect of AV integrity, as physical access can bypass many software-based self-protection mechanisms.

Key Takeaways

• Antivirus evaluations are often biased and unfair: Distinguish between AV products and their underlying engines to avoid miscounting detections and understand that many AV choices are deliberate trade-offs, not failures.
• Signatures remain relevant but are not the sole mechanism: 30-40% of detections still rely on signatures; understand that polymorphism and mutation are still viable evasion techniques, but over half of detections now use more advanced heuristics or machine learning.
• AVs make conscious trade-offs in capability and performance: Decisions regarding packer support, the number of hooked functions for dynamic analysis, and network interception methods are often strategic choices balancing protection, speed, and resource consumption.
• AVs collect vast amounts of user and system data: From kernel-level drivers for self-protection and network inspection (including MITM for HTTPS) to telemetry databases and caching mechanisms, AVs have deep system access, which can be reverse-engineered or exploited.
• AV update processes and threat model assumptions are critical vulnerabilities: Unencrypted update channels (e.g., Avast over HTTP) are supply chain risks, and the "pristine installation" assumption means AVs may fail to install or function correctly on already infected systems.
• Android AVs are significantly less robust: Due to platform limitations, Android AVs rely heavily on white lists, reputation, and abuse of accessibility services, offering limited on-device verification compared to desktop counterparts.

About the Speaker(s)

Marcus Botacin is a Computer Science Professor at Texas A&M University, where his research focuses on cybersecurity. He is dedicated to advancing the field by deeply understanding the internal workings of existing security systems, proposing new architectures (including hardware-supported antivirus), and identifying vulnerabilities to "break stuff." Botacin is an expert in machine learning security and has notably won the Microsoft, Endgame, and other companies' antivirus evasion competitions for three consecutive years, demonstrating his profound understanding of adversarial machine learning attacks against AVs. His work aims to foster a more accurate and constructive discourse around antivirus technology to drive innovation and improve defensive capabilities.