Human n’ Machine - Leveraging Automated Tool To Enhance Manual Analysis
Christian Das Neves (I.T security analyst and incident Handler · Compass security)
Swiss Cyber Storm 2022 · Day 1 · Panorama
Overview
In the contemporary landscape of cybersecurity, organizations face an ever-increasing volume and sophistication of threats. Automated tools have become indispensable for managing repetitive tasks and sifting through vast amounts of data, providing initial insights into potential breaches. However, as Christian Das Neves, an I.T. Security Analyst and Incident Handler at Compass Security, expertly elucidated in his talk at Swiss Cyber Storm, these tools frequently fall short of delivering a complete and accurate picture of the underlying threats. His presentation, "Human n’ Machine - Leveraging Automated Tool To Enhance Manual Analysis," delves into the critical interplay between automated analysis and human expertise in effective incident response, particularly concerning complex malware.

Key moments
- 0:25 Introduction: NIST incident handling and analysis focus
- 1:35 Understanding a typical malware infection flow (Emotet)
- 4:00 Automated tools miss obfuscated URLs and anti-analysis
- 8:00 Malware dynamically changes file names to evade detection
- 10:00 Malware behavior changes based on admin rights
- 12:00 Admin vs. user persistence mechanisms explained
- 14:30 Pros and cons of automated vs. manual analysis
- 17:00 Best practice: combining automated and manual analysis
Human n’ Machine - Leveraging Automated Tool To Enhance Manual Analysis
Speakers: Christian Das Neves, I.T security analyst and incident Handler, Compass security
Conference: Swiss Cyber Storm
YouTube: https://www.youtube.com/watch?v=MtHCwlOxz3A
Overview
In the contemporary landscape of cybersecurity, organizations face an ever-increasing volume and sophistication of threats. Automated tools have become indispensable for managing repetitive tasks and sifting through vast amounts of data, providing initial insights into potential breaches. However, as Christian Das Neves, an I.T. Security Analyst and Incident Handler at Compass Security, expertly elucidated in his talk at Swiss Cyber Storm, these tools frequently fall short of delivering a complete and accurate picture of the underlying threats. His presentation, "Human n’ Machine - Leveraging Automated Tool To Enhance Manual Analysis," delves into the critical interplay between automated analysis and human expertise in effective incident response, particularly concerning complex malware.
Das Neves's talk highlights a fundamental challenge: while automation offers speed and scalability, malware authors continuously evolve their techniques to evade automated detection and analysis. This creates a gap where automated systems, such as sandboxes, may report incomplete or misleading information, leaving organizations vulnerable to re-infection. The core premise of his presentation is that a synergistic approach, combining the efficiency of automated tools for initial triage with the depth and adaptability of manual analysis, is paramount for achieving robust and comprehensive incident handling. This article will explore the specific deficiencies of automated systems, the sophisticated anti-analysis techniques employed by modern malware, and the indispensable role of human analysts in bridging these gaps to ensure thorough containment and recovery.
Background
▶ Watch: Introduction: NIST incident handling and analysis focus (0:25)
Effective incident response is a cornerstone of organizational security posture. Christian Das Neves grounded his discussion in the NIST Incident Handling Guide, which outlines three essential goals: analysis, containment, and recovery. The talk specifically emphasizes the analysis phase, arguing that a superficial or incomplete understanding of an incident's scope, infection vector, and indicators of compromise (IOCs) renders subsequent containment and recovery efforts largely ineffective. Without thorough analysis, attackers can maintain persistence through hidden backdoors or web shells, leading to recurrent breaches.
To illustrate the complexities of malware analysis, Das Neves presented a detailed overview of a well-known malware infection flow, using Emotet as a prime example. The typical chain of infection begins with a user executing a malicious file, often a Word document containing a macro. This macro triggers a dropper, typically a PowerShell script or an executable, which is designed to download additional, more potent malware. These secondary payloads are frequently hosted on compromised servers or legitimate cloud providers like Dropbox or Amazon S3, making network-level blocking challenging. Once downloaded, the malware proceeds to gather information about the host system, establish persistence mechanisms to ensure continued access, and communicate with a Command and Control (C2) server. The C2 server then orchestrates further actions, including downloading additional malware (such as ransomware or info-stealers) or issuing direct commands. This cyclical process, often referred to as beaconing, underscores the dynamic nature of advanced threats.
The primary objectives during the analysis phase are to identify four critical elements: the dropper (to understand initial infection points), the network flow (to identify delivery servers and C2 infrastructure), the malware itself (to understand its capabilities and actions), and the C2 hosts (to sever communication and prevent further compromise). Das Neves meticulously demonstrated how automated tools, while helpful for initial identification, struggle to provide a complete picture of these elements due to sophisticated anti-analysis techniques employed by malware authors. This inherent limitation necessitates a deeper, often manual, analytical approach to uncover the full extent of the threat.
Key Findings
▶ Watch: Automated tools miss obfuscated URLs and anti-analysis (4:00)
Christian Das Neves's presentation unveiled several critical findings regarding the limitations of automated analysis tools and the indispensable role of manual expertise in modern incident response. The overarching discovery is that while automated solutions offer speed and initial insights, they are fundamentally susceptible to anti-analysis techniques embedded within sophisticated malware, leading to incomplete or inaccurate findings. This directly impacts the effectiveness of containment and recovery efforts.
One significant finding is the inconsistency and incompleteness of results across different automated sandboxes. Das Neves demonstrated how various sandbox environments, when presented with the same malicious document, yielded vastly different outputs regarding extracted URLs, file names, and execution paths. For instance, in the analysis of a dropper, one sandbox might identify only a single URL, another two, while only a manual review could uncover all five active URLs. This highlights that relying on a single automated report can leave critical attack infrastructure undetected.
Another key finding centers on the dynamic and evasive nature of malware indicators of compromise (IOCs). Malware frequently employs techniques to alter its behavior or characteristics based on the execution environment. This includes dynamically generating file names based on host variables, which can transform a single expected filename (e.g., 928.exe) into thousands of potential variations, rendering simple hash or filename-based blocking ineffective. Similarly, malware can check for specific environmental conditions, such as the presence of a network connection or administrator privileges, to determine its execution path or persistence mechanism. Automated tools, often running in isolated or standardized environments, may only observe a subset of these behaviors, missing crucial branches of the malware's logic.
Furthermore, Das Neves emphasized that manual analysis is crucial for overcoming these anti-analysis techniques. By stepping through code with a debugger, an analyst can observe the malware's decision-making processes, de-obfuscate hidden components, and uncover the full range of its capabilities and communication channels, particularly the highly obfuscated C2 infrastructure that automated tools often fail to identify. This deeper understanding allows for the extraction of precise IOCs, such as exact URLs rather than just domains (which is vital when attackers leverage legitimate cloud providers), and a comprehensive mapping of all potential infection vectors and persistence mechanisms.
Ultimately, the presentation underscored that the most effective strategy is a hybrid approach: leveraging automated tools for rapid initial triage and volume processing, followed by targeted manual deep dives for complex, evasive, or high-priority threats. This combination ensures that security teams benefit from the speed of automation while mitigating its inherent blind spots with the precision and adaptability of human intelligence.
Technical Deep Dive
▶ Watch: Malware behavior changes based on admin rights (10:00)
The technical core of Das Neves's talk meticulously dissected how malware employs various anti-analysis techniques to thwart automated detection and analysis, and how manual methods are essential to overcome these challenges.
Dropper Analysis and URL Evasion:
The initial stage of the Emotet infection flow involves a dropper, typically a Word document with a macro that executes a PowerShell script. Automated sandboxes can de-obfuscate these macros, revealing commands such as a Base64-encoded PowerShell script. However, the sophistication begins here. Das Neves presented an example where a PowerShell script contained multiple URLs. A simple regular expression to extract URLs might identify only two, while a human eye could discern five. Critically, some of these URLs were blended with legitimate domains, such as microsoft.com, designed to trigger false positives or confuse automated blocking systems. If an automated system were to block microsoft.com based on an incomplete regex match, it could cripple an organization's operations.
Furthermore, the malware incorporated a size check on the response from its download servers. If a sandbox environment blocks network access or returns an unexpected response size, the malware simply stops execution, appearing non-malicious and thus evading detection. This demonstrates how environmental assumptions in automated tools can be exploited. Das Neves showed that different sandboxes yielded inconsistent results, with some missing critical URLs. He stressed the importance of extracting exact URLs or routes rather than just domains, especially when attackers use legitimate cloud services (e.g., AWS S3 buckets or Google Drive links), to avoid blocking essential business services.
Dynamic File Naming and Evasion:
Once the dropper executes, it downloads additional malware. Initially, an analyst might expect a fixed filename, such as 928.exe. However, Das Neves revealed that malware frequently employs anti-detection systems that dynamically change the downloaded file's name based on host-specific variables like the system's hostname. Sandbox reports confirmed this, showing the same malware (identified by its MD5 hash) being saved with different filenames across various environments.
To counteract this, manual analysis with a debugger becomes crucial. By stepping through the malware's execution, an analyst can identify the logic responsible for generating these dynamic names. In the presented example, the malware took two words from an internal list and combined them to form the new filename. This mechanism could result in up to 4,000 potential filenames, making it impractical to block by name alone. Understanding this dynamic generation process through manual debugging allows defenders to focus on the execution pattern rather than the specific, ephemeral filename.
Execution Path and Persistence Mechanisms:
Malware persistence is a critical aspect of post-exploitation. Das Neves detailed how malware determines its persistence strategy based on its privilege level. Automated tools again showed inconsistent results, reporting execution paths in syswow64 in some instances and Local AppData in others. This discrepancy highlighted a common pitfall: automated sandboxes might run the malware under different user contexts (e.g., as an administrator vs. a standard user) without clearly indicating this.
Through a technical deep dive, Das Neves explained the malware's internal logic:
- Admin Rights Check: The malware first attempts to create a service. If successful, it sets an internal flag indicating administrator privileges.
- Admin Persistence: If admin rights are present, the malware uses the
SHGetFolderPathWindows API function. This function, documented by Microsoft, retrieves paths to well-known folders. By examining the parameters, Das Neves showed it targetedsyswow64(the 32-bit system folder on a 64-bit Windows system), creating a service for persistence. This ensures the malware starts every time the system boots. - User Persistence (Non-Admin): If admin rights are not available, the malware attempts to connect to its C2 server. If this connection is successful, it creates a registry key (e.g., in the
Runkey) to achieve persistence, launching the malware at boot or user login. If the C2 connection fails, the malware simply kills its process, preventing detection in sandboxes configured without network connectivity.
This complex logic means that to fully understand the malware's persistence, automated analysis requires executing the malware in multiple scenarios: once with administrator privileges and network access, and once as a standard user with network access. This multi-scenario testing is often beyond the scope or default configuration of typical automated sandboxes, underscoring the necessity for manual intervention or highly customized automated setups.
C2 Identification:
Finally, Das Neves noted that the identification of C2 servers remains one of the most challenging aspects for automated tools. C2 communication is typically highly obfuscated and can involve custom protocols, domain-generating algorithms (DGAs), or communication over legitimate channels. Automated systems often struggle to fully de-obfuscate and interpret this traffic, making manual analysis, often involving network traffic analysis and reverse engineering, indispensable for mapping the full C2 infrastructure.
In summary, the technical deep dive demonstrated that modern malware is designed to be polymorphic and environmentally aware, actively probing its execution environment and altering its behavior to avoid detection by automated systems. Overcoming these sophisticated anti-analysis techniques demands a human analyst's ability to interpret context, debug code, and systematically test different execution scenarios to uncover the complete threat landscape.
Demo / Proof of Concept
▶ Watch: Admin vs. user persistence mechanisms explained (12:00)
Christian Das Neves's presentation did not feature a live, interactive demonstration of a specific tool or a step-by-step proof of concept in the traditional sense. Instead, he effectively utilized screenshots and code snippets from various automated sandbox reports, debugger outputs, and malware code to illustrate his points. He walked the audience through a simulated analysis process, showing side-by-side comparisons of how different automated tools processed the same malware and highlighting the discrepancies and missing information.
For instance, he displayed the output of three different sandboxes, each yielding varying numbers of extracted URLs from a dropper, visually emphasizing the inconsistency of automated results. He also showed a snippet of PowerShell code, pointing out how a simple regex would miss cleverly obfuscated URLs. When discussing dynamic file naming, he presented sandbox reports indicating different filenames for the same malware hash, followed by a debugger screenshot illustrating the code logic responsible for generating these dynamic names. Similarly, for persistence mechanisms, he used flowcharts and Windows API documentation references to explain the malware's decision tree based on admin rights and network connectivity, rather than a live execution.
This approach served as a powerful demonstration of the analysis process itself, rather than a tool demo. It allowed Das Neves to convey the technical intricacies and the comparative strengths and weaknesses of automated versus manual analysis in a clear and compelling manner, effectively acting as a guided walkthrough of the challenges faced by an incident responder.
Defensive Implications
▶ Watch: Best practice: combining automated and manual analysis (17:00)
The insights presented by Christian Das Neves carry significant implications for cybersecurity defenders, underscoring the need for a re-evaluation of current incident response strategies. The primary defensive implication is the critical necessity of moving beyond sole reliance on automated analysis tools. While tools like sandboxes and SIEMs are invaluable for initial triage and handling high volumes of alerts, they cannot be the final arbiters of threat assessment, especially for sophisticated and evasive malware.
Defenders must invest in developing and retaining skilled human analysts capable of performing deep, manual malware analysis. This includes proficiency in reverse engineering, debugging (e.g., using tools like x64dbg or Ghidra), understanding assembly language, and interpreting complex API calls. Training programs should focus on equipping analysts with the knowledge to identify and bypass anti-analysis techniques, such as dynamic IOC generation, environment checks, and obfuscation methods.
A hybrid analysis workflow should be adopted as a standard operating procedure. Automated tools should serve as the first line of defense, efficiently processing known threats and providing initial indicators. However, any suspicious, ambiguous, or highly evasive samples must be escalated for manual investigation. This ensures that the speed of automation is balanced with the depth and accuracy of human expertise. Organizations should define clear criteria for when a manual deep dive is warranted.
When implementing network blocking strategies, defenders should prioritize precision over broad strokes. As Das Neves highlighted, blocking entire domains (e.g., microsoft.com or dropbox.com) based on incomplete automated analysis can lead to severe operational disruptions. Instead, efforts should focus on blocking exact URLs, specific file paths, or IP addresses identified through thorough analysis. This requires a deeper understanding of the malware's communication infrastructure.
Furthermore, defensive strategies must account for dynamic indicators of compromise (IOCs). Relying solely on static IOCs like file hashes or fixed filenames is insufficient when malware dynamically changes its properties. Defenders should instead focus on identifying behavioral IOCs, such as specific process spawning patterns, API calls, persistence mechanisms (e.g., service creation, registry key modifications), and unusual network traffic patterns. Endpoint Detection and Response (EDR) solutions that monitor these behaviors can be particularly effective.
Finally, organizations should consider customizing their automated analysis environments to better mimic real-world scenarios and test different malware execution paths. This could involve running sandboxes with and without administrator privileges, with and without network connectivity, and even simulating specific user environments. While this adds complexity, it can help uncover behaviors that would otherwise be missed by default sandbox configurations, providing a more comprehensive initial report for analysts.
By integrating these defensive implications, organizations can build a more resilient incident response capability that effectively counters the evolving tactics of sophisticated adversaries.
Key Takeaways
- Automated tools are essential for initial triage and scale, but they are inherently limited by sophisticated anti-analysis techniques employed by modern malware. Relying solely on them can lead to incomplete or misleading incident response.
- Manual analysis is indispensable for achieving a complete and accurate understanding of complex malware, especially when uncovering highly obfuscated Command and Control (C2) infrastructure, dynamic IOCs, and detailed persistence mechanisms.
- A synergistic human-machine approach is optimal, leveraging the speed and volume handling of automated tools for initial screening, and then employing the depth, adaptability, and contextual understanding of human analysts for critical deep dives.
- Malware actively employs anti-analysis techniques such as dynamic file naming (e.g., thousands of possible names), URL obfuscation (mixing legitimate and malicious URLs, size checks), and environmental checks (e.g., admin rights, network connectivity) to evade automated detection and analysis.
- Thorough understanding of malware persistence and execution paths requires multi-scenario analysis, often necessitating manual debugging to observe all logical branches (e.g., different behaviors for admin vs. non-admin users, or with/without network access).
- Incomplete analysis can lead to failed containment and recovery efforts, leaving organizations vulnerable to re-infection through missed backdoors or persistent access methods. Precise IOCs, including full URLs and detailed behavioral patterns, are crucial for effective blocking and remediation.
About the Speaker(s)
Christian Das Neves is an I.T. Security Analyst and Incident Handler at Compass Security. His role involves responding to security incidents and conducting in-depth analysis of threats. This presentation at Swiss Cyber Storm marked his first talk at the conference, where he shared valuable insights from his practical experience in bridging the gap between automated tools and manual expertise in the field of cybersecurity incident response.
Reviews
Dr. Zero (Offensive Security Researcher) — WEAK
This is a competently delivered Malware Analysis 101 class about why sandboxes aren't magic. If you didn't already know that malware dynamically generates filenames or checks for admin privileges, you shouldn't be working incident response. Fine for an intern onboarding session, but it brings absolutely zero net-new research to a conference stage.
Heather Calloway (CISO) — SOLID
This is a practitioner talk demonstrating how modern malware evades automated sandbox analysis by checking for execution environments and network connectivity. It won't change your strategy, but it’s a necessary reminder of why your SOC can't rely entirely on automated detonation to close tickets. I wouldn't send my board to this, but I'd ask my SOC director how we handle the escalation path for the evasive samples it highlights.