Utilizing Memory and Network Forensics for Scalable Threat Detection and Response

Andrew Case (Director of Research · Volexity)

SecTor 2016 · Day 2 · Tech 3 (801B)

Overview

In his SecTor presentation, Andrew Case, Director of Research at Volexity, delves into the critical need for scalable threat detection and response strategies in an era dominated by sophisticated, anti-forensics-minded attackers. The talk highlights the inherent failures of traditional incident response (IR) methods when confronted with modern threats that operate across hundreds or thousands of systems, often without leaving traces on disk. Case, a renowned expert in memory forensics and co-author of "The Art of Memory Forensics," advocates for a paradigm shift, emphasizing the combined power of advanced memory and network forensics to achieve comprehensive visibility and enable proactive threat hunting.

Watch on YouTube · Slides

Visual summary for Utilizing Memory and Network Forensics for Scalable Threat Detection and Response by Andrew Case
Visual summary for Utilizing Memory and Network Forensics for Scalable Threat Detection and Response by Andrew Case

Key moments

  1. 0:00 Introduction to scalable threat detection challenges
  2. 4:00 Modern attackers: memory-only malware and API hooking
  3. 11:30 Crucial questions for effective incident response
  4. 24:00 Memory forensics: defeating live tool interference
  5. 34:00 Network forensics: Passive DNS and NetFlow
  6. 46:00 Recommended open-source tools for detection and response
  7. 48:00 Q&A: DNS logging vs. passive DNS
  8. 50:00 Q&A: Memory dump integrity and SSL decryption

Utilizing Memory and Network Forensics for Scalable Threat Detection and Response

Speakers: Andrew Case, Director of Research, Volexity

Conference: SecTor

YouTube: https://www.youtube.com/watch?v=Z-juus67j0E

Overview

In his SecTor presentation, Andrew Case, Director of Research at Volexity, delves into the critical need for scalable threat detection and response strategies in an era dominated by sophisticated, anti-forensics-minded attackers. The talk highlights the inherent failures of traditional incident response (IR) methods when confronted with modern threats that operate across hundreds or thousands of systems, often without leaving traces on disk. Case, a renowned expert in memory forensics and co-author of "The Art of Memory Forensics," advocates for a paradigm shift, emphasizing the combined power of advanced memory and network forensics to achieve comprehensive visibility and enable proactive threat hunting.

The core premise of Case's discussion revolves around the idea that traditional security tools and investigative techniques are ill-equipped to handle the scale and stealth of contemporary cyberattacks. Attackers leverage memory-only malware, hook operating system APIs to evade detection, and utilize powerful, readily available toolsets to rapidly compromise entire networks. This talk provides a detailed roadmap for organizations to augment their networks and incident response capabilities, ensuring they can effectively answer crucial investigative questions, identify advanced persistent threats, and build robust defenses against the most challenging adversaries.

The importance of this presentation cannot be overstated for security professionals, incident responders, and network defenders. As attack surfaces expand and threat actors grow more adept at evading detection, understanding how to leverage deep system and network visibility becomes paramount. Case offers practical, actionable insights and open-source tool recommendations that enable organizations to move beyond reactive, signature-based defenses to a more proactive, data-driven approach, ultimately allowing for more effective and scalable threat detection and response.

Background

▶ Watch: Introduction to scalable threat detection challenges (0:00)

Historically, incident response focused on a limited number of systems. Attackers would typically compromise one machine, pivot to a critical data host, exfiltrate information, and then retreat. In this environment, anti-forensics techniques were less common or easily identifiable, making traditional disk-based forensics and live system analysis largely effective. Andrew Case's earlier work on insider threat investigations, for instance, involved analyzing one or two compromised machines, a scope that did not demand the massive scalability required today.

However, the landscape of cyberattacks has drastically evolved. Modern attackers, often state-sponsored or highly organized criminal groups, operate with scale as a core feature of their toolsets. They can rapidly compromise hundreds or even thousands of systems across distributed networks, utilizing sophisticated anti-forensics techniques to remain undetected. Case emphasizes several key reasons why traditional approaches fail:

  1. Scale of Compromise: Attackers no longer target a handful of systems; they leverage tools like PowerShell Empire (a free, open-source post-exploitation framework) to compromise entire networks in minutes. PowerShell Empire, for example, operates entirely in memory, encrypts all network traffic, and includes capabilities like keylogging, screenshots, and a port of Mimikatz for extracting plaintext and hashed credentials. The speaker highlights a recent client where Mimikatz was run on a domain controller, effectively compromising the entire network.
  2. Memory-Only Malware: A significant shift is the prevalence of malware that never touches disk. Case cites the Kaspersky breach report where the attackers successfully compromised Kaspersky's entire network because the malware on critical servers was memory-only. If the infected machine was rebooted, the malware disappeared, leaving no disk-based artifacts for traditional antivirus or endpoint detection and response (EDR) solutions to find.
  3. API Hooking: Even when malware does reside on disk, it often hooks operating system APIs. Traditional live incident response tools, whether user-land or kernel-land, rely on these very APIs to query system state (processes, network connections, file listings). By hooking these APIs, malware can selectively hide its presence, making it "literally impossible" to see on the live system. Case notes that companies are often compromised for years because their live tools simply cannot detect the hidden malware.
  4. Defender Tool Disadvantage: While attackers possess tools that scale to thousands of machines (e.g., remote PowerShell, Group Policy, scheduled tasks), defenders lack equivalent, widely available commercial or open-source solutions that can comprehensively verify the security of 10,000 endpoints simultaneously. This creates a significant imbalance, leaving defenders at a disadvantage.
  5. Limited Network Visibility: Many network sensors only look for known malicious websites or domains, rendering them ineffective against targeted attacks or new threats. Full network packet capture (PCAP) is often impractical due to volume, encryption (SSL by default for many sites), and the sheer processing power required.

These challenges underscore the urgent need for innovative approaches that transcend traditional disk-centric forensics and reactive monitoring, paving the way for the scalable, memory and network-focused strategies discussed in the talk.

Key Findings

▶ Watch: Crucial questions for effective incident response (11:30)

The presentation outlines several key findings and principles necessary for effective and scalable threat detection and response in modern environments:

  • Traditional Incident Response Fails Against Modern Threats: Current attack methodologies, characterized by scale, memory-only operations, and anti-forensics techniques, render traditional, disk-centric IR and live system analysis largely ineffective. Security teams cannot rely on tools that are susceptible to API hooking or that only scan for disk-based artifacts.
  • Memory Forensics is Indispensable: To overcome anti-forensics and detect memory-only malware, memory forensics is presented as the primary solution. By directly examining RAM, independent of operating system APIs, investigators can uncover hidden processes, network connections, injected code, and other transient artifacts that would be invisible on a live system. This capability allows for the detection of even the most advanced attackers.
  • Centralized, Comprehensive Endpoint Logging is Critical: Effective detection hinges on having detailed, historical endpoint data that is securely logged off-host. This includes minimum requirements like logins and logoffs and user/group creation (to detect backdoor accounts), and desirable additions like process creation and termination (with command-line arguments and hashes, ideally via tools like SCCM for Windows). Centralization defeats local log clearing by attackers and enables rapid, network-wide queries.
  • Enhanced Network Visibility is a Game Changer: Network data provides an undeniable truth about attacker lateral movement, command and control (C2), and initial compromise vectors that endpoint data alone cannot provide. Key technologies include NetFlow (for internal and external connection tracking), Passive DNS (for recording all DNS resolutions, identifying malicious domains and their changing IP addresses), and HTTP request/response logging (for reconstructing exploit chains and identifying unencrypted C2).
  • Scalability Through Data-Driven Pivoting: Instead of manually investigating thousands of machines, organizations must collect rich, historical endpoint and network data that allows for rapid querying. This enables investigators to pivot from a single indicator of compromise (IOC) to a small subset of potentially affected systems, dramatically reducing the scope of an investigation and making deep-dive forensics manageable.
  • Threat Hunting Becomes a Necessity and a Reality: With comprehensive visibility and historical data, proactive threat hunting becomes not only possible but essential. By actively searching for anomalies and suspicious behaviors across critical systems using memory and network forensics, organizations can detect sophisticated threats that automated tools miss, shifting from a reactive stance to a proactive defense.

These findings collectively emphasize the need for a holistic, data-centric approach to security that integrates deep system visibility with broad network intelligence, enabling defenders to counter the advanced tactics of modern adversaries.

Technical Deep Dive

▶ Watch: Network forensics: Passive DNS and NetFlow (34:00)

The core of Andrew Case's scalable detection and response strategy lies in augmenting both endpoint and network visibility with specific logging and forensic capabilities.

Endpoint Enhancements:

The first step on endpoints is to log events that matter and centralize them to prevent anti-forensics.

  • Essential Logging:
  • Logins and Logoffs: Crucial for tracking attacker movement. If a user account is compromised, centralized login/logoff records quickly reveal where the attacker leveraged those credentials.
  • User and Group Creation/Modification: Attackers often create backdoor accounts. Auditing these events helps identify persistent access.
  • Desirable Logging (More Intensive but Highly Valuable):
  • Process Creation and Termination: On Windows, enabling process auditing in event logs captures every process start, user, time, and command-line arguments. Linux and macOS achieve similar results with auditd. This data, when centralized, allows for queries like "what ran on my Linux server?" Challenges include attackers renaming executables (e.g., a.exe, w.exe) or using generic cmd.exe/powershell.exe without detailed command arguments.
  • SCCM (System Center Configuration Manager): For Windows environments, SCCM is highlighted as "crucial." When configured for process auditing, it provides the full path, SHA-1 hash, and user for every executed process. The SHA-1 hash is particularly powerful, allowing defenders to identify renamed malicious executables (e.g., w.exe could be Mimikatz if its SHA-1 hash matches known Mimikatz binaries via services like VirusTotal). This context is vital for discerning legitimate activity from stealthy malware.
  • Centralization of Logs: All endpoint logs must be sent to a secure, centralized server. Attackers routinely clear local log files. Centralization ensures log integrity and enables rapid, network-wide querying.

Memory Forensics:

Case unequivocally states that memory forensics is the only reliable way to detect advanced malware that employs anti-forensics or operates entirely in memory.

  • Bypassing Live Tool Limitations: Live IR tools are vulnerable to API hooking, where malware intercepts OS calls to hide its presence. Memory forensics tools, like Volatility, operate by directly accessing and parsing raw RAM, reconstructing data (processes, network connections, loaded modules, injected code) independently of the live operating system's potentially compromised APIs.
  • Detecting Memory-Only Malware: As demonstrated by the Kaspersky breach, malware can exist solely in RAM, leaving no disk footprint. A memory capture is the only way to find such infections.
  • Historical Context from RAM: Even after a malicious process terminates, artifacts (e.g., command-line arguments, network buffers) can linger in memory. Memory forensics can recover this historical data, which would be unavailable on the live system.
  • Scalability: Memory forensics, when automated for acquisition, can be more scalable than traditional disk imaging. The Volatility Framework is an open-source example, offering over 250 plugins to analyze various operating systems (Windows, Linux, Mac, Android) and extract virtually any data loaded into RAM.

Network Enhancements:

Network data provides critical, independent visibility into attacker actions, especially lateral movement and communication.

  • NetFlow: Provides records of all network connections (source/destination IP, ports, protocols). This is invaluable for tracking lateral movement within the network ("where did this internal system connect to?") and identifying outbound C2 connections. Unlike endpoint connection logs, NetFlow can be retained indefinitely, offering long-term historical context.
  • Passive DNS: Described as "the cheapest and most effective technology or process you can do for network security." A passive sensor (network tap) records all DNS requests and replies.
  • Malware C2: Malware must resolve domain names before contacting C2 servers. Passive DNS captures these resolutions, allowing defenders to query "which machines in my environment resolved bad-domain.org?"
  • Dynamic DNS: Attackers frequently use dynamic DNS providers or domains with short Time-To-Live (TTL) values, constantly changing IP addresses. Passive DNS records all IP addresses a domain resolves to over time, enabling comprehensive blocking and historical searches across web proxy logs or NetFlow.
  • HTTP Requests and Responses: For unencrypted web traffic, logging HTTP requests and responses can reconstruct attack chains. Case describes scenarios where a user's request to a legitimate website leads to a malicious advertisement, which then serves a JavaScript or Flash exploit. These logs help identify the full exploit chain, discover other potentially compromised users who followed the same path, and block all associated malicious servers.

Integrated Analysis and Automation:

The power of these capabilities is realized by integrating and analyzing the collected data.

  • Automated Log Analysis: With centralized logs, automated queries can detect anomalies:
  • "Why did Bob from accounting log into a domain controller?" (identifies out-of-policy access).
  • "Why is an admin logging in outside of normal hours?"
  • "Why does this user account, which shouldn't be a domain admin, have processes running as one?" (detects privilege escalation).
  • Threat Hunting Queries (Network):
  • "Show me every time a newly registered domain was resolved." (Malware authors register domains in bulk, legitimate companies don't).
  • "Show me resolutions of dynamic DNS providers." (Often malware hosting farms).
  • "Look for unusual user agents in HTTP logs." (Malware often uses hard-coded or distinct user agents).
  • "Identify suspicious file downloads (e.g., .exe files) via HTTP logs."
  • Statistical Analysis: "Which domains were resolved by less than 100 people in my 10,000-person organization?" or "Which IP addresses did only one employee talk to?" These queries highlight outliers that automated threat intelligence might miss.

By combining these endpoint and network forensic capabilities, organizations can build a comprehensive, historical dataset that enables rapid, targeted investigations and proactive threat hunting, transforming their ability to detect and respond to advanced threats.

Demo / Proof of Concept

▶ Watch: Recommended open-source tools for detection and response (46:00)

While Andrew Case’s talk provides numerous real-world examples and compelling arguments for the efficacy of memory and network forensics, it is a conceptual and strategic presentation rather than a live demonstration. The speaker references various open-source tools and frameworks that implement the discussed techniques, such as Volatility for memory forensics, Graylog and the ELK stack for log aggregation and analysis, and Bro (now Zeek) or Suricata for network capture. However, the presentation itself does not include a live demo or a step-by-step proof of concept of these tools in action. The focus is on the architectural and procedural changes required to achieve scalable detection and response, rather than an interactive showcase of specific tool functionality.

Defensive Implications

▶ Watch: Q&A: Memory dump integrity and SSL decryption (50:00)

Implementing the strategies outlined by Andrew Case requires a fundamental shift in defensive posture, moving away from reactive, signature-based approaches to proactive, data-driven security.

  1. Prioritize Centralized, Comprehensive Endpoint Logging:
  • Log Everything Critical: Ensure all logins/logoffs, user/group creation/modification, and process creation/termination events (including command-line arguments and hashes) are logged. For Windows, leverage SCCM for rich process metadata (SHA-1 hashes are invaluable for identifying renamed malware).
  • Secure Centralization: All logs must be forwarded to a secure, off-host logging solution (e.g., Graylog, ELK stack) to prevent attackers from clearing local traces. This also enables rapid, correlation across the entire environment.
  • Retention Policy: Establish a robust log retention policy. While a week is insufficient, several months to a year provides the necessary historical context for breach detection and investigation.
  1. Integrate Memory Forensics into Incident Response & Threat Hunting:
  • Acquisition Capability: Develop the capability to rapidly acquire memory dumps from any endpoint, especially critical servers or suspected compromised machines.
  • Proactive Analysis: Regularly perform deep memory forensics on critical systems (domain controllers, exchange servers, source code repositories) as part of a threat hunting program. This helps uncover memory-only malware or API-hooking threats that bypass traditional EDR.
  • Tooling: Utilize open-source frameworks like Volatility for comprehensive memory analysis across diverse operating systems.
  1. Enhance Network Visibility with Strategic Data Capture:
  • Implement Passive DNS: This is identified as the "cheapest and most effective" network security measure. Deploy passive sensors to record all DNS requests and replies. This provides crucial intelligence on C2 domains, dynamic DNS usage, and newly registered domains.
  • Collect NetFlow: Implement NetFlow to track all internal and external network connections. This is essential for understanding lateral movement and identifying suspicious outbound communications over extended periods.
  • Log HTTP Requests/Responses: Where SSL decryption is feasible and legally permissible, log HTTP requests and responses to reconstruct exploit chains and identify unencrypted C2. If not, focus on SSL certificate analysis and domain resolution from Passive DNS.
  • Network IDS/IPS: Deploy robust network intrusion detection/prevention systems like Bro (Zeek) or Suricata to detect known malicious network patterns and extract metadata.
  1. Automate Detection and Querying:
  • Build Automated Alerts: Leverage centralized log and network data to create automated alerts for anomalous behaviors:
  • User accounts logging into unusual systems (e.g., accounting user on a domain controller).
  • New user/group creation in critical security groups.
  • DNS resolutions to newly registered domains or dynamic DNS providers.
  • Unusual user agents or suspicious file downloads over HTTP.
  • Outlier network connections based on statistical analysis.
  • Rapid Querying: Ensure the chosen logging and network data solutions support fast, complex queries across vast datasets to enable quick pivoting during an investigation.
  1. Embrace Proactive Threat Hunting:
  • Dedicated Teams/Schedule: Establish dedicated threat hunting teams or integrate regular hunting exercises into security operations. Case notes that security-conscious companies perform quarterly or even 24/7 threat hunting.
  • Hypothesis-Driven: Use the rich, historical data from endpoints and networks to formulate hypotheses about attacker activity and then actively seek evidence to confirm or deny them.
  • Beyond Automation: Acknowledge that automated tools alone cannot keep up with sophisticated attackers. Human analysts, guided by comprehensive data, are required to find the "unknown unknowns."

By adopting these defensive implications, organizations can build a resilient security architecture that provides the necessary visibility and analytical capabilities to detect, respond to, and ultimately deter even the most advanced and stealthy cyber threats.

Key Takeaways

  • Modern Attackers Bypass Traditional Defenses: Sophisticated adversaries leverage memory-only malware and API hooking to evade traditional disk-based forensics and live system analysis, rendering many existing security tools ineffective.
  • Memory Forensics is Essential for Stealthy Threats: Direct analysis of RAM, using tools like Volatility, is crucial for detecting malware that operates entirely in memory or actively hides its presence from the operating system's APIs.
  • Comprehensive, Centralized Logging is Non-Negotiable: Securely collecting and centralizing detailed endpoint logs (logins, process creation with hashes via SCCM, user/group changes) and network data (NetFlow, Passive DNS, HTTP requests/responses) is foundational for historical context and anti-forensics evasion.
  • Passive DNS is a High-Impact, Cost-Effective Network Defense: Implementing Passive DNS provides invaluable intelligence on malicious domain resolutions, dynamic DNS usage, and newly registered domains, significantly enhancing early detection of C2 and targeted attacks.
  • Scalability Relies on Data-Driven Pivoting: By collecting rich, queryable historical data from both endpoints and the network, incident responders can quickly narrow down investigations from thousands of machines to a manageable few, enabling targeted deep-dive forensics.
  • Proactive Threat Hunting is the Future: Leveraging combined memory and network forensics, along with advanced log analysis, enables organizations to move beyond reactive detection to proactive threat hunting, uncovering sophisticated threats that automated systems often miss.

About the Speaker(s)

Andrew Case is the Director of Research at Volexity, a leading incident response and threat intelligence company. He is a recognized expert in the field of memory forensics and is a co-author of the seminal book, "The Art of Memory Forensics: Detecting Malware and Threats in Windows, Linux, and Mac Memory." Andrew is also a prominent developer of open-source forensic tools, including the widely used Volatility Framework for memory analysis and Registry Decoder for registry forensics. His professional background includes extensive experience in insider threat investigations, and in his current role, he focuses on scalable threat detection and response in large, complex network environments. His work emphasizes innovative approaches to cybersecurity challenges, particularly in combating advanced persistent threats and anti-forensics techniques.

Reviews

Dr. Zero (Offensive Security Researcher) — SOLID

Andrew Case knows more about the Windows kernel pool than most Microsoft engineers. This is a rock-solid, no-bullshit primer on why live-response tools get lied to by rootkits, delivered by the guy who literally wrote the Volatility framework.

Heather Calloway (CISO) — STRONG ACCEPT

A sobering look at the limitations of standard EDR against memory-resident threats. While enterprise-wide memory forensics remains operationally out of reach for most teams, Case's architectural requirements for logging and passive DNS are mandatory reading for your security engineering directors.

→ Top-rated talks at SecTor 2016

All talks from SecTor 2016