Sysmon Deep Dive: Real Detection Scenarios You Can Reproduce

Sysmon Deep Dive: Real Detection Scenarios You Can Reproduce

Speakers: Peter Morin, Leader of PWC's OT Cyber Security Practice, PwC

Conference: FIRSTCON

YouTube: https://www.youtube.com/watch?v=6_xz_WRlEIE

Overview

In this compelling talk at FIRSTCON, Peter Morin, a seasoned leader in PwC's OT Cyber Security Practice, provided a comprehensive deep dive into Sysmon (System Monitor), a powerful and often underutilized tool for advanced endpoint telemetry. Morin passionately advocated for Sysmon's critical role, particularly in Operational Technology (OT) environments where the deployment of traditional Endpoint Detection and Response (EDR) solutions is frequently restricted by vendor limitations and operational sensitivities. The presentation underscored Sysmon's ability to fill significant visibility gaps left by default Windows logging, offering a robust, free, and low-overhead alternative for detecting sophisticated threats.

Morin's central thesis revolved around the evolving threat landscape, characterized by increasingly permeable network perimeters, rampant use of encryption, and the adversary's reliance on legitimate system tools (LOLBINs) to evade detection. He highlighted the urgent need to reduce dwell time—the period an attacker remains undetected in a network—which currently averages 11 days globally and 5 days for ransomware incidents. Sysmon, by providing granular, unfiltered event data, emerges as a pivotal tool for identifying and containing attackers during the crucial lateral movement phase, which accounts for approximately 80% of an attack's duration. The talk served as a practical guide, demonstrating how Sysmon's rich logging capabilities, coupled with intelligent configuration, can empower defenders to transform raw data into actionable threat intelligence.

Background

▶ Watch: Introduction: Why Sysmon is vital for OT security (0:00)

The modern cybersecurity landscape is defined by an "increasingly permeable perimeter," a concept Morin introduced to describe the blurring boundaries of enterprise networks due to cloud adoption, shadow IT, and the pervasive use of encryption by both legitimate applications and malicious actors. This complexity makes traditional perimeter-based defenses less effective. Compounding this challenge, particularly in OT environments, is the frequent inability to deploy full-fledged EDR solutions. Vendors of industrial control systems often prohibit third-party software installations, leaving security teams with limited options for endpoint visibility and threat response. Morin, drawing from 15 years in OT and 30 years in the industry, emphasized that Sysmon is often the most sophisticated telemetry tool permitted in such sensitive operational settings.

The overarching goal in cybersecurity is to drastically reduce dwell time, the period between initial compromise and detection. While global average dwell time has decreased to around 11 days (and 5 days for ransomware), Morin posed a critical question to the audience: "Can you survive 11 days? Could you survive 5 days? Could you survive one day?" This highlights the imperative for enhanced detection capabilities. Attackers spend roughly 80% of their time engaged in lateral movement after gaining an initial foothold, attempting to discover and access high-value assets. This phase represents a prime opportunity for defenders to identify and trap adversaries, as defenders inherently possess a deeper understanding of their own networks than the attackers. However, this is complicated by the nonlinear nature of incidents and the widespread use of Living Off the Land Binaries (LOLBINs), PowerShell, WMI, and RDP – legitimate tools that can be weaponized, making malicious activity appear benign and akin to finding "a needle in a stack of needles."

Default Windows logging, primarily via the Windows Event Viewer, offers a baseline level of visibility but suffers from significant limitations. Morin pointed out that these logs frequently miss crucial data points, exhibit inconsistent reporting (e.g., blank command line fields in Event ID 4688), lack native filtering capabilities, and fail to capture common attack behaviors like thread injection or comprehensive network telemetry. Key information such as DNS queries, detailed command-line arguments, and parent process relationships are either absent or severely limited, rendering standard Windows events insufficient for robust threat detection and forensic analysis. It is against this backdrop of critical visibility gaps and operational constraints that Sysmon emerges as an indispensable tool.

Key Findings

▶ Watch: Introducing Sysmon: Origins, capabilities, and key advantages (6:00)

Sysmon, or Microsoft System Monitor, was originally developed internally by Microsoft, based on the Process Monitor (Procmon) foundation, by Mark Russinovich and his team at Sysinternals. It was released to the public in 2014 and has since undergone numerous iterations, becoming a cornerstone for advanced Windows endpoint logging. A significant development highlighted by Morin is its recent inclusion as a built-in feature in Windows 11 and Windows Server 2012 (with relevant patches), making it even more accessible and legitimate for environments previously hesitant to install "third-party" tools.

The talk emphasized several key findings regarding Sysmon's capabilities and advantages:

• Free and Well-Maintained: Sysmon is a free utility from Microsoft, actively maintained and updated, ensuring its relevance against evolving threats.
• Minimal Performance Overhead: Extensive testing has shown that Sysmon imposes virtually no performance hit on systems, making it suitable for deployment across entire fleets, including sensitive OT systems. It leverages Event Tracing for Windows (ETW), the same efficient mechanism used by the operating system for its own logging.
• Deep Telemetry: Operating as a kernel-mode device driver, Sysmon sits low in the Windows stack, enabling it to capture highly granular and low-level system activity that often bypasses higher-level monitoring.
• Separate and Persistent Logging: Sysmon logs to its own dedicated event log, ensuring it doesn't overwrite standard Windows Event Logs (e.g., Security, System). Its configuration and logging persist across system reboots.
• Rich Event Data: Unlike the limited default Windows events, Sysmon provides a wealth of detailed information for each event. For instance, Sysmon Event ID 1 (Process Create) includes not only the process name, command line, and parent process but also cryptographic hashes (MD5, SHA1, SHA256, IMPHASH) of the executable. This hashing capability is a "super cool thing" that allows defenders to identify and track specific binaries (e.g., malicious word.exe imposters) across an entire enterprise via a SIEM.
• Comprehensive Event Types: Sysmon logs a wide array of activities beyond basic process creation, including:
• Network Connections (Event ID 3): Details on source/destination IP, port, protocol, and the process initiating the connection. This is crucial for detecting command-and-control (C2) activity and exfiltration.
• DNS Queries (Event ID 22): Provides visibility into DNS lookups made by processes.
• File Creation (Event ID 11): Logs the creation of files, including their paths and names, useful for detecting persistence mechanisms or dropped malware.
• Registry Activity (Event IDs 12, 13, 14): Monitors creation, modification, and deletion of registry keys and values, invaluable for malware analysis and forensic investigations.
• Process Access (Event ID 10): Logs when one process accesses another, critical for detecting credential theft attempts against processes like lsass.exe.
• Sysmon Service State Change (Event ID 4): Logs when the Sysmon service itself is stopped or started, providing a critical alert for tampering attempts.
• Complementary to EDR: Morin stressed that Sysmon is not a replacement for EDR but a powerful enhancement. EDR tools often filter data, providing only what they deem "interesting" or detected as malicious. Sysmon, conversely, provides complete, unfiltered telemetry, which is vital for thorough threat hunting and for catching threats that EDR might miss due to evasion techniques or unknown malicious signatures. For example, some EDRs might only alert on LSASS access if performed by a known tool like Mimikatz, whereas Sysmon Event ID 10 logs all such accesses.
• Highly Configurable XML Schema: A standout feature is the ability to precisely control what Sysmon logs through XML configuration files. These files allow for fine-grained filtering (include/exclude rules) based on process names, command-line arguments, image paths, signature status (e.g., Microsoft-signed), network ports, and more. This addresses the concern about "a crap ton of data" by enabling defenders to log only relevant events, and configurations can be dynamically changed and deployed across the enterprise for specific incident response (IR) scenarios.

Technical Deep Dive

▶ Watch: Complementing EDR: Sysmon for unfiltered threat hunting data (9:00)

Sysmon's power lies in its deep integration with the Windows kernel and its flexible configuration. At its core, Sysmon leverages Event Tracing for Windows (ETW), a highly efficient, kernel-level logging mechanism. This allows Sysmon to capture events with minimal overhead and high fidelity, directly from the operating system's internal workings. As a kernel-mode device driver, it operates at a privileged level, granting it unparalleled visibility into system activities.

A key differentiator is the richness of Sysmon's event data compared to standard Windows Event Viewer logs. Morin illustrated this with a comparison between EVT 4688 (Process Create) and Sysmon Event ID 1 (Process Create). While EVT 4688 might occasionally have blank fields for critical data like the command line, Sysmon Event ID 1 consistently provides comprehensive details, including:

• ProcessId and ProcessGuid
• Image (executable path)
• CommandLine
• ParentProcessGuid and ParentImage
• User
• Crucially, Hashes (MD5, SHA1, SHA256, IMPHASH) of the executable, enabling unique binary identification.

The true flexibility of Sysmon is unlocked through its XML configuration files. These files dictate precisely which events Sysmon should log and under what conditions. This granular control is essential for managing log volume, focusing on specific threats, and adapting to different operational needs. Configuration files can be created manually or using community tools, then pushed out to endpoints using standard deployment mechanisms like Ansible, Puppet, Group Policy, or SCCM, allowing for near-instantaneous changes across an enterprise.

Morin provided several practical examples of XML filter conditions:

• Targeting Network Activity: To monitor network connections initiated only by Google Chrome, a filter could specify Image condition contains chrome.exe for Sysmon Event ID 3 (Network connection initiated).
• Detecting Thread Injection: To specifically log thread injection attempts into critical processes like winlogon.exe and lsass.exe, a rule could target Sysmon Event ID 8 (CreateRemoteThread) with TargetImage condition is lsass.exe or winlogon.exe.
• Excluding Trusted Binaries: To reduce noise, Sysmon can be configured to exclude events from processes signed by trusted vendors. For example, Signature condition contains Microsoft can be used to ignore image loads from Microsoft-signed executables.
• Identifying Suspicious Network Origins: Malicious processes often run from unusual locations. A filter for Sysmon Event ID 3 could look for network connections where Image condition begins with C:\ProgramData\, as legitimate programs rarely initiate network activity from this directory.
• Monitoring Suspicious File Creations: Attackers frequently drop scripts or drivers. Sysmon can be configured to log Sysmon Event ID 11 (File created) for specific file extensions, such as .scr (screensaver), .sys (system driver), .vbs (Visual Basic script), or .hta (HTML Application), which are often used in malicious campaigns.

Sysmon's rich data, when ingested into a Security Information and Event Management (SIEM) system like Splunk (as demonstrated by Morin), becomes a powerful foundation for threat hunting and detection. Morin detailed several use cases:

• DNS Traffic Identification: Sysmon Event ID 3 can log all DNS queries, allowing a SIEM to identify suspicious domains or unusual DNS activity.
• PowerShell Execution Monitoring: Sysmon can capture detailed PowerShell command lines, including those using encoded commands (-EncodedCommand), which are frequently used to obfuscate malicious payloads.
• Registry Modifications from Command Line: Sysmon Event ID 1 can reveal processes like command.exe executing reged add commands, indicating potential persistence or system alteration.
• Abnormal Process Spawning: By correlating Sysmon Event ID 1 data, a SIEM can detect command.exe or powershell.exe being spawned by unusual parent processes (e.g., Winword.exe, Excel.exe), rather than the typical explorer.exe or another command prompt instance.
• Sysmon Tampering Detection: Sysmon Event ID 4 (Sysmon service state changed) is logged the moment the Sysmon service is stopped, providing a critical alert if an attacker attempts to disable logging.
• Metasploit Hunting: Sysmon Event ID 3 can be configured to detect network connections to common Metasploit Meterpreter shell ports like 4444 and 5555.
• Credential Theft Detection: Sysmon Event ID 10 (ProcessAccess) is crucial for detecting attempts to access lsass.exe (e.g., by Mimikatz or ProcDump) for credential dumping.
• Unique Binary Search by Hash: The Hashes field in Sysmon Event ID 1 allows defenders to collect unique hashes of all executed binaries, submit them to services like VirusTotal, and then search the SIEM for any machine that executed a known malicious hash, enabling rapid containment.
• Session Tracking with Login GUID: Sysmon logs a LoginGuid for each user session. By extracting this GUID, defenders can trace all process creations (Sysmon Event ID 1) and other activities within a specific user's login session, providing a comprehensive timeline of attacker actions.

These examples underscore how Sysmon provides the raw, rich telemetry necessary to build sophisticated detection rules and conduct thorough forensic investigations, even without the presence of an EDR solution.

Demo / Proof of Concept

▶ Watch: Customizing Sysmon logging with XML configuration files (16:00)

While no live, interactive demonstration was performed during the talk, Peter Morin effectively illustrated the practical application of Sysmon through numerous "use cases" presented as conceptual proofs of concept. Each use case demonstrated how Sysmon's detailed logging, combined with intelligent XML configurations and SIEM analysis (specifically using Splunk as an example), could detect specific attack techniques.

For instance, Morin showed how Sysmon Event ID 1 logs, when ingested into Splunk, could highlight PowerShell execution with encoded commands, revealing attempts to obfuscate malicious scripts. He presented Splunk queries that would identify abnormal parent-child process relationships, such as command.exe being spawned by Microsoft Office applications rather than explorer.exe. The utility of unique binary hashes was demonstrated by showing how Sysmon could collect these hashes, allowing for quick cross-enterprise searches and external validation via services like VirusTotal. Furthermore, the concept of detecting Metasploit activity was visually explained by showing how Sysmon Event ID 3 for network connections could be filtered to identify communication on common Meterpreter ports.

These examples, complete with screenshots of hypothetical Splunk outputs, served as clear demonstrations of Sysmon's capability to provide the necessary data for effective threat detection and hunting. They functioned as blueprints for defenders to implement similar monitoring and alerting strategies within their own environments, validating the core premise that Sysmon delivers actionable intelligence for real-world security scenarios.

Defensive Implications

▶ Watch: Real-world detection scenarios: PowerShell, Metasploit, credential theft (20:00)

The detailed insights provided by Sysmon have profound defensive implications for organizations, particularly those operating in constrained environments like OT.

1. Enhanced Visibility in EDR-Restricted Environments: For OT networks or other critical infrastructure where full EDR deployment is unfeasible due to vendor restrictions, Sysmon serves as an indispensable tool for gaining deep endpoint visibility. Its low-overhead, kernel-mode operation, and official Microsoft backing make it a permissible and effective alternative for collecting crucial telemetry.
2. Complementary to Existing EDR Solutions: Even for organizations with EDR, Sysmon acts as a powerful complement. It provides unfiltered, raw event data that can expose threats missed by EDRs (due to evasion techniques or limited scope), offering a critical layer of defense and a robust dataset for proactive threat hunting.
3. Proactive Threat Hunting: The granular data from Sysmon, especially process hashes, detailed command lines, and network connections, empowers security analysts to conduct sophisticated threat hunts. By searching for specific indicators of compromise (IOCs) or anomalous behaviors across their entire fleet, defenders can uncover hidden threats before they escalate.
4. Rapid Incident Response: In an incident, Sysmon's ability to quickly adapt its logging configuration via XML files is invaluable. Defenders can deploy highly specific configurations to collect forensic data related to known attacker techniques or IOCs, accelerating investigation and containment efforts. The LoginGuid feature allows for comprehensive session tracking, providing a clear timeline of attacker actions.
5. Detection of LOLBIN Abuse: Sysmon's detailed logging of command-line arguments and parent-child process relationships is crucial for detecting the malicious use of legitimate system tools (LOLBINs, PowerShell, WMI, RDP). Defenders can create rules to alert on unusual patterns of these tools' execution.
6. Supply Chain Integrity: By hashing every executed binary, Sysmon provides a mechanism to verify the integrity of software. Defenders can compare these hashes against known good baselines or threat intelligence feeds (like VirusTotal) to detect unauthorized modifications or the introduction of malicious executables.
7. Tampering Detection: Monitoring Sysmon Event ID 4 ensures that defenders are immediately alerted if an adversary attempts to disable the logging service itself, providing a last-ditch warning against sophisticated attackers.
8. SIEM Integration is Key: To maximize Sysmon's value, integrating its logs into a SIEM (e.g., Splunk) is critical. This enables centralized collection, correlation, visualization, and alerting across the entire enterprise, transforming raw event data into actionable security intelligence.
9. Strategic Deployment: While Sysmon can be deployed on all Windows hosts without significant performance impact, organizations new to it might start with critical servers and high-value assets before expanding to workstations. The use of community-provided configuration files (e.g., SwiftOnSecurity's) can accelerate initial deployment.

In essence, Sysmon equips defenders with unparalleled visibility into Windows endpoint activity, enabling them to reduce dwell time, enhance detection capabilities, and respond more effectively to the complex threats of the modern cybersecurity landscape.

Key Takeaways

• Sysmon is a powerful, free, and low-overhead Microsoft utility that provides deep, kernel-level telemetry for Windows endpoints, essential for environments where traditional EDR solutions are not feasible.
• It offers significantly richer and more consistent event data than default Windows logs, including critical details like process binary hashes, full command lines, parent processes, network connections, and DNS queries.
• Sysmon is highly configurable via XML files, allowing defenders to precisely control logging volume and focus on specific threat detection scenarios, with the flexibility to adapt configurations rapidly during incident response.
• Key detection capabilities include identifying abnormal process spawning, PowerShell encoded commands, suspicious network activity, registry modifications, and the unique identification of malicious binaries through hashing.
• Integrating Sysmon logs into a SIEM is crucial for enterprise-wide threat hunting, centralized alerting, and effective forensic analysis, enabling defenders to correlate events across multiple systems.
• Sysmon complements EDR tools by providing unfiltered data, enhancing resilience against evasion techniques, and filling visibility gaps for comprehensive endpoint security.

About the Speaker(s)

Peter Morin is the esteemed Leader of PwC's OT Cyber Security Practice in Canada. With a distinguished career spanning 30 years in the industry, including 15 years specifically dedicated to operational technology (OT) security, Peter is a recognized expert in safeguarding critical infrastructure. His extensive experience in OT environments has made him a strong proponent of Sysmon, particularly due to the frequent limitations on deploying traditional EDR tools in such sensitive settings. Peter is passionate about finding practical and effective ways to detect threats and reduce dwell time, and he actively shares his knowledge, including through a podcast he co-hosts.