SIEM Slam: Tricking Modern SIEMs with Fake Logs and Confusing Blue Teams

SIEM Slam: Tricking Modern SIEMs with Fake Logs and Confusing Blue Teams

Speakers: Ozgun Kultekin (Offensive Security Engineer, Trend Micro Group); Asil Veral

Conference: Hacktivity

YouTube: https://www.youtube.com/watch?v=_bdVMbyPkfI

Overview

In the realm of cybersecurity, Security Information and Event Management (SIEM) systems are the bedrock of detection and response, offering blue teams a centralized view of an organization's security posture. However, what if the very data these systems rely upon could be manipulated by an attacker? Ozgun Kultekin's Hacktivity talk, "SIEM Slam: Tricking Modern SIEMs with Fake Logs and Confusing Blue Teams," dives deep into this unsettling possibility, demonstrating how compromised systems can be leveraged to inject fabricated logs, mislead defenders, and buy critical time for malicious operations.

Kultekin, an Offensive Security Engineer at Trend Micro Group, unveils a novel attack vector that exploits common SIEM configurations, specifically focusing on Splunk. The presentation outlines a methodology and introduces a tool, log_slapper, designed to intercept, modify, and inject false security events into a SIEM. This research is not merely theoretical; it provides a tangible blueprint for red teams to create plausible, yet entirely fictitious, attack scenarios that can divert blue team resources and attention away from actual threats.

The implications of "SIEM Slam" are profound. It challenges the fundamental assumption of trust in log data, forcing security professionals to reconsider the integrity of their most vital detection mechanisms. By showcasing how attackers can craft convincing narratives of compromise on non-existent hosts or attribute malicious actions to innocent assets, Kultekin highlights a critical blind spot in many enterprise security architectures. The talk serves as a stark reminder that advanced adversaries are continuously seeking ways to undermine defensive capabilities, and that even the most sophisticated SIEM deployments can be rendered ineffective if their underlying data streams are compromised.

Background

▶ Watch: Introduction and talk motivation (0:00)

The genesis of this research stems from practical challenges encountered during breach and attack simulation exercises. Red teams frequently find themselves needing dwell time—uninterrupted access within a compromised environment—to conduct reconnaissance, exfiltrate data, or escalate privileges without detection. While simply killing a SIEM agent (like a Splunk Universal Forwarder) might seem like an obvious move to avoid logging, it's often a noisy operation that immediately alerts blue teams. Kultekin's team sought a more subtle approach: instead of stopping the flow of information, why not manipulate it?

SIEM solutions are ubiquitous in modern enterprise security. They collect, aggregate, and analyze logs from a vast array of sources, including Linux, Windows, cloud environments, applications, and even specialized systems like SCADA. This data forms the basis for threat detection, incident response, compliance auditing (e.g., PCI DSS, GDPR), and forensic investigations. The speaker noted Splunk's significant market presence, even referencing its multi-billion dollar acquisition as an indicator of its critical role. A typical Splunk architecture involves forwarders (agents on data sources) sending logs to indexers (storage and processing), which are then queried by search heads or presented via the Splunk Web UI.

The core problem for an attacker, even after gaining initial access and root/system privileges on a machine, is the inherent traceability of their actions. Every command executed, every file accessed, every network connection made leaves a digital footprint that SIEMs are designed to capture. Without a mechanism to obscure or misdirect these traces, an attacker's window of opportunity is severely limited. This led to the central hypothesis of "SIEM Slam": if an attacker could create fake logs, they could mimic an attack on a different, perhaps more critical, asset (e.g., a payment server), launch a completely fabricated ransomware scenario, or simply flood the SIEM with noise, thereby diverting blue team attention and buying precious time for their true objectives.

Key Findings

▶ Watch: Attackers' dilemma: limited time, need fake logs (9:45)

The "SIEM Slam" research uncovered several critical vulnerabilities and demonstrated a powerful new method for red team operations:

1. Plain Text Log Transmission: A foundational discovery was that many modern SIEMs, including Splunk in its default configuration, transmit log data between forwarders and indexers in plain text over TCP. This unencrypted communication channel is a significant weak point, making the traffic susceptible to interception and manipulation.
2. Real-Time Log Interception and Modification: Kultekin demonstrated that an attacker with root privileges on a compromised host running a SIEM forwarder can intercept this plain text TCP traffic in real-time. By parsing the raw log data, specific fields such as executed commands, hostnames, and IP addresses can be extracted and altered.
3. Fabricated Security Incidents: The log_slapper tool, developed as part of this research, enables the injection of entirely fabricated security events. This allows attackers to create convincing, yet non-existent, threats—such as a ransomware attack on a phantom "payment server"—effectively creating a "false flag" operation to distract blue teams.
4. Hiding Real Attack Activities: Beyond creating fake incidents, the technique can be used to obscure genuine malicious activity. By generating a high volume of irrelevant or misleading logs, an attacker can create a smokescreen, making it exceedingly difficult for blue teams to pinpoint real threats amidst the noise.
5. Weak Default Certificates: Splunk's default platform certificates were highlighted as a security concern. All Splunk installations reportedly ship with the same root certificate, meaning that if an attacker compromises one instance and extracts this certificate, they could potentially authenticate and communicate with any other Splunk instance relying on the default configuration.
6. Lack of Protected Process Light (PPL) on Windows: A critical vulnerability identified for Splunk on Windows is its failure to implement Protected Process Light (PPL). This Windows security feature prevents unauthorized processes from injecting code or manipulating other protected processes. The absence of PPL makes Splunk processes on Windows vulnerable to tools like Echo Mirage, which can be used to intercept and modify traffic, effectively replicating the log_slapper attack on a Windows host.

Technical Deep Dive

▶ Watch: Introducing Log Slapper: intercepting/modifying SIEM traffic (12:50)

The core of the "SIEM Slam" attack hinges on the ability to perform a Man-in-the-Middle (MITM) attack on the log data stream between a SIEM forwarder and its indexer. The speaker focused on Splunk, illustrating its architectural components: data sources, forwarders (Universal or Heavy), indexers, and search heads. The critical insight was that the default communication between a forwarder and an indexer often occurs in plain text.

The attack methodology, as implemented in the log_slapper tool, involves several key steps:

1. Intercepting Traffic:

• On a Linux host with root privileges (where the Splunk forwarder is installed), the log_slapper tool leverages IP tables to create a Netfilter Queue (NFQUEUE). This queue intercepts all outgoing TCP packets destined for the Splunk indexer.
• For Windows hosts, while IP tables isn't directly applicable, the speaker noted that the lack of Protected Process Light (PPL) in Splunk's Windows implementation makes it susceptible to process injection tools like Echo Mirage. An attacker with system privileges could inject into the Splunk process, intercept its network traffic in plain text, and modify it before it's sent.

1. Parsing and Modification:

• Once packets are intercepted, log_slapper parses the raw TCP payload to extract relevant log data. This includes critical fields such as the executed command, hostname, and source IP address.
• The tool then modifies these fields according to predefined rules. For instance, an attacker could change the original hostname (osgun's machine) and IP address (e.g., ending in .64) to a completely fabricated hostname (TR demo) and IP address (10.30.100.2). Similarly, the actual executed commands are replaced with a sequence of fake commands designed to tell a specific story (e.g., a ransomware infection).
• Crucially, after modification, the tool must correctly fix the TCP packet structure, including recalculating checksums and lengths, to ensure the modified packet is accepted by the indexer as legitimate.

1. Generating Fake Logs with slap_now:

• To make the attack more dynamic and realistic, log_slapper works in conjunction with a companion tool called slap_now.
• slap_now takes two configuration files: an execution file and a host file.
• The execution file contains a sequence of commands an attacker wants the target machine to appear to have executed (e.g., ping osunmalicioussite.com, wget randomR.tar.gz, tar -xvf randomR.tar.gz, cd randomR, ./ransomware). It also includes sleep commands to simulate realistic human interaction.
• The host file defines the original host details (e.g., osgun's machine, 192.168.1.64) and the desired fake host details (e.g., TR demo, 10.30.100.2).
• Instead of directly executing the commands from the execution file, slap_now generates unique hashes for each command (e.g., sudo 580blahblahhash). It then executes these hashed commands on the compromised machine.
• In parallel, log_slapper monitors the NFQUEUE for outgoing traffic. When it identifies a log entry containing one of these specific hashes generated by slap_now, it intercepts it. log_slapper then replaces the hash with the corresponding fake command from the execution file and substitutes the original host details with the fake details from the host file.

1. Re-injection and SIEM Ingestion:

• Once modified, the packet is re-injected into the network stream, destined for the Splunk indexer. The indexer processes this seemingly legitimate log entry, storing it and making it searchable.
• The result is that blue team analysts, viewing the Splunk Web UI, will see log entries originating from the fabricated host (TR demo, 10.30.100.2) executing the fake ransomware commands, while the actual compromised machine (osgun's machine) appears clean or generates unrelated logs.

This sophisticated interplay between slap_now generating "dummy" traffic and log_slapper transforming it in transit allows for the creation of highly convincing, yet entirely synthetic, security incidents that can effectively misdirect and confuse security operations teams. The ability to create brand new logs, not just modify existing ones, further enhances the attacker's capabilities.

Demo / Proof of Concept

▶ Watch: Blue team defenses: SSL/TLS and Splunk vulnerabilities (28:20)

Kultekin provided a live demonstration of log_slapper and slap_now in action, showcasing how to create a fake ransomware incident within a Splunk environment.

The demo began with a view of the Splunk dashboard, displaying legitimate logs from the compromised machine, identified as "osgun's machine" (with an IP address ending in .64). These logs indicated that the machine was active and healthy, setting the baseline for normal operations. The speaker emphasized that blue teams would already be monitoring this machine due to its compromise, necessitating a distraction.

Next, the log_slapper tool was initiated on the compromised Linux host, requiring root privileges. The tool immediately began inspecting real-time outbound TCP traffic from the Splunk Universal Forwarder. Its primary function at this stage was to intercept logs, identify the original hostname and IP address, and replace them with the pre-defined target values from the host file – in this case, changing "osgun's machine" to "TR demo" and the original IP to "10.30.100.2".

Upon searching for "TR demo" in the Splunk UI, an initial query would yield no results. However, shortly after log_slapper began its work, logs originating from "TR demo" started to appear. This was a critical moment: the SIEM was now receiving and displaying logs from a host that, in reality, did not exist on the network. This alone could cause confusion for a blue team.

To create the full fake ransomware scenario, the slap_now tool was then executed on the compromised machine. slap_now took the execution file (containing a sequence of ransomware-related commands with sleep intervals) as input. Instead of directly executing these commands, slap_now ran obfuscated, hashed commands (e.g., sudo 580blahblah) on the compromised host. These dummy commands generated real, albeit nonsensical, log traffic.

As these hashed commands were generated and sent by the Splunk forwarder, log_slapper intercepted them in the NFQUEUE. It recognized the specific hashes, replaced them with the actual, malicious-looking commands from the execution file, and injected them back into the data stream, all while maintaining the fabricated "TR demo" hostname and "10.30.100.2" IP address.

The final result, as displayed in the Splunk Web UI, was a clear and detailed log sequence attributed to "TR demo" (10.30.100.2). These logs showed the machine "pinging osunmalicioussite.com," followed by "downloading randomR.tar.gz," then "decompressing it," and finally "executing ./ransomware." The audit logs appeared entirely legitimate, complete with proper timestamps and source details, yet every piece of information—the host, its IP, and the actions performed—was entirely fabricated by the attacker. This compelling demonstration underscored the ease with which a sophisticated attacker could create a highly convincing, yet completely false, security incident to divert blue team resources.

Defensive Implications

▶ Watch: Q&A: Why create noise instead of stealth? (36:00)

The "SIEM Slam" attack vector highlights a severe vulnerability in how many organizations configure and trust their SIEM solutions. Kultekin's core message to defenders is unequivocal: "Never trust logs and never trust your SIEM" in their default state. To combat this, several hardening options and recommendations for both blue teams and SIEM creators were proposed.

1. Encrypt Log Traffic (SSL/TLS):

• The most straightforward and critical defense is to encrypt the communication between forwarders and indexers using SSL/TLS. This would prevent attackers from intercepting and modifying log data in plain text.
• However, Kultekin acknowledged the challenges: managing certificates (issuance, rotation, expiration) can be complex and resource-intensive, especially in large environments. There's also a potential for performance degradation or delays in log ingestion due to encryption/decryption overhead, which can be a concern for high-volume SIEM deployments.
• Crucially, Splunk's default platform certificates were noted as a weakness, as they reportedly use the same root certificate across all installations. This means if an attacker gains access to this default root certificate from one instance, they could potentially authenticate and communicate with any other Splunk instance relying on these defaults. Organizations must deploy their own, unique, properly managed certificates.

1. Operating System Hardening:

• Windows PPL (Protected Process Light): A significant finding was that Splunk processes on Windows do not utilize PPL. This makes them vulnerable to process injection techniques (e.g., with Echo Mirage) that allow an attacker with system privileges to inspect and modify their network traffic. Kultekin strongly urged SIEM creators to implement PPL for their processes on Windows to prevent such tampering.
• Linux IP Tables: While IP tables were used by the attacker in the demo, defenders can also leverage them. Blue teams should monitor for unusual IP table rules, particularly NFQUEUE configurations, which could indicate an attacker setting up an interception point. Regular integrity checks of critical system configurations are vital.

1. SIEM Creator Responsibilities:

• Checksum Mechanisms: SIEM vendors should implement checksums or cryptographic signatures for log entries during transmission. This would allow the indexer to verify the integrity of incoming logs and detect any unauthorized modifications.
• Modify Default Settings: Vendors should re-evaluate default configurations, particularly regarding plain text communication. Secure-by-default should be the standard, requiring explicit action to downgrade security rather than upgrade it.
• Integrate OS-Level Protections: As mentioned, adopting OS-specific security features like PPL on Windows is crucial for protecting SIEM agents from local compromise.

1. Blue Team Best Practices:

• Skepticism and Anomaly Detection: Blue teams must cultivate a healthy skepticism towards log data, especially when anomalies arise. If a critical incident suddenly appears on a seemingly unrelated or non-existent host, it warrants deeper investigation beyond just trusting the log entry.
• Out-of-Band Verification: For high-priority alerts, blue teams should establish procedures for out-of-band verification. This could involve cross-referencing logs with endpoint detection and response (EDR) telemetry, network traffic analysis (NDR), or directly inspecting the alleged compromised host.
• Threat Hunting: Proactive threat hunting for indicators of log manipulation (e.g., suspicious IP table rules, unexpected process injections into SIEM agents, unusual certificate activity, or sudden shifts in log volume/source) is essential.

In essence, while encryption is the primary technical mitigation, a multi-layered defense combining technical controls with enhanced blue team vigilance and a critical re-evaluation of SIEM trust assumptions is necessary to counter the "SIEM Slam" threat.

Key Takeaways

• SIEMs are Vulnerable to Log Manipulation: Modern SIEMs, particularly Splunk in default configurations, are susceptible to having their log data intercepted, modified, and injected with fake information due to plain text communication between forwarders and indexers.
• Plain Text Communication is a Critical Flaw: The default use of unencrypted TCP traffic for log transmission is a significant security weakness that allows attackers with root privileges on a compromised host to perform Man-in-the-Middle attacks on the log stream.
• Fabricated Incidents Distract and Deceive: Attackers can leverage tools like log_slapper to create highly convincing, yet entirely fictitious, security incidents (e.g., ransomware on a non-existent server) to misdirect blue teams, buy themselves dwell time, and hide their true objectives.
• Encryption and Custom Certificates are Paramount: Implementing SSL/TLS encryption for all SIEM log traffic, using unique, properly managed custom certificates (not default vendor certificates), is the most effective technical countermeasure against this attack.
• SIEM Vendors Must Enhance Agent Security: SIEM creators should prioritize implementing stronger process protections (e.g., PPL on Windows) and integrate checksums or cryptographic signatures for log integrity verification to prevent local tampering of their agents.
• Blue Teams Need Healthy Skepticism: Defenders must adopt a mindset of skepticism towards log data, especially during high-alert situations. Out-of-band verification and cross-referencing with other security tools are crucial to validate the authenticity of reported incidents and prevent falling victim to sophisticated log manipulation.

About the Speaker(s)

Ozgun Kultekin is an Offensive Security Engineer at Trend Micro Group, bringing a fresh perspective to the field at 23 years old. His professional focus spans a broad range of offensive security disciplines, including penetration testing across diverse assets like web applications, Active Directory, and SCADA systems. Kultekin is also deeply involved in breach and attack simulation, where he plans and executes red team operations and collaborates closely with blue teams in purple teaming exercises. His expertise extends to developing evasion and bypass techniques for various security technologies such as Web Application Firewalls (WAF), Endpoint Detection and Response (EDR), Network Detection and Response (NDR), Data Loss Prevention (DLP), and email security solutions. Kultekin is also contributing to a project aimed at integrating attack simulation and manual purple teaming activities into a MITRE ATT&CK framework heatmap, potentially as an open-source tool in the future.

Asil Veral is also credited as a speaker for this talk.