SAP (Anti-)Forensics: Detecting White-Collar Cyber-Crime
Yvan Genuer (Senior Security Researcher · Onapsis)
TROOPERS23 · Day 1 · Defense & Management · Defense & Management
Overview
In this insightful talk, "SAP (Anti-)Forensics: Detecting White-Collar Cyber-Crime," Yvan Genuer, a Senior Security Researcher at Onapsis, delves into the critical yet often overlooked domain of forensic investigation within SAP environments. Genuer highlights the immense footprint of SAP systems, which underpin an estimated 90% of the world's business application layer, making them prime targets for sophisticated cyber-attacks, ranging from data espionage and financial fraud to outright sabotage. The presentation serves as a crucial update to a field that has seen little public discourse in the past decade, building upon Genuer’s extensive white paper on the subject.

Key moments
- 2:00 Emphasizing SAP's global business criticality
- 6:00 Introducing the comprehensive SAP forensics white paper
- 18:00 Syslog's importance and critical default size limitation
- 19:00 Attacker's ability to pollute SAP syslog entries
- 25:00 Dynamic manipulation of developer trace levels and content
- 31:00 Monitoring changes on critical SAP business objects
- 41:00 Crucial to backup/export logs; attackers can tamper
- 43:00 SAP rejected syslog pollution as a vulnerability
SAP (Anti-)Forensics: Detecting White-Collar Cyber-Crime
Speakers: Yvan Genuer, Senior Security Researcher, Onapsis
Conference: TROOPERS
YouTube: https://www.youtube.com/watch?v=ZPRojPkFxrk
Overview
In this insightful talk, "SAP (Anti-)Forensics: Detecting White-Collar Cyber-Crime," Yvan Genuer, a Senior Security Researcher at Onapsis, delves into the critical yet often overlooked domain of forensic investigation within SAP environments. Genuer highlights the immense footprint of SAP systems, which underpin an estimated 90% of the world's business application layer, making them prime targets for sophisticated cyber-attacks, ranging from data espionage and financial fraud to outright sabotage. The presentation serves as a crucial update to a field that has seen little public discourse in the past decade, building upon Genuer’s extensive white paper on the subject.
The talk addresses the growing need for specialized SAP forensics capabilities, driven by an increasing number of vulnerabilities and security incidents. Genuer meticulously dissects the complexities inherent in SAP system architecture, which often involves multiple instances, diverse operating systems, and proprietary data formats. He not only enumerates the various data sources available for investigation but also exposes critical anti-forensic techniques attackers can employ to obscure their tracks, emphasizing the necessity for proactive and robust logging configurations. This article summarizes and expands upon Genuer’s findings, providing a comprehensive guide for security professionals tasked with safeguarding these vital business systems.
Background
▶ Watch: Emphasizing SAP's global business criticality (2:00)
SAP systems form the backbone of global commerce, managing critical business processes, financial transactions, and sensitive data for enterprises worldwide. The sheer ubiquity and criticality of SAP mean that a compromise can have catastrophic consequences, including intellectual property theft, financial manipulation, and operational disruption. Despite this high-stakes environment, public knowledge and specialized expertise in SAP forensics have lagged, with the most significant public research dating back nearly a decade. Genuer underscores this gap, noting that previous talks at conferences like TROOPERS, while foundational, are now outdated given the evolution of SAP systems and attack methodologies.
Yvan Genuer brings a wealth of experience to this subject, with a career spanning 20 years in SAP, including a decade dedicated to SAP security research. His background as an SAP Basis Administrator provides a unique perspective on the intricacies of these systems, and his track record includes discovering over 100 SAP vulnerabilities. This talk, notably his first "defensive" presentation, reflects a shift in focus towards empowering defenders. Genuer's research culminated in a comprehensive 60-page white paper, which details 19 different SAP data sources relevant to forensic investigations. He meticulously differentiates between "business investigations" (focused on what an attacker did post-exploitation, e.g., modifying financial data) and "technical investigations" (focused on how an attacker gained access and their technical path), with his work primarily addressing the latter. This distinction highlights the interdisciplinary communication required during incident response, where technical findings can inform business impact and vice-versa.
Key Findings
▶ Watch: Syslog's importance and critical default size limitation (18:00)
Genuer's research uncovers several critical insights into the state of SAP forensics and anti-forensics:
- Complexity of SAP Environments: SAP deployments are rarely monolithic. They typically involve complex, multi-instance architectures spanning various operating systems (Linux, Windows), database types, and SAP technologies (ABAP, Java). This distributed nature means forensic evidence can be scattered across numerous hosts and components, making comprehensive investigation challenging.
- Insufficient Default Logging: A pervasive issue is that default logging configurations in many SAP systems are woefully inadequate for effective forensic analysis. Parameters like the syslog size, which defaults to a mere 10MB, often retain only a few hours of critical activity, leading to significant data loss during incidents.
- Prevalence of Anti-Forensic Techniques: Attackers with sufficient privileges can easily employ anti-forensic measures to hide their tracks. Genuer demonstrated syslog pollution, where arbitrary messages can be injected into the syslog, and highlighted how attackers can dynamically reduce trace levels or remove timestamps from logs, severely impeding investigations.
- Vulnerability in SAP Log Handling: Despite reporting the syslog pollution vulnerability to SAP, it was rejected as not being a security flaw, underscoring a potential disconnect in how certain logging integrity issues are perceived.
- Criticality of Configuration: The effectiveness of any SAP data source for forensics hinges entirely on its configuration. Enabling a log is not enough; its parameters must be appropriately set to capture relevant, detailed, and persistent information.
- Necessity for External Log Export: Given that an attacker with high privileges can manipulate or delete logs directly on the SAP system, exporting logs to an external, secure Security Information and Event Management (SIEM) system or storage is paramount for maintaining evidentiary integrity.
- Specialized Tools and Knowledge Required: While SAP provides some built-in tools for log analysis, understanding the nuances of each data source, its proprietary formats, and its configuration requires specialized SAP security and Basis knowledge.
These findings collectively emphasize that effective SAP forensics demands a proactive, comprehensive strategy that goes beyond default settings and anticipates sophisticated attacker methodologies.
Technical Deep Dive
▶ Watch: Dynamic manipulation of developer trace levels and content (25:00)
Forensic investigation in an SAP environment is a multi-faceted challenge due to the layered and distributed nature of SAP architecture. Genuer meticulously details various data sources, their characteristics, and potential anti-forensic manipulation techniques.
An SAP system typically comprises multiple layers: the network, the operating system (OS), the database (DB), and the business application layer, which is dominated by SAP. A single logical SAP system can consist of several physical hosts, each running one or more SAP instances (e.g., ABAP or Java instances), an SAP agent, and potentially standalone binaries like the SAProuter or Web Dispatcher. Each of these components can expose services and generate logs, meaning an attack trace might be distributed across several servers and log files. For instance, an exploit targeting a specific port (e.g., 3300) might only leave traces on the specific OS host (SAP OS 3 in Genuer's example) running the vulnerable service, necessitating a holistic collection strategy.
Genuer categorizes SAP data sources, detailing their relevance for technical investigations:
- Syslog (System Log):
- Description: The primary system log in SAP, akin to
var/messageson Linux. It captures a broad range of system events, not just security-specific ones. - Configuration: The most critical parameter is
rslg/max_diskspace/local, which defines the maximum size of the syslog file on the file system. By default, this is often set to a mere 10MB. - Anti-Forensics: This small default size means that in a busy production system, the syslog might only retain two hours of history, causing critical data loss. Genuer demonstrated a method to pollute the syslog by injecting arbitrary messages using a high-privileged user, making it possible to fabricate events (e.g., faking a system restart) or obscure genuine attack attempts. While reported, SAP did not classify this as a vulnerability.
- Forensic Value: Filtering by message IDs (e.g.,
US*for user-related events) can quickly narrow down relevant entries like user logins, lockouts, or failed attempts.
- **Developer Traces (DEV_W*):**
- Description: These are log files primarily intended for SAP developers and Basis administrators to debug issues. They are not security-dedicated but can contain invaluable information when something goes wrong.
- Mechanism: Developer traces are typically stored in files named
dev_w*(e.g.,dev_w0,dev_w1). A crucial anti-forensic aspect is their backup mechanism: upon each SAP instance restart, the current log is backed up to a.oldfile. If the system restarts twice after an incident, the original trace information is permanently lost. - Correlation: The syslog often includes a "process number" column. This number directly correlates to a specific
dev_w*file (e.g., process number 153 corresponds todev_w153), allowing investigators to link system-level events to more detailed process traces. - Anti-Forensics:
- Dynamic Trace Level: Attackers with OS command execution privileges can dynamically decrease the trace level using the
tpcommand withUSR1(decrease) orUSR2(increase) signals. Setting the level to 0 effectively disables tracing, making it impossible to log subsequent actions. trace/hide_sectdata: This parameter, by defaultON, replaces sensitive data like passwords with asterisks in the logs. An attacker can set it toOFFto potentially log credentials used in system communications.trace/resolution: Defaulting to1, this parameter controls timestamp resolution. Setting it to0removes timestamps entirely from the log files, severely hindering chronological analysis.
- Security Audit Log (SAL):
- Description: This is SAP's dedicated security log, designed to record security-relevant activities.
- Configuration: While often enabled by default in newer SAP systems (post-2020), its configuration is notoriously complex. Genuer stresses that simply enabling it is insufficient; it must be configured to capture events in the correct clients (e.g., production clients, not just client
000) and for the appropriate event types. A poorly configured SAL is as good as disabled.
- ICM (Internet Communication Manager) and Web Dispatcher:
- Description: The ICM acts as SAP's HTTP web server, handling web-based communications. The Web Dispatcher typically functions as a reverse proxy and load balancer, often exposed in the DMZ to handle incoming HTTP requests from the internet before forwarding them to backend SAP systems.
- Forensic Value: Both components log web-related activities, including HTTP requests, accessed web services, client IPs, and potential brute-force attempts. Genuer points to a list of "well-known URLs" associated with public exploits, emphasizing the need to monitor access to these paths.
- Anti-Forensics/Vulnerabilities: Genuer previously found authentication and Cross-Site Scripting (XSS) vulnerabilities in the Web Dispatcher's admin dashboard (now patched), highlighting that these critical perimeter components can themselves be targets or sources of compromise.
- SAProuter:
- Description: A standalone binary that acts as a software-based proxy for SAP protocols, often serving as an entry point from the internet into an organization's SAP landscape.
- Exposure: Genuer notes that over 5,000 SAProuters are publicly discoverable on the internet, making them significant potential attack vectors. Their logs are crucial for identifying external access attempts and initial compromise.
- Change Documents (CDHDR/CDPOS tables):
- Description: A built-in SAP feature that logs changes to critical business objects, often corresponding to specific database tables.
- Configuration: This feature is "partially" enabled by default, meaning that while some objects might log deletions, they might not log insertions or updates. Over 800 objects are logged by default.
- Forensic Value: Change documents provide highly granular business context, detailing who (user, department), what (object, field, old/new value), when (timestamp), and how (transaction code, e.g.,
SM34) a change occurred. Genuer provides an example where a user inserted five new IBANs usingSM34just before lunch, a detail that could be crucial if the user was on PTO. The issue lies in ensuring comprehensive logging (insert, update, delete) for all critical tables.
- User and Authorization Data (SUIM transaction):
- Description: The SUIM (User Information System) transaction is a powerful tool within SAP for querying and analyzing user accounts, roles, and authorizations.
- Forensic Value: It allows investigators to identify users with excessive privileges (e.g., OS command execution, sensitive table modification), detect anomalous user accounts (e.g., non-standard naming conventions, generic administrator accounts), or track changes to user master data. This is often an initial step in understanding an attacker's potential capabilities or identifying compromised accounts.
- Java System Logs:
- Description: Unlike ABAP systems, Java-based SAP systems centralize most of their logs into a single directory.
- Forensic Value: This centralization significantly simplifies log collection. SAP provides a web-based Log Viewer interface that offers categorized "view names" (e.g., "Developer Traces") and the ability to upload external log files for offline analysis, which is useful when a compromised system needs to be taken offline quickly.
- Anti-Forensics: A high-privileged attacker on any layer (database, SAP, OS) can easily delete the entire centralized log directory, emphasizing the need for immediate external backup.
- SAP Agents:
- Description: Mandatory components present on all SAP servers.
- Vulnerability: Genuer notes a recent increase in vulnerabilities found directly within SAP agents, making them a potential initial entry point for attackers. Their logs are therefore critical for detecting these types of compromises.
Genuer also mentioned a "SAP Log Collector" script, a combination of Bash and PowerShell, designed to automate log collection, calculate checksums, and compress files into a single archive, though it was still under development at the time of the talk.
Demo / Proof of Concept
▶ Watch: Monitoring changes on critical SAP business objects (31:00)
While a full live demonstration of a complex attack chain wasn't the primary focus of this defensive talk, Yvan Genuer effectively showcased key concepts and anti-forensic techniques. He explicitly demonstrated syslog pollution, showing how a high-privileged user could inject arbitrary, fake messages into the SAP system log. This live example underscored his point about the potential for attackers to manipulate logs and sow confusion during an investigation.
Furthermore, Genuer presented the concept of a "SAP Log Collector" script, indicating its functionality for automating the collection of disparate log files across an SAP system. Although the script itself was still "soon" to be available due to GitHub issues, its purpose and design were clearly articulated as a practical tool for forensic responders to quickly gather critical evidence, calculate checksums for integrity, and archive logs. These elements, though not a traditional exploit demo, served as practical proof points for both the vulnerabilities in log integrity and the tools needed for effective response.
Defensive Implications
▶ Watch: SAP rejected syslog pollution as a vulnerability (43:00)
Genuer's talk provides crucial guidance for defenders safeguarding SAP environments. The primary takeaway is that default SAP logging configurations are often inadequate, and proactive, thorough adjustments are essential.
- Enable and Configure Comprehensive Logging:
- Security Audit Log (SAL): Ensure the SAL is not only enabled but meticulously configured to capture all relevant security events across all production clients, not just client
000. This requires understanding the myriad of configuration parameters available. - Syslog: Critically, increase the size of the syslog file (
rslg/max_diskspace/local) well beyond the default 10MB to retain sufficient history for investigations. A few hours of log data is insufficient for complex incidents. - Change Documents: Review and enhance the logging for critical business objects and tables. Ensure that insertions, updates, and deletions are all comprehensively logged where necessary, rather than relying on partial default logging.
- Implement Robust Log Export and Backup:
- External Storage: The most critical defense against anti-forensic log manipulation is to export all SAP logs (syslog, developer traces, SAL, ICM, Web Dispatcher, SAProuter, Java logs) to an external, secure Security Information and Event Management (SIEM) system or dedicated log storage. This ensures evidentiary integrity even if an attacker gains high privileges and attempts to delete or alter logs on the SAP system itself.
- Checksums: When exporting logs, always calculate and store cryptographic checksums to verify their integrity during an investigation.
- Monitor and Respond to Anti-Forensic Indicators:
- Developer Trace Parameters: Actively monitor changes to parameters like
trace/hide_sectdata(to prevent exposure of sensitive data) andtrace/resolution(to ensure timestamps are always present). Alert on any attempts to modify these from their secure defaults. - Dynamic Trace Level Changes: Implement monitoring for OS commands or system signals that could dynamically reduce the developer trace level.
- Syslog Integrity: While SAP did not classify syslog pollution as a vulnerability, defenders should be aware of its possibility and scrutinize syslog entries for anomalies or inconsistencies, especially if high-privilege compromise is suspected.
- Multi-Instance and Component Awareness:
- Distributed Logs: Recognize that logs are distributed across multiple SAP instances, OS hosts, and standalone binaries (SAProuter, Web Dispatcher, Agents). Forensic collection must encompass all potentially relevant components.
- Patch Management: Keep all SAP components, including agents, Web Dispatcher, and SAProuter, fully patched to mitigate vulnerabilities that could serve as initial compromise vectors or allow log manipulation.
- Leverage SAP Tools and Specialized Knowledge:
- Built-in Analysis Tools: Utilize SAP's native tools for log analysis, as they are designed to interpret proprietary log formats.
- Expertise: Invest in specialized SAP security and Basis knowledge within incident response teams to effectively navigate the complexities of SAP environments and interpret diverse log sources.
- Foster Collaboration:
- Technical-Business Integration: Emphasize strong communication channels between technical forensic teams and business stakeholders. Business context can guide technical investigations, and technical findings can help assess business impact.
By implementing these defensive measures, organizations can significantly enhance their ability to detect, investigate, and respond to cyber-incidents within their critical SAP landscapes.
Key Takeaways
- SAP forensics is complex and requires specialized knowledge: Given the layered architecture, diverse technologies (ABAP, Java, various OS/DBs), and proprietary log formats, effective SAP incident response demands expertise beyond generic IT forensics.
- Default logging is often inadequate: Critical log sources like the syslog often have insufficient retention periods (e.g., 10MB default size for syslog, leading to only a few hours of data), and other logs may be partially configured or easily overwritten.
- Attackers employ sophisticated anti-forensic techniques: High-privileged attackers can pollute syslogs, dynamically disable or reduce trace levels, remove timestamps, and delete log files, making investigations extremely challenging.
- External log collection and secure storage are paramount: To preserve evidentiary integrity, all relevant SAP logs must be exported to an external, tamper-proof SIEM or storage solution immediately and continuously.
- Configuration is key to defensive posture: Meticulous configuration of all logging parameters (e.g., Security Audit Log, Change Documents, developer traces) is more critical than mere enablement to ensure comprehensive and useful data capture.
- Holistic approach across instances and components: Investigations must consider all SAP instances, OS hosts, database logs, and standalone binaries like SAProuter and Web Dispatcher, as evidence can be widely distributed.
About the Speaker(s)
Yvan Genuer is a Senior Security Researcher at Onapsis, a company specializing in enterprise resource planning (ERP) security, particularly for SAP systems. With an impressive two decades of experience in the SAP ecosystem, Genuer began his career as an SAP Basis Administrator, providing him with a profound understanding of the operational intricacies of these critical business applications. For the past 10 years, he has focused specifically on SAP security, delving into research that has uncovered more than 100 vulnerabilities within SAP platforms. This presentation at TROOPERS marks a significant milestone for Genuer, as it is his first "defensive" talk, shifting his focus from breaking systems to empowering organizations to protect them.
Reviews
Dr. Zero (Offensive Security Researcher) — SOLID
SAP is a black box that holds the global economy, and most IR teams wouldn't know how to pull a log from it if their lives depended on it. Genuer drops a 20-year Basis admin's worth of knowledge on how attackers actually blind these systems, naming specific parameters and anti-forensic tricks. Strong defensive research that actually matters.
Heather Calloway (CISO) — MUST SEE
This is mandatory material for any enterprise relying on SAP for its financial or operational backbone. The research exposes how structurally deficient default SAP logging is, effectively blinding incident response teams during a breach. If you run SAP, your security engineering and IT audit teams need to review this immediately.