GPT-like Pre-Training on Unlabeled System Logs for Malware Detection
Dmitrijs Trizna, Luca Demetrio (Assistant Professor · University of Genoa)
TROOPERS23 · Day 2 · Defense & Management · Defense & Management
Overview
This talk, presented by Dmitrijs Trizna and Luca Demetrio at TROOPERS, introduces a novel approach to malware detection by applying advanced natural language processing (NLP) techniques, specifically Transformer models akin to those powering large language models (LLMs) like GPT, to unlabeled system logs. The core premise is that the sequences of events captured in dynamic analysis reports or system logs can be treated as a "language," which these powerful models can learn to understand, enabling more robust and adaptable malware detection.

Key moments
- 0:00 Introduction: GPT-like for malware detection
- 5:00 Limitations of traditional static malware analysis
- 11:00 Defining the 'language' for malware logs
- 17:00 Self-supervised learning: GPT/BERT core methodology
- 23:40 Critical role of domain-specific data cleaning
- 30:00 Results: Effectiveness of unsupervised pre-training
- 34:00 Explainability: Peeking into the model, new IOCs
- 43:50 Q&A: Current industry adoption of these techniques
GPT-like Pre-Training on Unlabeled System Logs for Malware Detection
Speakers: Dmitrijs Trizna, PhD Student, University of Genoa; Luca Demetrio, Assistant Professor, University of Genoa
Conference: TROOPERS
YouTube: https://www.youtube.com/watch?v=XonIBwtelL0
Overview
This talk, presented by Dmitrijs Trizna and Luca Demetrio at TROOPERS, introduces a novel approach to malware detection by applying advanced natural language processing (NLP) techniques, specifically Transformer models akin to those powering large language models (LLMs) like GPT, to unlabeled system logs. The core premise is that the sequences of events captured in dynamic analysis reports or system logs can be treated as a "language," which these powerful models can learn to understand, enabling more robust and adaptable malware detection.
The speakers highlight the significant limitations of traditional static and dynamic analysis methods, particularly their susceptibility to obfuscation, their inability to generalize across different operating systems or attack types (e.g., living-off-the-land), and their heavy reliance on scarce labeled data. By leveraging self-supervised pre-training on vast quantities of readily available unlabeled log data, the proposed methodology aims to overcome these challenges, offering a path toward more efficient, accurate, and explainable behavioral malware detection. This research demonstrates a substantial performance improvement over existing dynamic analysis techniques and opens new avenues for utilizing the massive amounts of log data accumulated by security operations centers (SOCs).
Background
▶ Watch: Introduction: GPT-like for malware detection (0:00)
Traditional malware detection often begins with static analysis of portable executable (PE) files. This involves extracting features such as imported APIs, exported functions, section constants, or byte histograms. These features are then fed into supervised machine learning models, which learn to classify files as malicious or benign based on labeled examples. While fast and efficient for initial triage, static analysis suffers from several critical drawbacks. It is easily thwarted by obfuscation and packing, which alter the file's structure without changing its runtime behavior. Furthermore, it is limited to PE files, failing to detect threats like PowerShell scripts, VBA macros, or living-off-the-land attacks, and cannot provide insights into actual execution behavior. Crucially, if a file is not present on disk (e.g., memory-resident malware), static analysis is impossible, leaving only system logs as a source of forensic evidence.
To address these limitations, dynamic analysis emerged as a more robust approach. Malware samples are executed within controlled environments like virtual machines, sandboxes, or emulators, and their runtime behavior is observed. This yields a chain of events detailing API calls, network connections, file system operations, and registry modifications. Dynamic analysis is effective against obfuscation because the malware must eventually de-obfuscate and execute its malicious payload, revealing its true intent. However, traditional dynamic analysis for machine learning often still relies on hand-engineered features and requires extensive labeled datasets, which are costly and time-consuming to produce. Sandboxes are also resource-intensive, requiring environment resets after each execution.
The researchers address the resource challenge by utilizing Speakeasy, an open-source emulator from Mandiant (now Google Cloud). Speakeasy allows for efficient dynamic analysis of PE files without needing a full virtual machine, generating rich event logs through Python scripting. Using Speakeasy, they collected a dataset of approximately 70,000 malware samples (from seven types like TrickBot, droppers, ransomware) and 7,000 goodware samples. This data forms the "language" they aim to model: sequences of API calls, network activity, file accesses, and registry operations. This "language," however, is not natural language; it's a complex, verbose, and often messy stream of machine-generated events, posing unique challenges for machine learning models.
The talk then delves into the evolution of artificial intelligence, particularly in NLP, to lay the groundwork for their solution. Early deep learning models included Convolutional Neural Networks (CNNs), primarily used for computer vision but adaptable to sequence data, and Recurrent Neural Networks (RNNs), such as Long Short-Term Memory (LSTMs), which were dominant for sequential data processing due to their ability to maintain state over time. However, RNNs struggled with parallelization and long-range dependencies. A paradigm shift occurred in 2017 with Google's "Attention Is All You Need" paper, which introduced the Transformer architecture. Transformers, by relying solely on attention mechanisms—the ability to dynamically weigh the importance of different parts of an input sequence—revolutionized NLP, becoming the backbone of modern LLMs like GPT and BERT. This architecture allows for highly parallelized training and effectively captures complex relationships within long sequences.
Crucially, the success of LLMs is also attributed to self-supervised language modeling. Instead of requiring massive, hand-labeled datasets, these models learn from vast amounts of unlabeled text data (e.g., the entire internet) by performing tasks like predicting the next word (auto-regressive modeling, as in GPT) or filling in masked words within a sentence (masked language modeling, or MLM, as in BERT). This pre-training phase allows the model to develop a deep understanding of language structure and semantics. Following pre-training, a smaller, task-specific fine-tuning phase with labeled data adapts the model for specific applications, such as question answering, sentiment analysis, or, in this case, malware detection. This two-stage approach, particularly the ability to leverage unlabeled data for foundational learning, is central to the proposed malware detection system.
Key Findings
▶ Watch: Defining the 'language' for malware logs (11:00)
The central discovery of this research is that Transformer models, when adapted and pre-trained on large volumes of unlabeled system logs using self-supervised learning, significantly outperform traditional dynamic analysis malware detection methods. This approach not only provides superior detection capabilities but also offers valuable explainability insights.
Specifically, the researchers demonstrated that their Transformer-based model, which they internally refer to as Nebula, achieved a detection rate close to 80% at a 1% false positive rate on their test dataset. This performance substantially surpasses that of leading alternative techniques based on convolutional networks, which detected only approximately 50% of samples under the same false positive constraints. This performance gap underscores the effectiveness of the Transformer architecture and its attention mechanisms in discerning complex malicious patterns within dynamic execution traces.
A crucial finding relates to the power of unlabeled data. By employing Masked Language Modeling (MLM) during a pre-training phase on large quantities of unlabeled dynamic analysis logs, the model's performance dramatically improved. The researchers showed that increasing the size of the unsupervised pre-training data (up to five times larger than the supervised fine-tuning dataset) allowed the model to achieve performance nearly identical to a hypothetical "perfect" baseline where all data was labeled for supervised training. This indicates that organizations can effectively leverage their vast, existing stores of unlabeled system logs (e.g., from SIEMs, EDRs, or cloud logging services) to build highly capable malware detection models, drastically reducing the reliance on costly and scarce labeled datasets.
Furthermore, the research highlights the inherent explainability of Transformer models. Through their attention mechanisms, the model can reveal which specific sequences of events or individual tokens in the execution trace were most influential in its decision-making. During a demonstration, the model's attention activations on a Eurolast Trojan sample pinpointed suspicious elements like the non-standard exodose token (potentially an evasion attempt), the GetTickCount API call (known for anti-debugging), and specific file access patterns (e.g., dropping an INI file, accessing a windows/hash.dll file). This ability to "peek inside" the model provides security analysts with actionable insights, potentially leading to the discovery of novel Indicators of Compromise (IOCs) or a deeper understanding of malware behavior, which traditional black-box machine learning models often lack. An unexplained but highly activated interaction between TlsGetValue and WideCharToMultiByte API calls was also highlighted as a potential area for further reverse engineering investigation.
Technical Deep Dive
▶ Watch: Critical role of domain-specific data cleaning (23:40)
The technical implementation of this GPT-like malware detection system involves several sophisticated steps, starting from data generation to model architecture and training strategies.
The journey begins with data collection, which leverages Speakeasy, an emulation tool from Mandiant (now Google Cloud). Speakeasy takes a PE file and emulates its execution within a Python script, generating detailed dynamic analysis logs without the overhead of a full virtual machine. The researchers used Speakeasy to process a substantial dataset: approximately 70,000 malware samples and 7,000 goodware samples. These logs, representing API calls, network activity, file operations, and registry interactions, form the raw "language" for the model.
A critical and often overlooked step is data cleaning and normalization, where domain knowledge is paramount. Raw logs are extremely noisy and verbose. Without proper filtering, the unique vocabulary size can be enormous (approaching 1 million unique elements in their dataset). The researchers found that omitting this step led to a drastic drop in model performance, as evidenced by their ROC curves. Key cleaning steps included:
- Filtering out arbitrary values: Memory segments, specific hashes, and transient addresses, which are highly variable and do not generalize across different malware samples, were removed.
- Preserving essential behaviors: Only core actions directly related to malware behavior were retained, such as API calls, file access events, registry operations, and network communications.
- Normalization: Generic placeholders were introduced for highly variable but structurally similar data points. For instance, specific file paths like
C:\Windows\System32\some_hash.dllwere normalized toSYSTEM32_PATHfollowed by aHASH_VALUEplaceholder. Similarly, IP addresses were categorized asPUBLIC_IP,PRIVATE_IP, orLOOPBACK_IPinstead of using their literal values. This significantly reduces the vocabulary size (from ~1M to ~20k unique elements, a logarithmic scale reduction) and helps the model generalize across different samples.
Next, the cleaned event sequences undergo feature extraction, mirroring NLP pipelines.
- Tokenization: The continuous stream of log events is broken down into discrete units called tokens. Instead of simple whitespace splitting, the researchers employed Byte Pair Encoding (BPE). BPE is a data-driven tokenization method used in many LLMs; it iteratively merges the most frequent adjacent byte pairs, creating tokens that represent common subwords or, in this context, common sequences of log data (e.g.,
kernel32.dll,RegOpenKeyExA). This method automatically discovers meaningful "words" in the system log language. - Encoding: Each token is then mapped to a numerical representation. Crucially, these are not just arbitrary integers but embeddings. Embeddings are high-dimensional vectors (e.g., 98 numbers) that are dynamically updated during training. A key property of embeddings is that tokens with similar semantic or functional meanings (e.g., two API calls that both read from a file) will be located closer to each other in this high-dimensional space, allowing the model to understand relationships between different events.
The core of the model architecture is a Transformer network. Unlike early Transformer models which could be massive, this research used a more constrained architecture suitable for comparison with other methods: two encoder layers, with eight attention heads in each layer. This relatively small size (around 5-6 million parameters) still demonstrated significant power, especially when compared to models with billions of parameters.
The training strategy is a two-phase process:
- Self-supervised Pre-training: This is the "GPT-like" phase. A large portion of the collected data (70,000 samples from January, up to five times larger than the supervised fine-tuning set) was used without any labels. The model was pre-trained using Masked Language Modeling (MLM), similar to BERT. In MLM, a percentage of tokens in an input sequence are randomly "masked" (hidden), and the model is tasked with predicting the original masked tokens based on their context. This forces the model to learn the underlying structure, patterns, and relationships within the system log language. The choice of MLM over auto-regressive pre-training (like original GPT) was made because MLM is generally more suitable for inference tasks (classification) rather than generative tasks (creating new sequences).
- Supervised Fine-tuning: After pre-training, the model is fine-tuned on a smaller, labeled dataset. For this, the researchers used samples collected in April, ensuring a temporal shift to test the model's robustness against evolving malware. The dataset for fine-tuning was skewed, with approximately 25-30% benign samples and the rest being malware. During this phase, the pre-trained model's learned representations are adapted to the specific task of classifying a given log sequence as malicious or benign.
The model's performance was evaluated using ROC curves, which plot the true positive rate against the false positive rate across various classification thresholds. This rigorous evaluation demonstrated the superior performance of the Transformer-based approach compared to baselines.
Demo / Proof of Concept
▶ Watch: Results: Effectiveness of unsupervised pre-training (30:00)
While the talk did not feature a live software demonstration of the entire detection pipeline, it effectively served as a proof of concept for the model's capabilities, particularly its explainability features. The speakers demonstrated how the internal workings of the Transformer model could be leveraged to gain insights into why a particular detection was made, moving beyond the "black box" nature of many machine learning systems.
Using a random sample of the Eurolast Trojan as an example, the researchers showcased visualizations of the model's attention activations. These visualizations highlighted specific tokens and their relationships within the dynamic analysis log that were most influential in the model's classification decision.
Key observations from this demonstration included:
- Evasion Attempts: The model's attention mechanisms strongly activated on the token
exodose. This is a non-standard string and not typically seen in default system behavior, suggesting it might be an attempt by the malware author to evade detection by using an unusual or misspelled component name, which a human analyst might overlook. - Anti-Debugging Techniques: High attention was also observed for the
GetTickCountAPI call. This API is well-known in the security community for its use in anti-debugging techniques, often based on timing discrepancies. The model's ability to identify this as a critical indicator, even without explicit rules, demonstrates its learned understanding of malicious patterns. - File System Manipulation: The demonstration also showed strong activations around sequences involving the dropping of an
.inifile into a temporary folder, followed by accessing a file with a hash-like name within the Windows folder (windows/hash.dll). This multi-step behavior, involving both file creation and subsequent access, was clearly identified as malicious by the model. The researchers compared these attention insights with existing state-of-the-art explainable AI techniques, noting a strong alignment in highlighting these malicious sequences. - Potential for New IOCs: An intriguing observation was a strong activation between the return value of
TlsGetValueand arguments passed toWideCharToMultiByteAPI calls. While the researchers admitted they couldn't immediately identify the exact malicious context of this interaction, they suggested that for a skilled reverse engineer, this could serve as a valuable hint, potentially leading to the discovery of new, previously unknown Indicators of Compromise (IOCs).
This demonstration effectively illustrated that Transformer models offer more than just high detection rates; they provide a window into the complex behaviors driving their decisions, empowering defenders with deeper analytical capabilities.
Defensive Implications
▶ Watch: Q&A: Current industry adoption of these techniques (43:50)
The research presented has profound implications for cybersecurity defenders and offers several actionable insights:
- Leveraging Unlabeled Log Data: The most significant implication is the ability to utilize the vast, often untapped, repositories of unlabeled system logs that organizations already collect. Instead of relying solely on scarce and expensive labeled datasets, security teams can now pre-train powerful Transformer models on their own SIEM data, EDR logs, CloudTrail events, Sysmon logs, or AuditD records. This allows for the development of highly customized and context-aware detection models without the prohibitive cost of manual labeling. This "self-supervised" approach can effectively transform a massive data storage problem into a valuable resource for enhanced security.
- Improved Behavioral Malware Detection: The approach enables more robust detection of sophisticated malware that employs obfuscation, packing, or living-off-the-land (LotL) techniques. By focusing on the actual runtime behavior captured in logs, the model bypasses static analysis limitations. This means better protection against threats that don't rely on traditional PE files, such as malicious scripts (PowerShell, VBA), or legitimate tools used for nefarious purposes.
- Enhanced Adaptability and Future-Proofing: Models trained with this methodology can be more easily updated and adapted to new threats. As new malware variants emerge, their execution logs can be continuously fed into the pre-training phase, allowing the model to learn new patterns without requiring human analysts to label each new sample. This continuous learning capability can help security systems stay ahead of evolving attack techniques.
- Actionable Explainability for Threat Hunting: The inherent explainability of Transformer models, through their attention mechanisms, provides a powerful tool for threat hunters and incident responders. When a detection is made, analysts can investigate why the model flagged a particular activity by examining the highlighted log sequences. This can accelerate incident investigations, help confirm false positives, and validate true positives by providing concrete evidence of malicious behavior. It also facilitates the creation of more precise detection rules and Indicators of Compromise (IOCs).
- Discovery of Novel IOCs: As demonstrated, the model's attention can sometimes highlight suspicious interactions or sequences of events that are not immediately obvious to human analysts. This capability could lead to the discovery of entirely new Indicators of Compromise (IOCs) or reveal subtle malicious patterns that might otherwise go unnoticed. This proactive identification of novel threats is a significant advantage.
- Beyond PE-centric Security: The methodology is not limited to PE files. Since it operates on generic system logs, it can be applied to any system that generates timestamped event data, broadening the scope of automated detection across diverse environments, including Windows, Linux, cloud infrastructure, and various application logs.
In essence, this research provides a blueprint for building a new generation of behavioral malware detection systems that are more effective, adaptable, and transparent, fundamentally changing how defenders can leverage their existing data to combat advanced threats.
Key Takeaways
- Transformer models with self-supervised learning are highly effective for malware detection on system logs. By treating event sequences as a "language," these models can uncover complex malicious behaviors that evade traditional methods.
- Unlabeled system logs are a massive, underutilized resource. Organizations can leverage their existing terabytes of log data (SIEMs, EDRs, etc.) for pre-training, significantly reducing reliance on expensive and scarce labeled datasets.
- Domain knowledge is critical for data cleaning and normalization. Filtering arbitrary values and normalizing specific data points (e.g., file paths, IP addresses) are essential steps that drastically improve model performance and generalization.
- Explainability offers new insights for defenders and threat hunters. Transformer attention mechanisms can highlight specific malicious operations, potentially revealing novel Indicators of Compromise (IOCs) and aiding in faster, more informed incident response.
- The approach significantly outperforms traditional dynamic analysis methods. The research demonstrated detection rates close to 80% at a 1% false positive rate, compared to approximately 50% for convolutional network-based competitors.
- This paradigm shifts malware detection towards continuous learning and adaptability. Models can be continually updated with new unlabeled log data, allowing security systems to adapt more rapidly to evolving threats and extending detection capabilities beyond traditional PE-based malware.
About the Speaker(s)
Dmitrijs Trizna is affiliated with Microsoft, though the presented talk is a personal "pet project" undertaken with Luca Demetrio, not directly related to his work at Microsoft. He holds various certifications, including OSCP, and has previously spoken at prominent security conferences such as Defcon AI Village and Black Hat. Dmitrijs is currently pursuing a PhD at the University of Genoa, where Luca Demetrio is based.
Luca Demetrio is an Assistant Professor at the University of Genoa. He is known for his work in adversarial machine learning and maintains a library specifically designed to create adversarial attacks against machine learning detectors, jokingly referring to himself as "the bad guy" in this context. Luca has published research at notable conferences, including NeurIPS, and has contributed to European projects focusing on the stability of applications.
Reviews
Dr. Zero (Offensive Security Researcher) — SOLID
Trizna and Demetrio actually did the engineering. Instead of slapping 'AI' on a standard random forest, they built a Transformer to parse Speakeasy emulator logs using masked language modeling. It's a genuinely clever way to weaponize unlabeled execution data, even if their false positive rate isn't quite production-ready.
Heather Calloway (CISO) — SOLID
This research proves we can extract operational value from the petabytes of unlabeled system logs we already pay to store. It provides a compelling blueprint for using self-supervised Transformer models to build highly accurate, transparent behavioral detections without relying on expensive manual labeling.