Beyond Raw Bytes: Towards Large Malware Language Models

Beyond Raw Bytes: Towards Large Malware Language Models

Speakers: Luke Kurlandski

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=XjZozJdvbZo

Overview

Can the foundation model paradigm that has transformed natural language processing be adapted for malware analysis? This talk investigates the feasibility of training Large Malware Language Models (LMLMs) -- the malware analog to LLMs -- by adapting the core pillars of modern deep learning: large-scale data, efficient architectures, self-supervised pre-training, and task-specific fine-tuning. The research systematically explores three code representations (raw bytes, disassembly, and decompiled code via Ghidra), two efficient neural architectures (HRRformer and Mamba), and two pre-training objectives (masked and causal language modeling) across three downstream malware analysis tasks.

The key finding is that decompiled code representations combined with bidirectional Mamba and pre-training constitute the most effective strategy, and that NLP tokenization techniques can reduce PE binary sequence lengths to feasible ranges (median ~32,000 tokens) for training large models -- but only when applied to code-only representations, not entire binaries. The work establishes that pre-trained models consistently outperform models trained from scratch, validating the foundation model approach for the malware domain.

Background

▶ Watch: Two approaches to static malware classification: handcrafted vs raw bytes (0:00)

Static malware classification using machine learning has traditionally followed two approaches. The first uses handcrafted feature sets extracting salient information from PE binaries such as API imports, section entropy, and header metadata. This approach dominates both industry and academia due to its effectiveness and computational efficiency. The second treats PE binaries as raw byte sequences -- 256-value tokens processed by specialized neural networks -- offering a fully end-to-end approach without human biases but at significantly higher computational cost and typically lower performance.

Meanwhile, the broader deep learning field has been transformed by foundation models that learn from raw data (pixels for images, words for text) through massive-scale pre-training followed by task-specific fine-tuning. These models explicitly avoid handcrafted features. The natural question is whether this paradigm can be transferred to malware analysis, creating a foundation model that learns directly from malware binaries.

The challenge is significant: PE files can span several megabytes, standard transformer architectures exhibit quadratic memory and compute complexity making long sequence training impractical, and the entire concept of pre-training and fine-tuning is largely unexplored in the malware domain. The research addresses each of these challenges systematically.

Key Findings

▶ Watch: Tokenization works for code representations but not whole binaries (4:00)

• Decompiled code outperforms raw bytes and disassembly: Higher-level code representations are significantly easier for models to learn from. Decompiled C-like code produced by Ghidra yields the best results across all three downstream tasks, suggesting that lifting binaries to higher abstraction levels preserves the information needed for classification while making it more learnable.
• Tokenization works for code, not for whole binaries: NLP tokenization techniques (BPE and similar) dramatically reduce sequence lengths for code-only representations to a median of approximately 32,000 tokens, making large model training feasible. However, tokenization provides no meaningful compression when applied to entire PE binaries, an important negative finding.
• Pre-training consistently helps: Models pre-trained with language modeling objectives (both masked and causal) outperform models trained from scratch across all three downstream tasks, validating the transfer learning approach for malware.
• Mamba outperforms linear transformers: The modern RNN architecture Mamba (specifically bidirectional Mamba) consistently outperforms the linear transformer HRRformer in this domain, both in pre-training loss and downstream task performance.
• Deeper is better than wider for long sequences: Mathematical analysis shows that when training models on very long sequences from scratch, it is more memory-efficient to make networks deeper rather than wider, contrary to the typical depth-to-width ratios used in standard LLMs.
• Massive code redundancy in malware datasets: The researchers discovered tremendous redundancy where different malware samples with different family labels had identical code regions, likely due to code repackaging and redistribution. These redundant samples must be removed to prevent train-test leakage.

Technical Deep Dive

▶ Watch: Pre-training with masked and causal language modeling (8:00)

The research addresses four technical pillars of foundation model construction:

Data Representation: PE binaries are processed in three ways: extracting raw executable bytes from executable sections, disassembling with Ghidra, and decompiling with Ghidra to produce C-like code. The critical finding is that NLP tokenization (BPE) applied to code-only representations compresses sequence lengths dramatically (median ~32K tokens), while the same tokenization applied to entire PE binaries fails to achieve meaningful compression. This is because executable code has learnable statistical patterns similar to natural language, while raw binary headers and data sections do not.

Architecture Design: Both standard transformers (quadratic complexity) and most existing raw-byte malware classifiers are inadequate for long malware sequences. The research evaluates two linear-complexity architectures: HRRformer (a linear transformer using holographic reduced representations) and Mamba (a modern structured state-space model / selective RNN). A key architectural insight is derived from memory analysis: for a fixed GPU memory budget, deeper architectures are more parameter-efficient than wider ones for long-sequence training. The researchers opt for very deep, relatively narrow networks, departing from standard LLM scaling ratios.

Pre-training: Two self-supervised objectives are evaluated: Masked Language Modeling (MLM), which masks individual tokens and trains the model to predict them, and Causal Language Modeling (CLM), which is standard next-token prediction. Both are applied across all three input representations and both architectures. The decompiled representation with Mamba shows the strongest pre-training signal, with lower loss curves indicating the model can more effectively learn patterns from higher-level code.

Fine-tuning for Downstream Tasks: Pre-trained models are fine-tuned for three tasks by replacing the language modeling head with task-specific classification layers and retraining at a lower learning rate:

1. Malware detection: Binary classification (malware vs. benign), using temporally distinct train/test sets
2. Malware family classification: Multi-class classification into malware families
3. Behavioral tagging: Multi-label classification across ~50 behavioral categories

The bidirectional Mamba model with decompiled input and pre-training achieves the best results across tasks. For behavioral tagging (the hardest task with 50 classes), Hamming loss is low but Jaccard index remains modest, reflecting the inherent difficulty of multi-label prediction across a large label space.

Demo / Proof of Concept

▶ Watch: Fine-tuning for detection, family classification, and behavioral tagging (10:00)

The evaluation encompasses systematic comparison across all combinations of three input representations, two architectures, two pre-training objectives, and three downstream tasks. Key benchmarks include:

• Pre-training convergence: Decompiled code with Mamba achieves the lowest pre-training loss, confirming that higher-level representations are more learnable.
• Malware detection: Pre-trained models outperform from-scratch models using temporally distinct datasets to account for concept drift.
• Family classification: Multi-class accuracy improvements from pre-training, with decompiled representations showing the largest gains.
• Behavioral tagging: Multi-label classification across 50 behavioral categories, with pre-trained decompiled Mamba achieving the best Hamming loss.

The Q&A session revealed several important technical discussions: the tradeoff between pre-trained models performing better on out-of-distribution data versus non-LLM models performing better on in-distribution data, the practical motivation of deploying cost-effective models at scale versus using expensive off-the-shelf code reasoning models, and the relationship between code redundancy in malware datasets and classification difficulty.

Defensive Implications

▶ Watch: Q&A: cost-effective deployment vs off-the-shelf code reasoning models (16:00)

The LMLM approach has several practical implications for malware analysis operations:

• Scalable pre-training on unlabeled data: The ability to pre-train on vast quantities of unlabeled PE binaries means defenders can build increasingly powerful classifiers without requiring correspondingly large labeled datasets -- a significant advantage given the labeling bottleneck in malware analysis.
• Transfer learning across tasks: A single pre-trained model can be fine-tuned for multiple downstream tasks (detection, family classification, behavioral tagging), reducing the need to train separate specialized models for each analysis requirement.
• Cost-effective deployment: The choice of linear-complexity architectures (Mamba) over standard transformers, combined with deeper-rather-than-wider scaling, makes deployment more practical in industrial settings where millions of samples must be processed.
• Concept drift resilience: Pre-trained models showed better generalization to out-of-distribution samples, suggesting improved resilience to the concept drift that plagues malware classifiers as threat actors evolve their techniques.
• Decompilation as preprocessing: The finding that decompiled code outperforms raw bytes suggests that investing in decompilation infrastructure (Ghidra pipelines) as a preprocessing step for ML-based malware analysis is worthwhile despite the computational overhead.

Key Takeaways

• Large Malware Language Models are feasible: the foundation model paradigm (pre-train then fine-tune) transfers successfully to malware classification
• Decompiled code via Ghidra is the most effective input representation, outperforming both raw bytes and disassembly across all tasks
• NLP tokenization compresses code-only representations to trainable lengths (~32K tokens median) but fails on whole PE binaries
• Bidirectional Mamba (a linear-complexity modern RNN) outperforms linear transformers for this domain
• For long-sequence training, deeper networks are more memory-efficient than wider ones -- a departure from standard LLM scaling practices
• Pre-trained models consistently outperform from-scratch models, with the largest gains on out-of-distribution samples relevant to concept drift

About the Speaker(s)

Luke Kurlandski is a researcher at the Rochester Institute of Technology working under advisors Yin and Matt. The research was conducted in collaboration with Herel from Ariel University. Kurlandski demonstrated strong command of both the deep learning and malware analysis aspects of the work, engaging confidently with technical questions from the audience including researchers from UCL (Lorenzo Cavalaro), Carnegie Mellon University, Georgia Tech, and Georgetown University.