Training Specialist Models: Automating Malware Development

Training Specialist Models: Automating Malware Development

Speaker: Kyle, Offensive Security Researcher and Developer, Outflank (a Fortra division)

Conference: Black Hat USA 2025 — August 6-7, 2025, Mandalay Bay, Las Vegas

YouTube: https://www.youtube.com/watch?v=WKmEzRJZ6H4

Reading time: ~9 minutes

Type: Briefing

TL;DR

Outflank researcher Kyle trained a custom 7-billion-parameter LLM called Dante — built on Qwen 2.5 Coder and fine-tuned via supervised fine-tuning plus Reinforcement Learning with Verifiable Rewards (RLVR) — to generate functional, Microsoft Defender for Endpoint-evading shellcode loaders entirely through trial-and-error reinforcement learning, without being shown any working examples of evasive code. At a cost of roughly $1,350 in cloud GPU time, Dante outperforms DeepSeek R1 and rivals GPT-4o on this specific offensive task while refusing the request far less often.

Introduction

Large language models have become standard tools across software development, but their use in offensive security — specifically for automating the creation of evasive malware — has been limited by three compounding problems: privacy concerns when submitting proprietary research to third-party APIs, cost at scale for token-heavy automated workflows, and safety refusals that make models unreliable for red team automation. General-purpose models are trained to decline malware-related requests, and that tendency is increasing across the frontier model landscape.

Kyle's response was to train his own specialist model from scratch — or rather, from an open-source base. Drawing on recent advances in reinforcement learning for language models (particularly techniques pioneered by DeepSeek's R1), he built a training pipeline that teaches a model to write shellcode loaders by running them, testing whether they evade EDR, and rewarding the model when they do. The result is a model that learns the correct reasoning for producing evasive code entirely through trial and error, never having been shown a finished working example.

Why Generalist Models Fall Short for Offensive Security

Kyle proposed a simple conceptual framework to explain the gap between general-purpose and domain-specialist models: the quality of a model's performance on any given skill is roughly proportional to its parameter count divided by the number of skills represented in its training data. A large model trained on a broad dataset may perform adequately on many tasks, but concentrating the same parameters on a narrow domain yields much higher quality for that domain.

▶ Watch: Model Size vs. Skill Quality Framework (04:00)

This explains why frontier models like ChatGPT, Claude, and GPT-4o — despite their general coding competence — are poor fits for red team automation. Beyond the quality ceiling, they have explicit safety training that makes them refuse to write shellcode loaders, and that training is not trending toward more permissive behavior for offensive use cases. Local open-source models (Llama, Qwen base models runnable on consumer GPUs) avoid the refusal problem but lack the parameter count to produce reliable, technically sophisticated output.

The solution Kyle pursued: train a small, focused model on a narrow skill set using reinforcement learning so that it learns through feedback from an automated testing environment rather than from pre-curated human-written examples.

LLM Training Fundamentals: From Pre-Training to RLVR

Kyle provided a practical overview of modern LLM training stages to contextualize his approach.

Pre-training teaches a base model to predict the next token using massive internet text corpora. The resulting base model is a completion engine, not a chatbot — it continues text, but it does not answer questions or follow instructions.

Supervised Fine-Tuning (SFT) converts the base model into an assistant by training it on structured conversation examples with special tokens marking roles (system, user, assistant). SFT teaches the model to behave in a turn-based chat format and to emit an end-of-sequence token when it finishes responding.

Reinforcement Learning from Human Feedback (RLHF) further refines the model using human preference data. In the standard PPO implementation, human raters compare pairs of model outputs, and a separate reward model is trained to predict human preferences. The LLM is then trained against the reward model's scores.

▶ Watch: RLHF and RLVR Explained (08:00)

Reinforcement Learning with Verifiable Rewards (RLVR), introduced by DeepSeek in their February 2024 GRPO paper and popularized by DeepSeek-R1 in January 2025, replaces the reward model with a programmatic verifier. Instead of estimating a human preference score, the verifier checks whether the model's output is objectively correct — for example, whether a piece of code passes unit tests or produces the right answer to a math problem. This eliminates the need for human raters once the initial SFT dataset is built, making it dramatically more scalable. DeepSeek-R1 demonstrated that this approach, combined with chain-of-thought prompting, could produce reasoning capabilities comparable to OpenAI's o1 — an open-source model at a fraction of the training cost.

Why Malware Development Fits RLVR

Kyle applied Jason Wei's "Verifier's Law" framework (published by the OpenAI researcher in a blog post shortly before the talk) to evaluate whether malware development is a suitable RLVR training task. A good RLVR task should have an objective truth, be fast and scalable to verify, have low noise between training and production environments, and offer a continuous reward signal.

▶ Watch: Applying Verifier's Law to Malware (20:03)

Malware development checks every box. The objective truth is clear: fewer and lower-severity EDR alerts is always better. Verification is fast — no human review needed, just submit to a sandbox and check the API. Each sample is independent and can be evaluated in parallel across cloud infrastructure. The training environment can closely replicate a real target by installing the actual EDR product in a fresh VM. And the reward signal can be made continuous: with ten unit tests, five passes yields a 50% reward, enabling incremental learning rather than sparse binary feedback.

Building Dante: Architecture, Training Pipeline, and Results

Model and I/O design: Dante is based on Qwen 2.5 Coder 7B, chosen for its strong coding performance at a size that fits on accessible hardware. The model receives a structured prompt listing desired shellcode loader features (e.g., EXE format, EarlyBird injection, domain-join check) and outputs a structured XML response containing a Python encoding/encryption script, C/C++ source code for the loader, a Makefile, and a compile command. This structured format allows automated parsing and compilation into a final PE binary.

▶ Watch: Verifier Environment Design (24:03)

Training pipeline: The verifier environment creates a fresh sandbox VM, installs Microsoft Defender for Endpoint, domain-joins it, and detonates the compiled PE. A Cobalt Strike Team Server checks for beacon callbacks (confirming functionality), and the MDE API checks for alerts by hostname. Each training run follows two stages:

1. SFT stage: 13 hours on 8 H100 GPUs (~$250). Training data came from DeepSeek-R1 outputs — notably, the SFT dataset intentionally excluded working evasive examples to see if RLVR alone could teach evasion.
2. RLVR stage: 56 hours on 8 H100 GPUs (~$1,100). Total training cost: approximately $1,350.

Results: With a single prompt (pass@1), Dante produces a functional loader — one that successfully beacons to the Team Server — just over 15% of the time, and a fully evasive loader (zero MDE alerts, functional callback) just over 8% of the time. With pass@8 (sending the same prompt 8 times and taking the best result), functionality approaches 50% and full evasion improves substantially. Compilation success with pass@8 approaches 100%.

▶ Watch: Dante vs. Frontier Model Benchmarks (28:03)

Compared to frontier models, Dante at 7B parameters outperforms DeepSeek R1 (671B parameters — approximately 100 times larger) on evasive loader generation. GPT-4o is competitive on compilation rate but falls off sharply on evasion, in part because it refuses the task more frequently than Dante does. The model learned evasion techniques it was never explicitly shown — a consequence of RLVR allowing the model to explore and self-discover effective reasoning patterns.

Notable Quotes

"Many models are trained where they won't write malware or won't help with red teaming, and so they're especially not reliable for automated use cases when you have to rely on those kinds of techniques." — Kyle, 02:00

"The case study of writing evasive shellcode loaders is not the only way that you could implement something like this. There's lots of different types of problems that this exact methodology could apply to quite well." — Kyle, 00:00

"It learns in this kind of trial and error where it just tries to write a shellcode loader. It nearly always fails, but very occasionally it gets it right, and we go back and reward not only that output, but the chain of thought that led to that output." — Kyle, 30:04

"Low-cost models can absolutely outperform large generalists. Dante is one percent the size of DeepSeek R1, and DeepSeek is nearly incapable of this task." — Kyle, 30:04

Key Takeaways

• RLVR makes domain-specialist LLM training accessible. For $1,350 in GPU compute, a security researcher with no formal ML background trained a 7B-parameter model that outperforms a 671B frontier model on a specific offensive task. The methodology is reproducible and the cost is continuing to fall.
• The barrier to AI-assisted malware development is dropping fast. Dante demonstrates that automated EDR evasion research no longer requires human-written datasets of evasive code — the model can discover effective techniques autonomously through reward feedback.
• Refusal policies are not a reliable defense. Commercial models are training to refuse offensive security tasks more aggressively over time, but custom-trained models on open-source bases have no such constraints. Defenders cannot assume LLM safety training will be a meaningful bottleneck.
• Any verifiable task is a candidate for RLVR. The methodology generalizes beyond shellcode loaders to any offensive or defensive problem with an objective truth, automated verification, and parallelizable evaluation — including vulnerability discovery, payload encoding, and detection rule evasion.
• Defenders should use AI-generated loaders for detection engineering. The same pipeline that generates evasive samples can generate large volumes of diverse malware variants for training and testing EDR detection logic — a significant defensive opportunity.

Slides

No slides PDF was available for this talk. Kyle's model and training code are available via the QR code shared at the end of the presentation; he can be reached on X (formerly Twitter) or at the Fortra booth.