Enhancing Command Line Classification with Benign Anomalous Data

Enhancing Command Line Classification with Benign Anomalous Data

Speakers: Ben Gelman, Data Scientist, Sophos; Sean Bruzman, Data Scientist, Sophos

Conference: Black Hat USA 2025 — August 6-7, 2025, Mandalay Bay, Las Vegas

YouTube: https://www.youtube.com/watch?v=om5x9aFrnLE

Reading time: ~8 minutes

Type: Briefing

TL;DR

Sophos data scientists Ben Gelman and Sean Bruzman show that anomaly detection — long dismissed as too noisy for production security use — excels at finding one specific thing: rare benign commands that supervised classifiers have never seen. By pairing multi-algorithm anomaly detection with OpenAI o3-mini as an automated labeler, the researchers built a self-updating pipeline that sources labeled benign data from the long tail of command distributions at scale, improving XGBoost classifier AUC from 0.61 to 0.89 on the hardest production edge cases.

Introduction

Every security operations team running a command line classifier faces the same unglamorous problem: the model slowly degrades. Edge cases accumulate. Retraining requires fresh labeled data. Analysts — already stretched thin — end up spending hours labeling command lines that the model has started misclassifying.

At Black Hat USA 2025, Ben Gelman and Sean Bruzman from Sophos described how they tried to solve this with an automated pipeline, discovered that their initial approach backfired in an instructive way, and ended up building something more useful than what they originally planned. The talk's full title — "Anomaly Detection Betrayed Us, So We Gave It a New Job" — captures the arc exactly.

The result is a practical, modular pipeline that any team with a large command line dataset, a cybersecurity model, and access to a small reasoning LLM can deploy to continuously augment training data without touching analyst time.

▶ Watch: Introduction and Problem Setup (00:00)

The Original Plan and Why It Failed

The Sophos team started with a hypothesis: use anomaly detection to automatically find malicious command lines, then confirm the labels with a large language model, and feed those labels back into the classifier. A fully automated malicious label factory.

The system failed immediately. Anomaly detection finds anomalies — things that deviate from the distribution — not malicious things specifically. When the team sent those anomalies to an LLM for labeling, the LLM confidently labeled many of them as malicious, but the labeled examples turned out to be benign. Sysadmin commands, unusual startup scripts, legitimate but rare administrative operations — all of them looked "weird" by anomaly detection standards, and the LLM agreed they were suspicious.

The result was catastrophically low malicious precision. Injecting these mislabeled examples into training made the classifier worse at the one job it existed to do.

But the team noticed something unexpected: when the LLM labeled anomalous commands as benign, it was almost always right. The model was excellent at recognizing the absence of malicious intent, even in commands it had never seen before.

▶ Watch: The Failed Hypothesis and the Unexpected Discovery (02:00)

The Real Problem: The Benign Long Tail

Understanding why this discovery matters requires understanding how supervised command line classifiers fail in production. The distribution of benign commands follows a long-tail curve: common operations like cd, ls, and basic shell commands appear constantly and are well-represented in any training set. But rare benign operations — the kind a system administrator runs on a Tuesday morning during a maintenance window, or a legitimate automation framework might execute once a week — appear so infrequently that training datasets built from "clean" systems barely cover them.

The standard approach to sourcing benign labels — labeling commands from machines or organizations with no malicious activity over a 30-day period — captures the head of the distribution well but misses the long tail almost entirely. The consequence: the classifier, seeing a rare but legitimate command in production, has nothing to compare it to and frequently misfires with a false positive. At scale (Sophos processes approximately 50 million command lines per day), even a 2% false positive rate means 1 million false positives per day — enough to incapacitate a SOC.

The benign long tail is exactly where anomaly detection excels. Anomaly detection is unsupervised, scales to any dataset size, and by definition finds things that are rare and unusual — which is precisely what the long tail contains.

▶ Watch: The Benign Long Tail Problem (08:00)

The Pipeline: Three Algorithms, One LLM, One Classifier

The production pipeline Gelman and Bruzman built operates at two scales simultaneously.

Full scale (50 million commands/day): Expert-engineered features — obfuscation markers, character statistics, entropy calculations, case-switching indicators — feed an Isolation Forest anomaly detector. Isolation Forest works by counting the average number of random splits required to isolate a data point; points that isolate in fewer splits are more anomalous. It is designed for anomaly detection and handles the full dataset size efficiently on CPU hardware.

Reduced scale (4 million commands/day sample): Instead of hand-engineered features, this branch uses GinaEmbeddings V2 — a transformer encoder model pre-trained on code with PowerShell support — to convert command lines into semantic vectors. Two modified algorithms then identify anomalies: K-means clustering (anomalies are points far from any cluster center, using Euclidean distance) and PCA (anomalies are points with high reconstruction error — those that lose the most information when dimensionality is reduced). These three algorithms are fundamentally different from each other, which produces diverse anomaly samples rather than redundant ones.

After anomaly detection, both pipelines converge at a deduplication step. Exact duplicates are removed by string matching; near-duplicates are removed by running the candidates through the embedding model and filtering out any pair with cosine similarity above 0.8. This dramatically reduces the number of commands that need LLM review.

The surviving anomalies go to OpenAI o3-mini for labeling. The prompt uses rudimentary chain-of-thought: the model must explain what the command does, describe ways it could be benign, describe ways it could be malicious, then return a verdict. The LLM's false negative rate for malicious commands (labeling malicious as benign) is near zero — which is exactly the property needed for this use case, since incorrectly labeling benign commands as benign is not harmful, and the goal is only to source benign labels.

Confirmed benign anomalies are added to the training set for an XGBoost binary classifier, which is retrained regularly with the augmented data.

▶ Watch: Full Pipeline Overview and Algorithm Details (10:00)

LLM Labeling: Replacing Analyst Toil at Scale

The o3-mini labeling step deserves particular attention because it is the component that makes the pipeline viable. Without it, the bottleneck is the same one that motivated the entire project: analyst time.

After deduplication, the anomaly set is still in the hundreds to thousands of commands per day — too many to ask analysts to label regularly, not too many for an LLM to handle cheaply. In the demonstration, Bruzman shows an example command (a complex Wildfly server startup script flagged as anomalous by the anomaly detectors) and walks through o3-mini's labeling rationale: the model identifies it as a well-configured, controlled startup script, notes that an adversary could theoretically use a similar structure to mask behavior, but ultimately labels it benign — correctly.

The cost of running o3-mini on the deduplicated anomaly set is low enough that it does not appear in the team's cost analysis as a limiting factor. The LLM is described as "cheap, fast, and effective" for this specific task.

▶ Watch: o3-mini LLM Labeling in Practice (26:02)

Results: AUC Improvements on Production Edge Cases

The team evaluated performance across two test sets and two baselines.

Baselines:

• Aggregated Baseline (AB): Labels sourced from sandboxes, customer investigations, telemetry, and regex rules — a mature labeling pipeline.
• Regex Baseline (RB): Labels sourced only from static regex rules — a rudimentary starting point.

Test sets:

• Time split test set: Commands occurring after the training period — the easier evaluation.
• Manual labels test set: Commands that broke previous production models — the harder evaluation, targeting real edge cases.

Results on the hard test set are the headline:

• Aggregated Baseline → Full Scale anomaly augmentation: 0.61 → 0.89 AUC
• Regex Baseline → Full Scale anomaly augmentation: 0.70 → 0.76 AUC

All three anomaly methods (Isolation Forest, K-means, PCA) beat both baselines on both test sets. Performance improvements saturated quickly — most gains appeared within the first few days of augmentation. On the easy test set (time split), all methods met or exceeded baseline performance, confirming that adding diverse benign anomaly data does not hurt performance on the original distribution.

The full-scale method showed a particular advantage over the reduced-scale methods when the starting baseline is rudimentary (the regex baseline), suggesting that teams with simpler labeling infrastructure should start there.

▶ Watch: Evaluation Results and AUC Improvements (28:02)

Notable Quotes

"Anomaly detection finds weird stuff by definition, and that weird stuff caused the LLM to claim benign weirdness as malicious weirdness, which absolutely tanks the malicious precision." — Ben Gelman ▶ 04:00

"For a hundred million commands — a scale easily reached by many large or even medium companies — a two percent false positive rate means two million false positives. If a SOC has two million FPs, I'm gonna be hearing about that within the hour." — Sean Bruzman ▶ 06:00

"You can swap out each step of the pipeline: the data, the features, the anomaly detection, the LLM, and even the classifier you're trying to improve. You can change the methods like tires, but the laps of the race are the same." — Ben Gelman ▶ 14:01

"LLMs are the key point that underpins our whole process, just like this singular Jenga block supports the entire tower." — Sean Bruzman ▶ 26:02

Key Takeaways

• Anomaly detection's real value is finding rare benign data, not rare malicious data. Reframing anomaly detection's role from threat detector to benign data sourcer unlocks a scalable, unsupervised path to training set improvement.
• LLMs enable automated benign labeling pipelines that were not previously feasible. o3-mini's near-zero false negative rate on malicious commands makes it a reliable gatekeeper for confirming benign labels without analyst involvement.
• Diverse anomaly algorithms are better than one. Isolation Forest, K-means, and PCA find different kinds of anomalies due to their fundamentally different mechanisms; combining them yields broader coverage of the benign long tail.
• The pipeline is modular and generalizable. Any team with a large data stream and a binary cybersecurity classifier can implement the approach, swapping components as needed.
• Monday morning action plan: Pick a model that needs updating, get recent data, run Isolation Forest, send anomalies to a small reasoning LLM, have an analyst spot-check the benign labels, and retrain. That is enough to start.

Slides PDF: Not available at time of publication.