Enhancing Command Line Classification with Benign Anomalous Data
Black Hat USA 2025 · Day 1 · Briefings
Sophos data scientists Ben Gelman and Sean Bruzman show that anomaly detection — long dismissed as too noisy for production security use — excels at finding one specific thing: rare benign commands that supervised classifiers have never seen. By pairing multi-algorithm anomaly detection with OpenAI o3-mini as an automated labeler, the researchers built a self-updating pipeline that sources labeled benign data from the long tail of command distributions at scale, improving XGBoost classifier AUC from 0.61 to 0.89 on the hardest production edge cases. ---
AI review
Sophos found that anomaly detection's real value in command-line classification is sourcing labeled benign data, not finding malicious data — a reframing that improved XGBoost AUC from 0.61 to 0.89 on production edge cases. The failed hypothesis section is more valuable than most success stories, and the o3-mini labeling pipeline is immediately deployable. Honest, practical, and actually solves a problem that every production SOC team faces.