Lex Sleuther - A Novel Approach to Script Language Detection

Lex Sleuther: A Novel Approach to Script Language Detection

Speaker: Aaron James

Conference: BSidesSF 2025 — April 26-27, 2025, San Francisco

YouTube: Watch the full talk

Reading time: 7 minutes

TL;DR

Script language misidentification is a quiet but costly failure mode in large-scale malware analysis pipelines — at CrowdStrike's processing volume, even a 10% miss rate translates to hundreds of unanalyzed scripts per day. Aaron James built Lex Sleuther, a tool that lexes input files against multiple language grammars simultaneously and uses linear regression to assign language scores, achieving a false negative rate lower than Google's Magika while operating on just six targeted file types.

Introduction

When a malware analyst submits a file to an automated analysis pipeline, they expect to get execution data back. But that outcome depends entirely on the pipeline routing the file to the correct dynamic analysis environment — and that routing decision depends on correctly identifying the file's type. For binary formats, this is a largely solved problem. For scripts, it is not.

Aaron James, a security researcher on CrowdStrike's threat intelligence team, opened his BSidesSF 2025 talk by framing the problem with a number: at CrowdStrike's scale, 2.6 million files are processed every day. Of those, around 8% are text files, and of those, less than 1% are executable scripts. That sounds manageable — until you apply the math. Even at 90% classification accuracy, hundreds of scripts per day fall through unanalyzed. Static analysis and dynamic analysis are tightly interdependent; if the wrong script runtime is selected, the malware never executes, and the failure is nearly invisible. The pipeline sees silence rather than an error. "How you distinguish between an incorrectly identified script and a script that's plain broken is not obvious," James noted.

The tool he built to address this gap is called Lex Sleuther. It takes a file as input, runs it through multiple custom lexers, and produces a scored verdict indicating which of six supported script languages — Python, PowerShell, Bash, HTML, and others — is most likely. The approach is narrow by design, unconventional in its mechanics, and, as James demonstrated, competitive with state-of-the-art tools on the metrics that matter most.

The Landscape of Existing Tools

▶ Watch: Surveying existing tools (04:10)

James briefly surveyed the existing tooling before justifying his own. The canonical standard, libmagic (the Unix file utility), uses a suite of static signatures that excel at identifying binary formats but perform poorly on free-form text. Script identification requires a fundamentally different architecture, and libmagic's false negative rate in that domain makes it unreliable for this purpose.

YARA rules are the next natural candidate — and James dismissed them with characteristic bluntness. The core problem is that you write them yourself and maintain them indefinitely. Rules calibrated for specific samples tend to over-match in the wild, and there is no principled way to tune them systematically for language-level classification. "Using YARA rules to identify script languages is like using a hammer to drive a screw," he said. "You can get surprisingly far with it, but you really should go find that screwdriver."

The screwdriver he pointed to is Guesslang, a deep neural network with a linear classifier used by VS Code that supports over 100 languages. The breadth is the problem: in a context where you care about six languages, supporting a hundred introduces noise and unnecessary classification surface. Guesslang is also resource-intensive and not suitable for constrained environments. An internal CrowdStrike study found it 6% more accurate than the legacy heuristics system — but also 1,000 times slower, a tradeoff the team declined to accept.

Google's Magika, released in early 2024, was a more serious contender. Trained to handle both binary and text formats, it achieves over 99% accuracy in under five milliseconds per file, with a model small enough to fit on a floppy disk. James was candid that Lex Sleuther's development predated Magika's release, and that Magika's existence might have changed the calculus had it arrived sooner. But Magika does have measurable weaknesses, and narrow scope turns out to be a real advantage.

Building Lex Sleuther: Six Lexers and a Linear Regression

▶ Watch: The design insight — lexers not parsers (14:30)

James walked through the design exploration that led to Lex Sleuther's architecture. One early idea — sending every script to the dynamic analyzer and attempting execution in all six runtimes simultaneously — was rejected because tracing six processes and disambiguating which one succeeded creates a harder problem than the original one. A refinement was to run the scripts through six parsers instead of executing them, treating a clean parse as evidence of correct language identification. But writing and maintaining parsers for languages like Microsoft Batch is deeply impractical.

The key insight was simpler: don't parse. Parsing constructs abstract syntax trees. All Lex Sleuther actually needs is to know whether a file contains tokens characteristic of a given language. That reduces the problem to lexing alone — the first stage of compilation — which is far more tractable. Using a Rust lexer generator from crates.io with proc-macro syntax, James wrote six lexers. The result was what he cheerfully described as "I wrote six lexers and stapled them together."

▶ Watch: Token counts and linear algebra (17:45)

The scoring mechanism is deliberately unsophisticated. A file is run through all six lexers, and for each lexer, the counts of each token type are recorded. All token counts — from all six lexers — are concatenated into a single vector. That vector is the input; the output is a six-element score vector, one score per language. The relationship between input and output is modeled as a linear system, and the weights are derived by training on a corpus of pre-classified files using standard linear regression. James acknowledged this amounts to a feed-forward neural network with zero hidden layers: "You can call this machine learning if you want to. I just call it pre-calculus."

Training data was collected with a helper script James wrote called LSD (Lex Sleuther Dataset), which displays files for manual classification using a numpad, allowing roughly a thousand classifications per hour. Once trained, the model is invoked through a CLI designed to closely resemble the GNU file utility, and Python bindings are available as well.

Performance: Where Narrow Scope Becomes a Feature

▶ Watch: Head-to-head accuracy numbers (22:00)

James presented accuracy comparisons that required careful contextualization. In a head-to-head against Magika over all 200 of Magika's supported file types, Lex Sleuther performs poorly — expected, since it only supports six. When the comparison is limited to those six types, the accuracy numbers are competitive. But the more revealing metric is the false negative rate, where Lex Sleuther outperforms both the legacy CrowdStrike system (File ID, built on libmagic and YARA rules) and Magika itself.

"They're both pretty likely to get it right," James explained. "But Lex Sleuther is less likely to get it wrong."

In CrowdStrike's production pipeline, the tool is not used in isolation. The full classification system layers libmagic, YARA rules, Magika, and Lex Sleuther together — with a synthesis selector (James's preferred term for a "1,400-line if-else statement") that determines evaluation order and handles edge cases. The presence of Lex Sleuther in this ensemble brings script sample dynamic analysis efficacy to 97%.

Notable Quotes

"How you distinguish between an incorrectly identified script and a script that's plain broken is not obvious. Fun fact: most malware is broken."

— Aaron James

"You can call this machine learning if you want to. I just call it pre-calculus."

— Aaron James, on Lex Sleuther's linear regression scoring model

"While it is true that perfect can become the enemy of good, it's also a shame to let good enough be the enemy of better. Good enough can sometimes create holes."

— Aaron James

Key Takeaways

• Scale converts small miss rates into large absolute failures. At 2.6 million files per day, a 10% misclassification rate on the subset of executable scripts amounts to hundreds of invisible samples daily — a number worth engineering against.
• Narrowed scope is a structural advantage. General-purpose classifiers must handle hundreds of file types; a tool scoped to six can be tuned to outperform them on those six. Specificity enables optimization that breadth cannot.
• Lexing is sufficient where parsing is impractical. Token counts capture language-characteristic signal without the complexity of building full parsers, making the approach fast, portable, and maintainable.
• Ensemble systems beat single tools. CrowdStrike's production classifier layers multiple tools, each contributing its strengths. No single classifier, however accurate, substitutes for a well-ordered synthesis layer.
• Lex Sleuther is now open source. As of the talk's delivery at BSidesSF 2025, the tool is available publicly and can be installed via cargo install for anyone who wants to apply the same approach to their own classification pipeline.