LLMs-Driven Automated YARA Rules Generation with Explainable File Features & DNAHash

LLMs-Driven Automated YARA Rules Generation with Explainable File Features & DNAHash

Speakers: Wang Xiaochen and Liu Yiping, Alibaba Cloud Security (with contributors Wang Xiaoma and Cheng Cong)

Conference: Black Hat USA 2025 — August 6-7, 2025, Mandalay Bay, Las Vegas

YouTube: https://www.youtube.com/watch?v=0i8UhpUgw_0

Reading Time: ~8 minutes

Type: Briefing

TL;DR

Researchers from Alibaba Cloud Security introduced LMD-YARA, a four-stage framework that uses large language models (LLMs) and a novel binary hashing feature called DNAHash to automatically generate high-quality YARA rules from malware samples. Benchmarked against YARAgen and AutoYARA, LMD-YARA achieves higher detection rates and significantly lower false positive rates, while producing interpretable rules that explain why a sample was flagged.

Introduction

The volume and diversity of modern malware outpaces the capacity of human analysts to write detection rules manually. While automated YARA rule generation tools like YARAgen and AutoYARA have been available for years, they face persistent challenges: string-based features match benign files too often, detection coverage across malware families is inconsistent, and the rules themselves offer little insight into why a file matches.

Wang Xiaochen and Liu Yiping from Alibaba Cloud's malware analysis team presented LMD-YARA (LLM-Driven YARA), a system that re-architects the rule generation pipeline around three feature types, LLM-based decision-making, and a novel structural hash feature. The result is rules that are not only more accurate but that carry embedded explanations of malicious behaviors — a property that makes them directly useful for alert triage, not just binary detection.

System Architecture: Four-Stage Pipeline

▶ Watch: LMD-YARA Framework Overview (02:00)

LMD-YARA breaks rule generation into four sequential stages:

1. Feature Extraction

Three feature types are extracted from each malware sample:

• String features — IOC-related strings and natural language strings, filtered through an NLP model to exclude meaningless or trivial strings. Strings must be at least six characters. IOC-related patterns (C2 URLs, registry paths, API names) are prioritized.
• Function features — Decompiled pseudocode is extracted along with address offsets and line numbers that allow mapping back to assembly instructions. A function call graph provides relational context to the LLM. A proprietary function signature algorithm called "Finger" identifies and filters out third-party library code.
• DNAHash features — A novel hashing scheme that computes hash values for different structural segments of a binary file (based on PE or ELF structure fields), producing a set of field-based hash values. This feature is most useful when self-modifying code makes string and function features unreliable.

2. Feature Filtering

A pre-collected database of several million benign samples is used to filter out string and DNAHash features that appear too frequently in clean files. For function features, an importance evaluation algorithm ranks functions by the number of string references they make and their call frequency. The open-source static analysis tool KAPA is applied to mark potentially sensitive functions. The result is a set of "suspicious sensitive functions" suitable for rule generation.

▶ Watch: Function Feature Filtering and LLM Integration (10:01)

3. Feature Decision via LLM

This stage is where LMD-YARA diverges most sharply from earlier tools. The system uses an LLM to:

• Select the most useful string features from the filtered candidates.
• Analyze decompiled pseudocode and assign behavioral labels (e.g., "self-decryption," "process injection," "modify registry") drawn from a standardized taxonomy of malicious behaviors.
• Identify opcode sequences within labeled functions that are suitable for YARA signature generation.

The LLM prompt includes not just pseudocode but also the function's call graph context and KAPA labels for its sub-functions. The model outputs the function label, an explanation, and the line numbers and lengths of code segments suitable for opcode feature extraction.

Experiments found that Qwen3 2032B performs best for identifying function behavior from pseudocode. For environments where response speed or cost is constrained, Qwen3 can be used as a teacher model to generate high-quality training data for fine-tuning smaller models via Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO).

4. Rule Generation

Opcode sequences identified by the LLM are mapped from pseudocode line numbers back to assembly instructions (using the previously preserved offset information), converted to YARA hex signatures, and wildcarded at displacement operands. Stack-balancing instructions are removed from the beginning and end of sequences.

For string and DNAHash features, two-dimensional clustering across both files and features identifies the patterns that are most consistently shared within a malware family. A rule-scoring mechanism combines detection coverage with false-positive risk. A rule compression step then merges similar rules via subset compression and intersection merging to minimize rule count and file size — important for teams that deploy rules at high frequency.

The DNAHash Feature

▶ Watch: DNAHash Feature Explanation (06:00)

DNAHash addresses a gap in existing tools: malware that uses self-modifying code, packing, or obfuscation to defeat string and function analysis. By hashing structural fields within the binary (rather than content), DNAHash produces features that remain stable even when code is dynamically modified.

In practice, DNAHash features are clustered using a heuristic approach that sets a maximum number of differing hash segments between files to identify shared structural patterns. The experimental results showed that adding DNAHash to LMD-YARA's native YARA output improved detection rates — though even native LMD-YARA (without DNAHash) outperformed both YARAgen and AutoYARA.

Experimental Results

▶ Watch: Benchmark Comparison Results (18:02)

The team benchmarked LMD-YARA against YARAgen and AutoYARA on two datasets:

Public malware dataset (post-2025, top 60 families):

• LMD-YARA outperformed both tools on detection rate for training and test sets across family-level evaluation.
• YARAgen's simple rule mode ("YARAgen_gn") achieved high detection rates but generated significantly more false positives than either LMD-YARA or AutoYARA.
• LMD-YARA with DNAHash further improved detection rates beyond native YARA-only rules.

False positive rate (2.3 million benign samples):

• LMD-YARA significantly outperformed both tools — both in false positive samples per family and total false positives — demonstrating that interpretable function-level features reduce spurious matches.

AutoYARA's own public dataset:

• LMD-YARA outperformed both comparison tools on this dataset as well, confirming robustness across different benchmark conditions.

A detailed case study illustrated the advantage: for one malware family whose common strings were all standard Windows API names (filtered out by YARAgen's built-in whitelist), YARAgen had to rely on rare strings appearing in only a few samples, resulting in low coverage. LMD-YARA's inclusion of a YARA-Tower decryption function as a key feature achieved 100% detection with zero false positives on the benign dataset. The rule also revealed that the family performs self-decryption, process injection, and registry modification — behavioral context absent from competing tools.

Notable Quotes

"Most existing automated rule generation methods focus on YARA rules. These approaches use machine learning or deep learning to extract static features. However, they still face challenges in rule quality, false positive rates, and the interpretability of features."

— Wang Xiaochen ▶ 00:00

"Function features are important for creating explainable rules with low false positives."

— Liu Yiping ▶ 12:02

"LMD-YARA's interpretable rules reveal that the malware family may have behaviors such as self-decryption, process injection, and modify files or registries — information that goes beyond detection. This makes our rules not only effective for detection, but also useful for explaining alarms in real-world situations."

— Liu Yiping ▶ 22:03

Key Takeaways

• LLM-driven feature selection improves rule quality. Rather than treating all extracted features as equally valid, LMD-YARA uses an LLM to select the most semantically meaningful strings and identify behaviorally significant functions, reducing false positives without sacrificing coverage.
• DNAHash extends coverage to evasive malware. When self-modifying code defeats string and function feature extraction, structural binary hashing provides an additional detection dimension that other tools lack.
• Interpretability is a first-class output. LMD-YARA rules carry behavioral labels explaining why a sample matched — directly actionable for alert triage and threat intelligence.
• Fine-tuning small models with GRPO is practical. Qwen3-2032B serves as a strong teacher model; smaller fine-tuned models can maintain much of its function-labeling capability at significantly lower inference cost.
• Rule compression matters operationally. Subset compression and intersection merging reduce the total number of deployed rules, addressing real constraints in environments that update detections frequently.

Slides were not listed as available for this briefing.