A Deep Dive into Function Inlining and its Security Implications for ML-based Binary Analysis

A Deep Dive into Function Inlining and its Security Implications for ML-based Binary Analysis

Speakers: Omar Abusabha

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=s5hlyiTRdgs

Overview

Function inlining -- replacing a function call with the callee's body -- is one of the most common compiler optimizations. In CoreUtils compiled at O3, more than half of all functions are inlined, with the majority completely removed from the binary. This talk systematically investigates how inlining affects the static features that ML-based binary analysis models rely on, and introduces extreme inlining -- a technique using a single vanilla Clang compiler flag (-inline-threshold set to a large value) to dramatically increase inlining ratios without modifying compiler source code.

Across four downstream security tasks (binary code similarity, function name inference, malware detection/family prediction, and vulnerability detection), the researchers demonstrate non-negligible performance drops when models encounter inlined code, with extreme inlining causing even more severe degradation. The key insight: inlining introduces multiple semantic contexts into a single function, confusing models that expect each function to have a single coherent purpose.

Background

▶ Watch: Function inlining basics and its prevalence in compiled code (0:00)

Function inlining provides performance benefits by eliminating call overhead, improving cache locality, and enabling function-level optimizations on the combined code. In LLVM/Clang, inlining decisions are made through a combination of external factors (source code attributes like always_inline/noinline, optimization levels, compiler flags) and internal factors (built-in heuristics and a cost model that compares computed inlining cost against a threshold).

The cost model assigns an initial cost and threshold based on optimization level, then adjusts both values based on code characteristics of the call site and callee. If the final cost is below the final threshold, the function is inlined. This is done iteratively for every call site.

ML-based binary analysis models rely on three categories of static features: instruction-level features (counts of instruction types), control flow graph features (structure, loops, node counts), and call graph features (inter-function relationships). Inlining significantly alters all three categories by absorbing one function's code into another.

Key Findings

▶ Watch: LLVM inlining pipeline: external factors, heuristics, cost model (4:00)

• More than half of CoreUtils functions are inlined at O3, with the majority completely removed from the binary
• Extreme inlining is achievable using a single Clang flag (-inline-threshold=<large_value>) without compiler source modification, dramatically increasing inlining ratios
• Binary code similarity detection: Performance drops are concentrated in recall (inlined similar functions appear dissimilar) while precision remains stable (additional semantics don't make dissimilar functions appear similar)
• Function name inference: Normal inlining actually helps (additional context aids name prediction), but extreme inlining degrades performance due to abnormal patterns
• Malware detection/family prediction: Performance drops due to altered static features, with family prediction particularly affected because structural patterns essential for classification are disrupted
• Vulnerability detection: General MRR drops with inlining, though individual exceptions occur
• Training data augmentation with extreme-inlined samples can mitigate some performance loss (detailed in paper)

Technical Deep Dive

▶ Watch: Impact on binary code similarity: recall drops, precision stable (10:00)

LLVM Inlining Pipeline: External factors (attributes, optimization levels, flags) feed into the compilation process. The built-in heuristics first handle deterministic cases (always inline, never inline, recursive functions). Remaining call sites enter the cost model, which computes a final cost and threshold through code analysis. The -inline-threshold flag overrides the initial threshold, and setting it extremely high causes nearly all eligible functions to be inlined.

Extreme Inlining Construction: Using -inline-threshold=<very large value> combined with -flto (Link Time Optimization, enabling cross-translation-unit inlining), the researchers achieve drastically higher inlining ratios. No compiler source modification is needed -- this uses a vanilla Clang compiler with specific flags.

Feature Impact Analysis: Normalized values of instruction counts, CFG metrics (loops, nodes), and call graph metrics (in/out calls) all change significantly under extreme inlining compared to O0 baselines. The changes are non-uniform across functions, making it difficult for models to compensate through simple normalization.

Security Task Analysis: For binary code similarity, inlining introduces foreign semantic context from inlined callees into the caller. Models trained on functions with single coherent purposes encounter functions with multiple contexts, causing false negatives (similar functions appear dissimilar) but not false positives (the additional context doesn't create spurious similarity). For malware family prediction, the structural "fingerprints" that distinguish families are disrupted by inlining, causing the largest performance drops.

Demo / Proof of Concept

▶ Watch: Malware family prediction most affected by inlining changes (12:00)

Evaluation spans four security tasks using models including jTrans, Gemini, and task-specific classifiers. Datasets include CoreUtils, GNU Utils, SPEC benchmarks, and malware families (Mirai, Gafgyt). Three data conditions are tested: no inlining, normal inlining (O3), and extreme inlining.

The malware evaluation required source code access, using Mirai and Gafgyt source code to compile with different inlining levels. The vulnerability detection evaluation showed a general MRR drop, with one interesting exception where a specific function's rank improved due to inlining -- demonstrating that the effects are not uniformly negative.

Defensive Implications

▶ Watch: Q&A: controlling for function size and training data considerations (16:00)

• Model robustness requirements: ML-based binary analysis tools used in production security operations should be tested against inlined code, as normal compilation already causes significant feature changes
• Adversarial evasion via compilation flags: An adversary can use extreme inlining as an evasion technique against ML-based malware detectors and similarity tools using only vanilla compiler flags -- no sophisticated obfuscation needed
• Training data augmentation: Including extreme-inlined samples in training data can improve model robustness, though it doesn't fully mitigate the issue
• Token truncation risks: NLP-based models that truncate input to fixed token lengths may lose important semantic information when functions grow due to inlining
• Family prediction vulnerability: Malware family prediction models are most vulnerable to inlining-induced evasion, as the structural patterns they rely on are most disrupted

Key Takeaways

• Function inlining is ubiquitous (50%+ at O3) and significantly alters the static features ML models rely on
• Extreme inlining is achievable with a single Clang flag, requiring no compiler modification
• Performance drops are task-dependent: similarity recall drops, name inference is initially helped then hurt, malware family prediction is most affected
• Inlining adds foreign semantic context to functions, causing models to treat similar functions as dissimilar
• This represents a low-effort adversarial evasion technique against ML-based binary analysis tools
• Training data augmentation with extreme-inlined samples partially mitigates the problem

About the Speaker(s)

The presentation was delivered by Jeongam from Sungkyunkwan University on behalf of first author Omar Abusabha, who could not attend due to visa issues. The work provides a systematic investigation that multiple audience members (including from Carnegie Mellon) recognized as important for the ML-based binary analysis community.