A Deep Dive into Function Inlining and its Security Implications for ML-based Binary Analysis
Omar Abusabha
Network and Distributed System Security (NDSS) Symposium 2026 · Day 3 · Systems Security
**Function inlining** -- replacing a function call with the callee's body -- is one of the most common compiler optimizations. In CoreUtils compiled at O3, **more than half of all functions are inlined**, with the majority completely removed from the binary. This talk systematically investigates how inlining affects the static features that ML-based binary analysis models rely on, and introduces **extreme inlining** -- a technique using a single vanilla Clang compiler flag (`-inline-threshold` set to a large value) to dramatically increase inlining ratios without modifying compiler source code.
AI review
A useful adversarial robustness study showing that function inlining -- a standard compiler optimization controllable via a single Clang flag -- causes non-negligible performance drops across four ML-based binary analysis tasks. The extreme inlining technique as an evasion mechanism is the most interesting finding: an adversary can confuse ML-based malware detectors using only vanilla compiler flags, no sophisticated obfuscation needed.