Syscalls In Shellcode: Techniques For Malicious Functionality

Syscalls In Shellcode: Techniques For Malicious Functionality

Speakers: Bramwell Brizendine (Assistant Professor of Computer Science, University of Alabama in Huntsville)

Conference: HITB Amsterdam

YouTube: https://www.youtube.com/watch?v=179l_gnPp2U

Overview

In this insightful talk from HITB Amsterdam, Dr. Bramwell Brizendine, a leading expert in vulnerability and exploitation research, delves into advanced techniques for leveraging Windows syscalls within shellcode. The presentation, titled "Windows Syscalls in Shellcode: Advanced Techniques for Malicious Functionality," highlights a critical gap in existing offensive security practices: the underutilization of direct syscalls in shellcode, particularly within the WoW64 (Windows-on-Windows 64-bit) environment. Dr. Brizendine, the creator of the ShellWasp tool, demonstrates how to construct highly portable and stealthy syscall-only shellcode, offering a powerful avenue for evading modern Endpoint Detection and Response (EDR) solutions.

The core motivation behind this research stems from the observation that while traditional Windows API-based shellcode is ubiquitous, it is also highly susceptible to EDR detection. By bypassing the user-mode API layer and interacting directly with the kernel via syscalls, malicious functionality can operate with a significantly reduced footprint, making it much harder for security products to intercept or hook. This talk not only demystifies the complex mechanics of syscall invocation in WoW64 but also introduces practical, automated solutions through ShellWasp 2.0, enabling the creation of advanced malicious payloads that are both potent and resilient against contemporary defenses.

Background

▶ Watch: Introduction to Windows syscalls and agenda (0:00)

Traditional Windows shellcode primarily relies on techniques like Process Environment Block (PEB) walking to locate the Exports Directory and resolve the runtime addresses of Windows API functions. This approach grants access to over 10,000 APIs, making it the prevalent method for 99.9% of Windows shellcode. However, this reliance on documented APIs makes such shellcode easily detectable and interceptable by EDR solutions, which commonly hook these user-mode functions to monitor for malicious activity.

In contrast, Windows syscalls offer a direct path to the kernel. Most Windows APIs, at some point, directly or indirectly call an NTDLL function. These NTDLL functions serve as the final gateway from user mode to kernel mode, often having a one-to-one correspondence with a Windows syscall of the same name. The mechanism involves loading a System Service Number (SSN) into the EAX register, which then triggers the syscall. The primary appeal of using syscalls is their potential to bypass EDRs, as many EDR solutions focus their hooks at the user-mode API layer, leaving direct syscalls largely undetected. While advanced EDRs may attempt to detect syscalls, this remains a significant challenge, making syscall usage a trendy topic in the red team community.

A common misconception is that many NTDLL functions, such as NtAllocateVirtualMemory, are undocumented. While a few dozen are indeed documented, the vast majority are not, requiring researchers to consult alternative sources like undocumented function repositories, blog posts, or perform their own reverse engineering. This "undocumented" nature also implies that syscall implementations can change without notice, posing a significant challenge for shellcode portability.

Dr. Brizendine's research originated from an NSA grant to develop a shellcode analysis framework called Sharon. During this project, he realized that despite the existence of specialized "egg hunter" shellcode utilizing syscalls (e.g., NtProtectVirtualMemory for memory validation), there was a surprising lack of general-purpose syscall-only shellcode examples, especially for modern Windows environments. The only notable instance he found was a 2005 Windows XP-era shellcode that hardcoded SSNs, rendering it non-portable. This observation spurred a separate project to understand and reverse engineer syscalls for a shellcode context, focusing specifically on the 32-bit WoW64 environment, which remains the predominant architecture for Windows shellcode, despite the prevalence of 64-bit systems. The goal was to create "pure syscalling" shellcode, completely eschewing traditional Windows APIs.

The most significant hurdle to syscall portability is that SSNs are tied to specific OS builds or service packs. With over 13 different OS builds for Windows 10 alone, hardcoding SSNs means shellcode might only work on a single, specific version of the OS. Using the wrong SSN can result in calling an entirely different syscall, leading to unexpected behavior or crashes. While tables of pre-computed syscall values exist (like those from "Giroux"), they require constant updates and matching to specific OS builds.

Historically, syscalls in offensive security were largely confined to special-purpose egg hunters. These tiny shellcodes would use NtProtectVirtualMemory to scan process memory for a larger, tagged shellcode payload, then redirect control flow. Beyond this, only the aforementioned obsolete Windows XP example achieved more complex functionality (e.g., CreateProcess equivalent via registry persistence using four syscalls). However, a 2008 report on increased direct syscall usage in malware served as a major inspiration, leading to the development of several modern syscall tools.

These tools, such as Syscall Dumper (using RtlGetVersion for pre-computed values) and SysWhispers (using 64-bit PEB for OS build identification), primarily aimed to extend syscall usage to traditional C/C++ applications for EDR evasion, but were not designed for shellcode. A breakthrough came with Elephant Seal, who introduced a clever method to dynamically resolve SSNs by noting that NTDLL functions, when sorted by address, often have sequentially incrementing syscall values. This principle was later adopted and popularized by SysWhispers2, which focused on ZW functions and became one of the most widely used syscall tools. Other techniques like Hell's Gate extract SSNs by looking for the MOV EAX, SSN opcode in NTDLL, while Halo's Gate refines this by checking adjacent functions if EDR hooks interfere. The underlying "secret" of many modern dynamic techniques is this property of incrementing syscall values.

Key Findings

▶ Watch: Evolution of dynamic syscall resolution techniques (11:00)

Dr. Brizendine's research and the development of ShellWasp 2.0 address the critical need for portable, stealthy, and advanced syscall-only shellcode in WoW64 environments. Several key findings and contributions emerged from this work:

1. WoW64 Syscall Invocation Mechanics: The talk thoroughly dissects how syscalls are invoked in 32-bit applications running on a 64-bit OS. For Windows 7, the CALL DWORD FS:[0C0h] instruction (often referred to as FSC0) acts as the primary gateway. This instruction dereferences to a far jump with a selector of 33h, which is crucial for transitioning from 32-bit to 64-bit mode. While Windows 10 and 11 changed how NTDLL internally handles this (using hardcoded offsets susceptible to ASLR), the FSC0 address remains accessible, allowing it to be repurposed for consistent syscall invocation across modern Windows versions.

1. ShellWasp 2.0 - Automated Portable Syscall Shellcode Generation:

• Compact SSN Management: Instead of dynamic SSN resolution (which increases shellcode size), ShellWasp 2.0 uses pre-computed syscall values. To ensure portability, it first identifies the target OS build.
• Diverse OS Build Identification: ShellWasp 2.0 offers multiple methods to identify the OS build:
• Traditional 32-bit PEB walking via FS:[30h] and an offset of 0ACh.
• Using the User Shared Data at a hardcoded offset 7FFE0260h for Windows 10/11.
• A stealthier "double Heaven's Gate" technique, which involves transitioning to 64-bit mode, accessing the 64-bit TEB (pointed to by R12), then dereferencing to the 32-bit TEB and PEB, before returning to 32-bit mode. This method is designed to confuse debuggers.
• Cross-OS Invocation Compatibility: The tool intelligently generates shellcode that can work across Windows 7, 10, and 11, despite their fundamentally incompatible syscall invocation methods in WoW64. It checks the OS major version and adjusts the syscall function accordingly.
• Automated Parameter and Stack Management: ShellWasp handles the tedious details of setting up and managing the syscall array, restoring registers (like EDI), and calculating stack cleanup, providing a template for users to fill in specific parameter values.

1. Novel Stealthy Syscall Invocation Methods: Beyond the classic FSC0 method, ShellWasp 2.0 introduces new, less recognized ways to invoke syscalls:

• Leveraging Heaven's Gate combined with a JMP DWORD PTR [R15+F8h]. This constant offset, when used after a 32-bit to 64-bit transition, leads to the next stage of the WoW64 transition layer, offering a stealthier alternative to FSC0.
• A more sophisticated method using Heaven's Gate followed by a JMP DWORD PTR [R15+RCX*8], which targets the Turbo Thunk Dispatch. This approach closely mimics how the Windows OS internally handles syscall transitions, making it even harder to distinguish from legitimate system behavior.
• For Windows 7, an equivalent extended x64 code block is provided, which appears as confusing INC/DEC ESP/EBP instructions in a 32-bit debugger, but are actual R12/R13/R15 operations in 64-bit mode.

1. Demonstrated Advanced Malicious Functionality: The talk culminates in a sophisticated process injection proof-of-concept into Discord, executed entirely using 10 Windows syscalls, without any traditional Windows API calls. This showcases the ability to achieve complex actions like memory allocation, process enumeration, handle acquisition, file creation, section mapping, memory protection (RWX), and remote thread creation, all via direct syscalls.

1. CFG Bypass via Syscalls: Dr. Brizendine also demonstrated how to defeat Control Flow Guard (CFG) on Windows 10/11 using NtSetInformationVirtualMemory (specifically with VmCfgCallTargetInformation). This technique involves creating an exception for a specific memory region, allowing execution to proceed even when CFG would normally block it. The method highlights the necessity of reverse engineering undocumented syscall parameters by observing the behavior of corresponding documented kernel-level functions (e.g., SetProcessValidCallTargets).

These findings collectively present a powerful new paradigm for offensive security, enabling the creation of highly evasive and functional shellcode.

Technical Deep Dive

▶ Watch: Novel syscall invocation: R15F8 and Heaven's Gate (32:00)

The technical core of Dr. Brizendine's talk revolves around the intricate mechanisms of WoW64 syscall invocation and the innovative ways ShellWasp 2.0 manipulates them.

At its most fundamental, a Windows syscall in a 32-bit application on a 64-bit OS (WoW64) begins with an NTDLL function moving the System Service Number (SSN) into the EAX register. This is typically followed by XOR ECX, ECX to zero out ECX and LEA EDX, [ESP+4] to set up the stack pointer for arguments. The critical step is then invoking the syscall.

In Windows 7, this is achieved via CALL DWORD FS:[0C0h], often termed FSC0. This instruction dereferences to a far jump, specifically JMP FAR 33:XXXX, where 33h is the segment selector that triggers the transition from 32-bit to 64-bit mode. This transition is handled by WoW64.dll, which prepares the 64-bit context by extending 32-bit registers to 64-bit, moving stack values to corresponding 64-bit registers, and saving/restoring context. While Microsoft's NTDLL implementation in Windows 10 and 11 moved away from FSC0 for syscall invocation (using hardcoded offsets that are ASLR-dependent), the FSC0 address itself remains a consistent and exploitable entry point for shellcode, allowing the older Windows 7-style invocation to persist across modern OS versions.

ShellWasp 2.0's approach to portability addresses the SSN variability and invocation differences across OS versions. It opts for pre-computed SSN values to keep shellcode compact. The key is accurately identifying the target OS build. ShellWasp offers three primary methods for this:

1. 32-bit PEB Walking: This traditional method involves reading the OS build number from the PEB structure. Specifically, FS:[30h] points to the PEB, and an offset of 0ACh within the PEB structure (e.g., FS:[30h + 0ACh]) provides the OS build number (e.g., 4A64h for Windows 10 21H2). This is a well-understood and reliable technique.
2. User Shared Data: For Windows 10 and 11, the OS build number can be directly retrieved from a hardcoded memory address 7FFE0260h within the User Shared Data region. ShellWasp can also encode this address (e.g., via XOR or addition) to add a layer of obfuscation.
3. Double Heaven's Gate: This is a more advanced and stealthy technique. It involves a "Heaven's Gate" transition from 32-bit to 64-bit mode (typically via PUSH 33h; PUSH <destination>; RETF). Once in 64-bit mode, the R12 register conveniently points to the 64-bit TEB (Thread Environment Block). The 32-bit TEB is located at offset 0 within the 64-bit TEB. From the 32-bit TEB, the 32-bit PEB can be accessed, and then the OS build. After retrieving the necessary information, the shellcode transitions back to 32-bit mode. This method is designed to confuse debuggers, as most 32-bit debuggers will skip over the 64-bit code, making it difficult to trace.

Once the OS build is identified, ShellWasp manages a syscall array, where SSNs for the targeted OS versions are stored. The shellcode can then access these SSNs via EDI (which points to the array) and MOV EAX, DWORD PTR [EDI + Offset]. The actual syscall invocation is encapsulated in a function, rSyscallFunction. For cross-OS compatibility, ShellWasp introduces conditional logic based on the OS major version. If targeting both Windows 7 and Windows 10/11, it will check EDI - 4 (where the major version is stored) and branch to the appropriate invocation method, which differs slightly due to implicit stack expectations in Windows 10.

Beyond the standard FSC0 invocation, ShellWasp 2.0 introduces two new, more sophisticated methods to enhance stealth:

1. Heaven's Gate + JMP R15+F8h: After a Heaven's Gate transition to 64-bit mode, instead of FSC0, the shellcode executes JMP DWORD PTR [R15+F8h]. This constant offset, when combined with R15, directs control flow to a critical part of the WoW64 transition layer. This approach is less commonly recognized than FSC0 and can bypass detection mechanisms specifically looking for the FSC0 pattern.
2. Heaven's Gate + Turbo Thunk Dispatch: This method takes stealth a step further. After a Heaven's Gate transition, the shellcode uses JMP DWORD PTR [R15+RCX*8], which targets the Turbo Thunk Dispatch. This is the mechanism the Windows OS itself uses for certain syscalls, making the malicious activity blend in more seamlessly with legitimate system operations. This approach requires more intricate setup but offers superior evasion.
3. Windows 7 Extended x64 Code: For Windows 7 targets, a similar extended 64-bit code block can be used. When viewed in a 32-bit debugger, this code appears as a series of confusing INC and DEC operations on ESP and EBP. However, in 64-bit mode, these are actually operations involving R12, R13, and R15 registers, effectively obfuscating the true intent from typical 32-bit analysis tools.

A significant challenge in writing syscall shellcode is the manual preparation of parameters. Unlike high-level languages, shellcode requires explicit pushes for pointers, nested structures, and constants (which must be found as hex values from SDK headers, documentation, or reverse engineering). NTSTATUS codes returned in EAX are crucial for debugging, with 0x0 indicating success and other values providing specific error information. ShellWasp supports both NASM-style and inline assembly output for these 64-bit code blocks, catering to different development workflows.

Demo / Proof of Concept

▶ Watch: Demo: Pure syscall process injection into Discord (48:00)

Dr. Brizendine presented several compelling demonstrations to showcase the capabilities of syscall-only shellcode generated by ShellWasp 2.0.

The primary demonstration involved a complex process injection into Discord, executed entirely using 10 distinct Windows syscalls. The goal was to map the URLMON.DLL library into a running Discord process, modify its memory permissions, inject a second-stage shellcode (a simple message box in this case, but potentially anything malicious), and then execute it by creating a remote thread.

The steps, as shown in a pre-recorded WinDbg session targeting Windows 7, were:

1. NtAllocateVirtualMemory: Allocate a large region of memory (e.g., 0x10000 bytes) for SYSTEM_PROCESS_INFORMATION.
2. NtQuerySystemInformation: Retrieve system process information, then iterate through the entries, using string comparison (rep cmps) to locate the "Discord" process and extract its PID.
3. NtOpenProcess: Convert the obtained PID into a process handle for Discord.
4. Prepare the string "URLMON" as a UNICODE_STRING within an OBJECT_ATTRIBUTES structure.
5. NtCreateFile: Create a conceptual "file" for "URLMON" (this doesn't create a physical file on disk at this stage).
6. NtCreateSection: Create a section object for the "URLMON" file.
7. NtMapViewOfSection: Map the "URLMON" section into the Discord process's address space.
8. NtProtectVirtualMemory: Modify the permissions of the newly mapped memory region to RWX (Read/Write/Execute).
9. NtWriteVirtualMemory: Write the second-stage shellcode (the message box payload) into the RWX region within Discord. The demo showed this payload being written at a specific offset, e.g., 0x3000 into the mapped region.
10. NtCreateThreadEx: Create a new thread in the Discord process, pointing its entry point to the injected shellcode, causing it to execute.

The demo successfully produced a message box originating from the Discord process, confirming the complete syscall-based injection.

A crucial point was then raised regarding Control Flow Guard (CFG). While the injection worked seamlessly on Windows 7, attempting it on Windows 10/11 would result in a NTSTATUS 0xC0000029 (fast fail), which indicates a CFG violation. Dr. Brizendine demonstrated that CFG can be bypassed using NtSetInformationVirtualMemory with the VmCfgCallTargetInformation class. This syscall, while less documented, allows for the creation of an exception for a specific memory region, effectively disabling CFG checks for that area. The process of discovering the correct parameters for this syscall involves reverse engineering the corresponding kernel-level function, SetProcessValidCallTargets, by setting breakpoints and observing parameter values in a debugger.

Another simpler demo showcased terminating Notepad using four syscalls: NtAllocateVirtualMemory, NtQuerySystemInformation (to find Notepad's PID), NtOpenProcess (to get a handle), and NtTerminateProcess. This illustrated how even basic malicious actions can be achieved purely through syscalls, with the demo showing Notepad instantly disappearing.

The demonstrations effectively highlighted ShellWasp's ability to generate functional, portable, and stealthy syscall shellcode, capable of executing complex operations and bypassing modern exploit mitigations like CFG.

Defensive Implications

▶ Watch: Practical tips for syscall shellcode development (56:00)

The techniques presented by Dr. Brizendine have significant implications for defensive security, particularly in the realm of EDR efficacy and malware analysis. The core takeaway is that direct Windows syscalls offer a powerful avenue for EDR evasion, necessitating a shift in defensive strategies.

Traditional EDRs primarily rely on user-mode API hooking to detect malicious activity. By bypassing these hooks and interacting directly with the kernel via syscalls, malicious functionality can operate with a significantly reduced footprint in user-mode, making it harder to intercept and analyze. This means defenders cannot solely rely on monitoring CreateRemoteThread or WriteProcessMemory at the API level if adversaries are using NtCreateThreadEx or NtWriteVirtualMemory directly.

Key defensive considerations and strategies include:

1. Beyond User-Mode Hooking: EDR solutions must evolve to incorporate deeper visibility into kernel-level activity or, at the very least, monitor syscalls themselves. This is challenging due to the volume of legitimate syscalls, but heuristic analysis of unusual syscall sequences (e.g., NtAllocateVirtualMemory followed by NtProtectVirtualMemory to RWX, then NtWriteVirtualMemory and NtCreateThreadEx) could be effective.
2. WoW64 Transition Monitoring: The use of WoW64 transitions (e.g., Heaven's Gate) and 64-bit code within 32-bit processes, as demonstrated by the "double Heaven's Gate" and alternative invocation methods, can confuse debuggers and analysis tools. Defenders need tools that can seamlessly follow execution across 32-bit and 64-bit modes, such as 64-bit WinDbg.
3. Behavioral Analysis: Since direct syscalls are harder to detect by signature or simple API monitoring, a stronger emphasis on behavioral analysis is crucial. For instance, a legitimate application like Discord attempting to map an unexpected DLL, change memory permissions to RWX, or create a new thread with a suspicious entry point should be flagged, regardless of whether it used API calls or syscalls.
4. Process and Memory Integrity: Techniques like mapping URLMON.DLL into a target process and then writing shellcode into it can be detected by monitoring for unusual module loads or modifications to memory regions within critical processes. Memory scanning for RWX pages that contain executable code, especially in areas not typically executable, remains a valid defense.
5. Mitigation Bypass Detection: The demonstrated CFG bypass using NtSetInformationVirtualMemory is a critical concern. While the syscall itself might not be inherently malicious, its specific usage with VmCfgCallTargetInformation to disable CFG for arbitrary memory regions is highly suspicious and should be a strong indicator of compromise.
6. Tooling Awareness: The existence of tools like ShellWasp, which automate the creation of such advanced shellcode, lowers the barrier to entry for adversaries. Security researchers and defenders should be familiar with the capabilities and output patterns of such tools to better anticipate and detect their use.
7. Focus on Post-Exploitation: If initial injection or execution is achieved via stealthy syscalls, defenders should focus on detecting subsequent post-exploitation activities, which may eventually revert to more detectable API calls or exhibit suspicious network, file system, or process behaviors.

In essence, the talk underscores that adversaries are continually finding ways to operate "under the radar." Defensive strategies must adapt by gaining deeper visibility into system internals, focusing on behavioral anomalies, and understanding the evolving landscape of syscall-based evasion techniques.

Key Takeaways

• EDR Evasion: Direct Windows syscalls offer a potent method for bypassing traditional user-mode EDR hooks, significantly enhancing the stealth of malicious functionality compared to Windows API-based shellcode.
• Portability Challenges: Crafting portable syscall shellcode, especially in WoW64 environments, is complex due to the volatility of System Service Numbers (SSNs) across different OS builds and varying syscall invocation mechanisms (e.g., Windows 7 vs. Windows 10/11).
• ShellWasp Automation: Tools like ShellWasp 2.0 automate the generation of portable syscall shellcode templates, handling intricate details such as OS build identification, SSN management, and cross-OS invocation compatibility.
• Advanced Stealth Techniques: ShellWasp 2.0 incorporates novel stealth techniques, including "double Heaven's Gate" for OS build retrieval and alternative syscall invocation methods (e.g., JMP DWORD PTR [R15+F8h], Turbo Thunk Dispatch), which are designed to confuse debuggers and evade detection.
• Complex Functionality via Syscalls: It is possible to achieve advanced malicious functionalities, such as sophisticated process injection (demonstrated with 10 syscalls into Discord) and even Control Flow Guard (CFG) bypass (NtSetInformationVirtualMemory), purely through syscalls, albeit requiring significant manual effort in parameter and structure preparation.
• Evolving Defensive Needs: Defenders must adapt their strategies beyond user-mode API hooking, focusing on deeper behavioral analysis, monitoring WoW64 transitions, scrutinizing syscall sequences for anomalies, and understanding how new tools facilitate syscall-based evasion.

About the Speaker(s)

Dr. Bramwell Brizendine is an Assistant Professor of Computer Science at the University of Alabama in Huntsville. He is the founding director of the Verona lab, which specializes in vulnerability and exploitation research. A prolific creator in the offensive security space, Dr. Brizendine developed ShellWasp, the tool central to this talk, which focuses on advanced syscall techniques for malicious functionality. He is also the creator of JobRocket, a jump-point programming tool featured at the 2021 Hack in the Box Amsterdam conference, and Sharon, a shellcode analysis framework capable of analyzing Windows syscalls. Dr. Brizendine holds a PhD in Cyber Operations, a highly technical degree, and teaches a range of subjects including software exploitation, reverse engineering, malware analysis, and offensive security.