Breaking architecture barriers: Running x86 games and apps on ARM
Tony Wasserka
Gulaschprogrammiernacht 24 (GPN24) · Day 3 · ZKM Vortragssaal
Overview
In this insightful talk at GPN24, Tony Wasserka, a seasoned software engineer and long-time emulator developer, delved into the intricacies of FEX, an open-source x86-on-ARM emulator designed specifically for Linux. The presentation, titled "Breaking architecture barriers: Running x86 games and apps on ARM," highlighted the critical need for such a solution as ARM-based laptops gain traction, offering benefits like extended battery life and fanless operation. Wasserka emphasized that while proprietary platforms like macOS (Rosetta) and Microsoft Windows (Prism) have their own vendor-provided emulation layers, FEX steps in to fill this crucial gap for the open-source Linux ecosystem.

Key moments
- 0:00 Introduction to x86 on ARM emulation and FEX
- 5:30 Emulation explained: simple translation vs. nuanced interpretation
- 12:00 Deep dive: Binary recompilation via intermediate representation
- 19:00 Memory models (TSO) and their impact on performance
- 35:00 Performance optimization: GPU driver bridging for native speed
- 46:30 FEX in the wild: Asahi Linux, Parallels, Steam Deck
- 51:00 ARM 64EC and the Wine on ARM (WoA) architecture
- 59:00 Q&A: Apple CPUs' native x86 memory model support
Breaking architecture barriers: Running x86 games and apps on ARM
Speakers: Tony Wasserka, Software Engineer (Contractor at Valve)
Conference: GPN24
YouTube: https://www.youtube.com/watch?v=zFb_o9A2ABo
Overview
In this insightful talk at GPN24, Tony Wasserka, a seasoned software engineer and long-time emulator developer, delved into the intricacies of FEX, an open-source x86-on-ARM emulator designed specifically for Linux. The presentation, titled "Breaking architecture barriers: Running x86 games and apps on ARM," highlighted the critical need for such a solution as ARM-based laptops gain traction, offering benefits like extended battery life and fanless operation. Wasserka emphasized that while proprietary platforms like macOS (Rosetta) and Microsoft Windows (Prism) have their own vendor-provided emulation layers, FEX steps in to fill this crucial gap for the open-source Linux ecosystem.
The core challenge addressed by FEX is enabling users to run existing x86 applications, particularly demanding ones like video games, seamlessly and performantly on ARM Linux hardware. This is not merely a technical convenience but a vital component for the widespread adoption of ARM-based devices in the Linux world. Wasserka illustrated the complexity of translating an entire software stack – from CPU instructions to kernel interactions and graphics drivers – to ensure high compatibility and competitive performance against native x86 systems. The talk explored the sophisticated techniques employed by FEX, including dynamic binary recompilation, advanced optimization passes, and innovative solutions for handling architectural differences like memory models.
Wasserka's personal journey, from a tinkerer fascinated by console emulators like Dolphin to a quantum physicist and eventually a software engineer working on graphics drivers and now FEX, underscored the deep expertise driving the project. His passion for making "impossible" cross-architecture execution a reality resonated throughout the presentation, culminating in a demonstration of FEX's capability to run popular 3D games and even Windows titles via Wine/Proton on ARM Linux. The increasing adoption of FEX by major players like Valve for the Steam Deck, Parallels, and Crossover further solidifies its position as a pivotal technology for the future of ARM on Linux.
Background
▶ Watch: Introduction to x86 on ARM emulation and FEX (0:00)
The computing landscape has long been dominated by the x86 architecture, particularly in desktop and laptop environments. However, the emergence of ARM-based processors, initially prominent in mobile devices, has gradually extended into the personal computing space. Driven by advantages such as superior power efficiency, leading to longer battery life and fanless designs, ARM CPUs are becoming an increasingly attractive alternative. Devices like Apple's M1/M2 MacBooks and various Windows-on-ARM laptops exemplify this shift.
This transition, while offering significant hardware benefits, introduces a substantial software compatibility challenge. Applications and operating systems are typically compiled for a specific CPU architecture. Running software designed for x86 on an ARM processor requires a translation layer, often referred to as emulation or binary translation. Without such a layer, users migrating to ARM-based systems would be unable to run a vast library of existing software, including productivity tools, development environments, and, critically for many, video games.
Proprietary solutions have emerged to address this on their respective platforms. Apple developed Rosetta 2 for macOS, allowing x86_64 applications to run on Apple Silicon Macs with remarkable performance. Microsoft introduced Prism (now often referred to as Windows on ARM emulation) for its ARM-based Windows devices. However, for the Linux ecosystem, a widely adopted, high-performance, and open-source solution for x86-on-ARM emulation was conspicuously absent. This is the void that FEX (FEX-emu) aims to fill.
FEX is an open-source project that has been under development for over seven years, initially operating somewhat "under the radar" due to the relatively small number of ARM Linux users. However, with the increasing availability and performance of ARM hardware, coupled with initiatives like the Steam Deck and efforts to bring Linux to Apple Silicon devices, FEX's relevance has grown exponentially. The problem it solves is multifaceted, extending beyond simple instruction translation to encompass complex interactions with the operating system kernel, memory management, and graphics hardware, all while striving for near-native performance.
Key Findings
▶ Watch: Deep dive: Binary recompilation via intermediate representation (12:00)
The central finding and contribution of the talk is the robust and highly performant FEX emulator itself, which provides an open-source solution for running x86 applications on ARM Linux systems. This project demonstrates that sophisticated, high-speed cross-architecture binary translation is achievable within an open-source framework, successfully bridging a critical compatibility gap.
Key findings and contributions highlighted by Wasserka include:
- Advanced Binary Recompilation: FEX employs a sophisticated dynamic binary recompilation engine. Instead of direct instruction-to-instruction mapping, it decodes x86 instructions into an internal Intermediate Representation (IR). This IR allows for architecture-agnostic optimization passes, similar to those found in traditional compilers, significantly improving the efficiency and compactness of the generated ARM code. This approach effectively mitigates much of the inherent overhead associated with emulation.
- Robust x86 Feature Support: FEX supports a broad spectrum of x86 instruction sets, including complex vectorization extensions like SSE and AVX. Notably, FEX was among the first x86-on-ARM emulators, alongside Apple's Rosetta, to offer AVX support, showcasing its advanced capabilities and ability to keep pace with modern x86 instruction sets.
- Intelligent Memory Model Handling: One of the most challenging aspects of x86-on-ARM emulation is reconciling the differing memory models (e.g., x86's Total Store Order, TSO, versus ARM's weaker ordering). FEX implements sophisticated heuristics to ensure x86 memory ordering guarantees are upheld on ARM without crippling performance. Specific examples include optimizing local stack accesses and a bespoke hack for Unity games that targets a very specific instruction pattern (store relative to a register by 61 bytes) to enable expensive memory emulation only when absolutely necessary, yielding massive speed improvements.
- Seamless Linux Kernel Integration: FEX acts as a comprehensive compatibility layer, translating x86 syscalls to native ARM Linux syscalls, redirecting file system accesses to provide x86 versions of libraries, and managing multi-threading and signaling. This deep integration ensures that x86 applications perceive a native x86 Linux environment.
- Performance-Critical Optimizations: To achieve competitive performance, FEX implements several crucial optimizations:
- GPU Driver Bridging: For performance-sensitive graphics APIs like Vulkan and OpenGL, FEX can bypass x86 driver emulation entirely. It intercepts calls and directly translates them to the native ARM Vulkan/OpenGL driver on the host, dramatically reducing overhead and memory usage, leading to significant FPS gains (e.g., from 5 FPS to 60 FPS in one observed game).
- Code Caching: To mitigate the overhead of dynamic recompilation, FEX caches compiled ARM code to disk. This means subsequent runs of an application or re-entering previously compiled game areas benefit from pre-compiled code, eliminating micro-stutters and improving startup times.
- Adoption and Industry Impact: FEX's effectiveness is validated by its adoption across various significant projects and commercial offerings:
- Asahi Linux: Used to run x86 Linux software on Apple Silicon Macs, often in conjunction with a MicroVM to handle memory page size differences.
- Parallels Desktop: Credits FEX for contributing to their x86 virtualization work.
- Crossover: The professional Wine offering has integrated FEX to support Windows applications on ARM Linux, enabling demanding titles like Cyberpunk 2077 to run at "quite capable frame rates."
- Valve (Steam Deck): Valve has been actively funding FEX development as a contractor, leveraging it for game support on the upcoming ARM-based Steam Frame (likely a misstatement for Steam Deck's ARM-based future or a new device, given Steam Deck is x86 currently, but the intent is clear: Valve is investing in ARM Linux gaming).
- "Wine on ARM" (WoA) Architecture: FEX supports a highly optimized architecture inspired by Microsoft's ARM 64EC, where Wine itself is compiled natively for ARM. This allows for direct calls from recompiled x86 code into native ARM Wine and Windows libraries, drastically simplifying the emulation stack, reducing overhead, and boosting performance.
These findings collectively position FEX as a mature and highly impactful project, essential for the future of ARM-based computing on Linux, particularly for gaming and demanding applications.
Technical Deep Dive
▶ Watch: Performance optimization: GPU driver bridging for native speed (35:00)
At its core, FEX tackles the monumental task of translating the entire x86 software stack to run on ARM hardware. Wasserka broke this down into several layers, starting from the hardware up to the application.
The traditional stack comprises:
- x86 CPU (hardware)
- Linux kernel (operating system)
- Userland libraries (e.g., glibc, Java runtimes, graphics drivers like Vulkan/OpenGL)
- Game engines (e.g., Unity, Unreal Engine)
- Application/Game (e.g., Super Kitten Adventure)
FEX effectively replaces the x86 CPU with an ARM CPU and inserts itself as a comprehensive translation layer that makes the ARM system appear as an x86 system to the application and its underlying libraries.
Binary Recompilation
The cornerstone of FEX's CPU emulation is dynamic binary recompilation. When an x86 application executes, FEX doesn't interpret each instruction sequentially. Instead, it:
- Decodes streams of raw x86 instruction encodings (ones and zeros) to understand their precise semantics.
- Maps these decoded instructions to an internal, architecture-agnostic Intermediate Representation (IR). This IR is a higher-level language specifically designed for FEX, capturing the intent of each x86 instruction (e.g.,
pushinstruction becomes an IRpush, register writes becomestore registerprimitives). This often results in a "blow-up" in terms of intermediate instructions, but it provides a more workable abstraction. - Applies optimization passes on the IR, much like a traditional compiler. These passes identify and remove unneeded instructions, combine multiple instructions into more optimal sequences, and reduce the overall complexity. For example, in a simple loop, FEX can reason globally about the loop body to optimize flag computations, which would otherwise incur significant overhead.
- Generates optimized ARM assembly code from the IR. This process aims to produce the smallest and fastest possible ARM output.
Wasserka demonstrated that even a simple x86 decrement operation (dec ebx) can translate to multiple ARM instructions (sub w6, #1, cset, rmif) due to the need to handle flag computation. x86 CPUs unconditionally set status flags (e.g., overflow, negative) after mathematical operations, which applications might query. FEX must emulate this behavior, but its optimization passes are crucial for discarding flag computations when they are not actually used by the application, eliminating around 20 unnecessary instructions in some cases.
The sheer scope of the x86 instruction set, ranging from 1,000 to 6,000 different instruction encodings, makes this IR and optimization approach essential. Crafting bespoke optimization rules for each pattern would be untenable. FEX also supports modern x86 vectorization extensions like SSE and AVX, being one of the earliest emulators to do so, highlighting its capability to handle complex and performance-critical instruction sets.
Memory Models and Performance Bottlenecks
A significant challenge in cross-architecture emulation lies in reconciling different memory models. x86 adheres to a Total Store Order (TSO) memory model, which provides strong guarantees about the order in which memory operations become visible to other threads. ARM, by contrast, has a weaker memory model, offering the processor more freedom to reorder operations for performance.
Wasserka illustrated this with a multi-threaded example where one thread computes a value (42) and sets a "done" flag, and another thread waits for the flag before printing the result. On x86, the result would reliably be 42. On ARM, due to potential reordering, the "done" flag might be written before the result, leading to a race condition and the old value (0) being printed.
FEX's role is to enforce x86's stronger memory guarantees on ARM. The naive approach involves inserting memory barriers after every load/store instruction and converting some operations to atomic memory operations. This ensures correctness but can halve performance, making it one of the biggest bottlenecks.
To mitigate this, FEX employs heuristics:
- Local Stack Access: For operations on local variables, FEX assumes no multi-threading interaction, allowing it to bypass expensive memory barrier insertions. This assumption holds well in practice and significantly improves performance for common code paths.
- Unity Game Hack: A highly specific optimization targets Unity games. Unity engines use ring buffers for multi-threaded command queuing, relying on x86's strong memory ordering. FEX detects a precise instruction pattern (a store relative to a register by exactly 61 bytes) and only for these specific instances enables the expensive emulation strategy. This "massive hack" dramatically boosts performance for Unity titles by allowing the "happy path" (fast, regular load/store) for most other memory accesses.
- ARM Architecture Extensions: Some modern ARM CPUs (e.g., Apple Silicon) offer built-in support for stronger memory models, which FEX can leverage via bespoke kernel interfaces from projects like Asahi Linux. However, enabling this can introduce a 20% performance overhead on native ARM code, presenting a trade-off.
Linux Kernel and Userland Interaction
Beyond CPU instruction translation, FEX also acts as a compatibility layer between the x86 application and the ARM Linux kernel. This involves:
- Syscall Translation: x86 applications issue system calls (e.g.,
read,write,mmap) to interact with the kernel. FEX intercepts thesesyscallinstructions, translates their parameters (which might reside in different registers on x86 vs. ARM), and dispatches them to the corresponding native ARM Linux syscalls. This involves explicit "exit points" from recompiled code into C++ handler functions within FEX. - File System Redirection: x86 applications expect to load x86 versions of shared libraries (e.g.,
libc). FEX hooks into file system accesses and redirects them to an x86 root directory, ensuring the application sees and loads the correct architecture-specific libraries. - Multi-threading and Signaling: FEX needs to be aware of and translate multi-threading primitives and inter-process signaling to ensure proper behavior across architectures.
- 32-bit x86 on 64-bit ARM: Running 32-bit x86 applications on a 64-bit ARM system introduces further complexity, as syscall interfaces and data structures (e.g.,
structsfortimequeries) might differ in size and layout. FEX handles this with "ugly repacking logic" to convert between 32-bit and 64-bit structures.
Performance Enhancements: GPU Bridging and Caching
Two significant architectural improvements enhance FEX's performance:
- GPU Driver Bridging: For graphics-intensive applications, emulating the x86 Vulkan or OpenGL driver is a major performance bottleneck. FEX implements a "Vulcan bridge" (and an OpenGL equivalent). By hooking into file system accesses, FEX intercepts attempts to load the x86 graphics driver library. Instead of emulating it, FEX injects its own bridge, which directly calls into the native ARM Vulkan/OpenGL driver installed on the host system. This completely sidesteps x86 emulation for graphics, saving translation overhead, recompilation time, and memory, leading to "dramatic" speedups (e.g., from 5 FPS to 60 FPS in one game).
- Code Caching: Dynamic recompilation is expensive, especially during initial game startup or when encountering new code paths. FEX addresses this with a caching mechanism. Once x86 code is recompiled to ARM, the resulting ARM binaries are saved to disk. On subsequent runs, FEX can load this pre-compiled code, eliminating recompilation overhead and preventing "micro-stutters" when new game areas are loaded.
Wine on ARM (WoA) Architecture with ARM 64EC
FEX also supports a highly optimized architecture for running Windows applications via Wine/Proton, dubbed Wine on ARM (WoA). This approach leverages Microsoft's ARM 64EC calling convention.
Traditionally, running Wine on FEX would involve:
- ARM hardware
- ARM Linux kernel
- FEX (recompiling x86 Wine, x86 Windows libraries, and x86 Linux libraries)
- x86 application
With WoA, the architecture is dramatically simplified:
- The Wine core itself is compiled natively for ARM.
- ARM 64EC is a custom calling convention that allows recompiled x86 code to call native ARM code with minimal overhead, almost directly.
- Microsoft can compile their Windows libraries to ARM 64EC.
- This means the recompiler block moves up the stack. FEX only needs to recompile the x86 application itself. The application then talks to ARM 64EC Windows libraries, which in turn use native ARM Linux libraries.
- This eliminates the need to recompile Wine, many Windows libraries, and the kernel compatibility layer for Linux libraries. The stack condenses significantly, leading to less memory overhead, fewer recompilations, and substantially faster execution.
This "WoA" approach, inspired by Microsoft's design, represents a massive simplification and performance gain, offering a glimpse into the future of cross-platform compatibility on ARM.
Demo / Proof of Concept
▶ Watch: FEX in the wild: Asahi Linux, Parallels, Steam Deck (46:30)
Tony Wasserka prepared a multi-part demonstration to showcase FEX's capabilities, despite acknowledging its historical tendency to "fail spectacularly in different ways."
The initial part of the demo, intended to be a subtle reveal, involved running the presentation itself (an HTML page in Firefox) as an x86 application under FEX. Although there were initial technical glitches with the Linux audio driver crashing and the presentation software, Wasserka eventually got it working, highlighting that if it hadn't failed, the audience wouldn't have noticed that Firefox was running as an emulated x86 application rather than a native ARM one. This demonstrated FEX's ability to seamlessly integrate x86 applications into the native ARM Linux desktop environment, appearing just like any other application in the taskbar with native window decorations.
The main demonstration focused on gaming:
- 3D Linux Game (Ukulele): Wasserka launched the 3D platformer game "Ukulele" through Steam, which was itself running as an x86 application under FEX. He noted that Steam, being based on Chromium Embedded Framework (similar to Electron) with complex sandboxing mechanisms (using Bubblewrap), is ironically one of the most challenging applications to emulate to get the UI working. Once the game launched, it demonstrated FEX's capability to run full 3D games with "reasonable performance." While it lagged slightly on the external display during the presentation, Wasserka assured the audience it ran "fluent" on his laptop screen, showcasing its potential.
- Windows Game via Wine/Proton (Tunic): The most impressive part of the demo involved stacking emulation layers to run a Windows-only game, "Tunic," on ARM Linux. This setup involved:
- ARM CPU
- FEX emulating an x86 CPU
- x86 Wine/Proton running on the emulated x86 CPU
- "Tunic" (a Windows game) running on x86 Wine/Proton.
Despite this multi-layered emulation, "Tunic" ran at "quite capable frame rates," even displaying its built-in FPS counter. Wasserka pointed out that this particular Windows game performed more fluently on the external display than Ukulele, underscoring the variability and complexity of emulation performance across different titles and configurations.
The demos, despite some live technical challenges, effectively proved FEX's core claim: it can successfully run complex x86 applications, including demanding 3D games and Windows titles via compatibility layers, on ARM Linux systems with performance that is competitive and often quite playable.
Defensive Implications
▶ Watch: Q&A: Apple CPUs' native x86 memory model support (59:00)
While FEX is primarily a compatibility layer rather than a security tool, its "defensive implications" are significant in the broader context of software longevity, platform choice, and secure migration.
- Extending Software Lifespan: FEX allows organizations and individual users to continue utilizing existing x86 proprietary or legacy software on new ARM-based hardware. This defends against forced obsolescence of critical applications when transitioning to more power-efficient ARM platforms, protecting investments in software licenses and training.
- Enabling Secure ARM Adoption: By providing a robust and well-maintained open-source emulation layer, FEX makes ARM Linux a more viable and attractive platform. This diversification of hardware architectures can indirectly contribute to security by reducing reliance on a single dominant architecture, potentially distributing attack surfaces.
- Accuracy as a Security Feature: The meticulous approach FEX takes to emulate complex x86 behaviors, such as memory models (TSO) and intricate syscall translations, is crucial for stability and, by extension, security. Inaccurate emulation could lead to undefined behavior, crashes, or subtle memory corruption issues that could be exploited. FEX's focus on correctness, even with performance trade-offs and clever heuristics like the Unity ring buffer hack, ensures that applications behave as expected, reducing the likelihood of emulation-induced vulnerabilities.
- Open-Source Transparency: As an open-source project, FEX benefits from community scrutiny and contributions. This transparency allows security researchers and developers to audit the codebase for potential vulnerabilities or architectural flaws, enhancing its overall trustworthiness compared to opaque proprietary solutions.
- Mitigating Supply Chain Risks: For specialized embedded systems or custom Linux distributions on ARM, FEX provides a mechanism to run essential x86 tooling without requiring a separate x86 device. This can simplify supply chains and reduce the attack surface associated with managing multiple hardware architectures.
- Sandboxing and Isolation: While FEX itself is a translation layer, its integration with technologies like Bubblewrap (used by Steam for sandboxing games) demonstrates how emulation can be combined with existing Linux security mechanisms to further isolate emulated applications, preventing them from impacting the host system.
In essence, FEX defends against the "lock-in" effect of architecture-specific software, empowering users and developers with greater flexibility and control over their computing environment, all while striving for a stable and accurate emulation that minimizes security risks inherent in complex translation processes.
Key Takeaways
- FEX is the open-source x86-on-ARM emulator for Linux: It fills a critical compatibility gap, enabling users to run existing x86 applications, including demanding games, on power-efficient ARM Linux hardware.
- Dynamic Binary Recompilation is Key: FEX uses a sophisticated process of decoding x86 instructions, translating them to an Intermediate Representation (IR), applying compiler-like optimizations, and then generating highly efficient ARM assembly code.
- Complex Architectural Differences are Managed: FEX meticulously handles intricate x86 features like flag computations, vectorization extensions (SSE, AVX), and especially the strong x86 memory model (TSO) on ARM's weaker model, using clever heuristics to balance correctness and performance.
- Performance is Achieved Through Innovation: Techniques like GPU driver bridging (for Vulkan/OpenGL) directly calling native ARM drivers and caching recompiled code are crucial for achieving competitive frame rates and smooth user experiences.
- Widespread Adoption Validates FEX's Success: Projects like Asahi Linux, Crossover, Parallels, and especially Valve (for the Steam Deck/Frame) are leveraging FEX, demonstrating its maturity and impact on the ARM Linux ecosystem.
- "Wine on ARM" (WoA) Represents a Future Optimization: By using ARM 64EC, FEX enables Wine itself and many Windows libraries to run natively on ARM, significantly simplifying the emulation stack and boosting performance for Windows applications.
About the Speaker(s)
Tony Wasserka is a self-described "tinkerer turned quantum physicist turned software engineer." His journey into emulation began with a deep fascination for video game console emulators, particularly Dolphin, where he eventually became part of the core development team. After studying physics and writing his thesis in quantum physics, he transitioned back to software engineering for his professional career. He has worked on significant open-source graphics drivers, including the PowerVR driver at Imagination Technologies and the AMD Radeon driver (shipping in Mesa). Currently, he is a key developer working on x86-on-ARM emulation, with Valve supporting his work on FEX as a contractor. He can be followed on Mastodon for more insights into his projects.
Reviews
Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT
Solid systems talk that actually explains how a real emulator works under the hood. Wasserka knows his shit—this is his project, his code, his war stories. Not groundbreaking research, but a clear, technically honest walkthrough of a hard engineering problem with good taste in what details to highlight.
Heather Calloway (CISO) — SOLID
Deep technical talk on x86-to-ARM binary translation for Linux gaming. No security content, no governance angle. Impressive engineering work, but this is systems programming for emulator enthusiasts, not CISO territory.