Proposal and Implementation of a Reduced Assembly Set Compiler for Enabling CPU Security Measures

Proposal and Implementation of a Reduced Assembly Set Compiler for Enabling CPU Security Measures

Speakers: Hiroaki Sakai

Conference: CODE BLUE

YouTube: https://www.youtube.com/watch?v=mf3kxpxRAmc

Overview

In this CODE BLUE talk, Hiroaki Sakai, a Global Jisshi Engineer and renowned author, introduces a novel approach to compiler design: the Reduced Assembly Set Compiler (RASC), specifically his implementation named Rask. The presentation addresses the growing complexity of supporting diverse CPU architectures, particularly in the embedded systems domain, where custom hardware often lacks robust compiler toolchains. Sakai argues that traditional compilers prioritize performance, leading to intricate code generation logic that hinders rapid adaptation to new instruction set architectures (ISAs).

Sakai's solution, Rask, radically simplifies the compiler's output by limiting the generated assembly patterns to an absolute minimum, offloading complex operations to software-based "built-ins." This design philosophy dramatically reduces the effort required to port the compiler to a new CPU, claiming a target of just one day. The talk culminates in a captivating live demonstration where Sakai successfully ports his compiler, NLCC (Norukushi Compiler), to a Renesas V850 architecture within minutes, showcasing the practical feasibility of his "one-day compiler" concept.

This innovation is particularly significant for the security community. By simplifying compiler development and porting, Rask enables security researchers and embedded developers to quickly integrate custom security measures directly into the compiled code for a wide array of existing and emerging CPU architectures. This capability empowers defenders to implement tailored protections, such as stack protectors or memory randomization, even on obscure or proprietary hardware where traditional compiler modifications would be prohibitively complex and time-consuming.

Background

▶ Watch: Introduction and speaker's unique assembly expertise (0:00)

The landscape of modern computing, especially in embedded systems, is characterized by an explosion of diverse CPU architectures. Unlike general-purpose computing dominated by a few major ISAs, embedded environments frequently utilize specialized processors, each with its unique instruction set. This fragmentation presents a significant challenge for toolchain developers: creating a compiler that can efficiently generate code for each new architecture is a monumental task, often requiring deep expertise and substantial development time. Traditional compilers, such as GCC or LLVM, are designed for maximum performance optimization, which translates into highly complex code generation logic, making them difficult to modify or port to new, often niche, ISAs.

Hiroaki Sakai's extensive background directly informs the Rask project. As the author of several technical books, including "Dai Neketsu Assembler Mon" (Great Passionate Assembler Gate), he has developed a unique "Nekketsu Assembler Method" (Passionate Assembler Method). This method involves inferring a CPU's instruction set architecture (ISA) by analyzing compiler output, rather than relying solely on documentation. Through this approach, Sakai has gained experience reading over 60 different CPU assemblers and successfully executing "Hello World" programs on more than 30 CPU simulators. This profound, hands-on understanding of how compilers interact with diverse ISAs laid the groundwork for his idea of a simplified compiler.

Sakai begins by clarifying fundamental compiler concepts. A compiler translates high-level programming languages (like C) into assembly language. An assembler then converts this assembly language into machine code, which the CPU can directly execute. While assemblers typically map assembly instructions to machine code one-to-one (with some exceptions like macro assemblers), compilers handle complex expressions, such as mathematical formulas, function calls (or procedures/subroutines), and conditional branches (e.g., if statements). CPUs primarily perform basic operations like addition or subtraction, so a compiler must decompose complex expressions (like A B + C D) into multiple, primitive assembly instructions. This process, known as syntax analysis, is a core function of a compiler.

From a development perspective, Sakai shares insights from his experience building compilers: they heavily rely on recursion, a technique often avoided by embedded engineers due to difficulties in estimating stack usage. Compilers involve pure logic programming, allowing much of the core logic to be conceived mentally, provided one is not constrained by external specifications (like OS or shell interfaces, which require extensive research). Compilers also make minimal use of system calls, primarily for I/O or memory allocation.

Crucially, Sakai emphasizes the role of the compiler as a distinct layer in the software stack, alongside the CPU, OS kernel, and libraries. Understanding this layer expands the scope of problem-solving, particularly in security. He uses stack protectors as a prime example of a security feature that necessitates a multi-layered, cooperative approach. A stack protector, designed to detect buffer overflows, requires a source of randomness (ideally from the CPU or kernel, not easily guessable software-generated values), and a robust error handling mechanism (e.g., an abort function from a standard library) when a compromise is detected. A compiler, by integrating these checks, acts as a pivotal component that orchestrates interaction between these layers. Existing compilers, driven by performance maximization, are too complex to easily modify for such custom, security-focused integrations, thus necessitating a simpler alternative like Rask.

Key Findings

▶ Watch: Compiler's role in multi-layer security (Stack Protector) (7:30)

The central finding of this talk is the successful proposal and implementation of the Reduced Assembly Set Compiler (RASC), dubbed Rask, which drastically simplifies compiler development and porting for diverse CPU architectures.

1. "One-Day Compiler" Concept: Rask is designed with the singular goal of making compiler porting to new CPU architectures as easy and fast as possible. Sakai demonstrates that a new architecture can be supported by defining as few as 64 assembly patterns, with a practical target of roughly one day for full adaptation.
2. NLCC Implementation: Sakai’s concrete implementation of Rask is called NLCC (Norukushi Compiler). NLCC has successfully been ported to 12 different CPU architectures, a feat Sakai believes is unique for a personal compiler project.
3. Self-Hosting and Ecosystem: NLCC is capable of self-hosting, meaning it can compile its own source code. It also integrates with NLlibC, a standard C library developed with similar principles of simplicity and portability, allowing for a fully closed and self-contained compilation environment.
4. Strategic Simplification: Rask achieves its portability by making strategic trade-offs:

• Limited Register Usage: It uses a minimal, fixed set of registers.
• Restricted Assignment Types: Only register-to-register assignments are directly generated; memory-to-memory assignments are avoided.
• Fixed Prologue/Epilogue: Function entry and exit sequences are standardized and simplified.
• Extensive Built-in Operations: Complex arithmetic (multiplication, division), bitwise operations (XOR, bit inversion), and even logical comparisons are implemented in software (e.g., loops of additions/subtractions, or combinations of simpler logical gates) if a specific hardware instruction is not explicitly provided. This is a critical simplification; simply returning -1 for an instruction tells Rask to use its built-in software implementation.

1. Code Size vs. Portability: While Rask's optimizations can reduce code size by approximately 52% compared to unoptimized output, the resulting binaries are still around 6 times larger than those produced by highly optimized compilers like GCC. This trade-off explicitly prioritizes ease of porting and rapid development over raw code efficiency.
2. Demonstrated Security Integration: Sakai illustrated how easily security features can be integrated at the compiler level by implementing a simple stack protector. This experimental protector bit-flips the return address in the function prologue and reverts it in the epilogue, demonstrating the ease of adding custom, architecture-agnostic security controls.
3. Live Porting Validation: The live demonstration of porting NLCC to a Renesas V850 architecture in approximately 15 minutes (following an initial 2 hours of preparation for the base port) served as a powerful validation of the "one-day compiler" claim, proving the practical speed and simplicity of the Rask methodology.

Technical Deep Dive

▶ Watch: RASK's core: Minimal assembly patterns and built-in functions (15:00)

The core innovation of Rask lies in its architectural design, which deliberately restricts and simplifies the compiler's output to achieve maximum portability. This is a stark contrast to traditional compilers that generate highly optimized, complex assembly code.

1. Rask Design Principles: Prioritizing Portability

Rask's design is driven by the principle of simplification and generalization. Instead of generating highly specific, performance-tuned assembly sequences for every possible operation, Rask limits itself to a minimal, common set of patterns. This makes the "backend" (the part of the compiler that generates assembly for a specific CPU) significantly smaller and easier to write.

2. Limited Register Usage

A fundamental simplification in Rask is the limited use of registers. Most CPU architectures have a multitude of general-purpose and special-purpose registers. Rask restricts itself to using only a few designated registers for general operations, typically two for arithmetic operations (e.g., EAX and EDX in X86, or R10 and R11 in V850 during the demo). This simplifies register allocation and management, as the compiler doesn't need complex algorithms to utilize all available registers optimally.

3. Restricted Assignment Operations

Rask further simplifies code generation by limiting assignment operations. It primarily generates register-to-register moves. Direct memory-to-memory assignments, which can be complex and architecture-specific, are avoided. Instead, any memory operation (load or store) involves moving data between a register and memory, making the pattern consistent across architectures.

4. Standardized Function Prologue and Epilogue

Functions in C require specific setup (prologue) and teardown (epilogue) code. This often involves saving and restoring registers, managing the stack pointer (ESP/SP), and setting up a frame pointer (EBP/FP). Rask standardizes these sequences into a few fixed lines of assembly. For example, on X86, a prologue might involve PUSH EBP and MOV EBP, ESP. This fixed structure means that when porting, only these few, predictable lines need to be translated for the new architecture, rather than implementing complex, variable logic. The demo showed how return addresses and non-volatile registers (like R6-R9 in V850) are saved and restored using these standardized sequences.

5. Extensive Built-in Operations (Software Fallbacks)

This is perhaps the most critical technical aspect of Rask. Many complex operations that CPUs might have dedicated hardware instructions for (e.g., multiplication, division, XOR, bit shifts, negation, sign extension) are, by default, implemented in software as built-ins.

• Arithmetic: Multiplication is performed via repeated addition in a loop. Division is done via repeated subtraction.
• Bitwise/Logical: XOR can be derived from OR and AND operations (A XOR B = (A OR B) - (A AND B)). Bit inversion (NOT) is also handled in software.
• Comparisons: Instead of implementing every possible comparison instruction (e.g., greater than, less than or equal to, not equal), Rask relies on a minimal set (e.g., BE for equality, BLT for less than) and derives others by inverting comparisons or combining results.

The compiler writer signifies that a hardware instruction is not available or desired by simply returning -1 for that instruction definition. Rask then automatically injects the software-based built-in. This dramatically reduces the number of unique assembly instructions a port needs to define. In the V850 demo, Sakai deleted almost all complex arithmetic and logical instructions, relying entirely on Rask's built-ins.

6. Limited Optimization

Rask applies only a restricted set of optimizations. These are typically common, local optimizations that can be chained together to reduce code size. While effective (reducing code size by ~52% in some cases), they are not as aggressive or complex as those found in compilers like GCC, which aim for peak performance. This trade-off results in larger, but more easily verifiable and portable, binaries (approximately 6 times larger than GCC-optimized code).

7. Assembly Pattern Specification

The core of porting Rask to a new architecture involves defining a small set of assembly patterns. For example, to move a value, one might define MOV %1, %0 (move value from operand 1 to operand 0). To get the stack pointer's address, LEA %0, [ESP] might be defined. The speaker showed how these patterns are specified, and how they become the building blocks for code generation. The goal is to define these patterns for basic operations, and let the built-ins handle the rest.

8. NLCC Implementation Details

NLCC, the working implementation of Rask, is written in C. It supports self-hosting and works with NLlibC, a custom standard C library designed with similar principles. The Makefile for NLCC demonstrates its self-hosting capability: first, GCC builds NLCC, then NLCC builds NLlibC, then NLCC (built by NLCC) rebuilds itself, and finally, runs a suite of test programs. This ensures the compiler's robustness and self-sufficiency.

The live porting to Renesas V850 during the talk concretely demonstrated these principles:

• Register Mapping: X86's EAX and EDX were mapped to V850's R10 and R11.
• Value Loading: V850 requires two instructions (MOV_HI, MOV_EA) to load a 32-bit immediate value into a register, which was specified.
• Stack Pointer: ESP was mapped to SP, and stack adjustments used ADI (add immediate) instruction.
• Memory Access: LDW (Load Word) and STW (Store Word) were used for memory loads and stores, often involving base-offset addressing.
• Function Calls: V850 uses JL (Jump and Link) for calls, saving the return address in LP. Parameters were passed in registers R6 through R9.
• Complex Operations: Almost all arithmetic, bitwise, and comparison operations (sign extension, bit inversion, negation, XOR, multiplication, division, shifts, greater than, less than or equal to, etc.) were removed from the V850-specific assembly definitions, allowing Rask's built-ins to handle them in software.
• Branching: Simple BR (Branch) for unconditional jumps and BE (Branch Equal) for conditional jumps were defined.

This live demonstration underscored how Rask's minimalist approach significantly reduces the intellectual burden and line count for porting, making it achievable in a very short timeframe.

Demo / Proof of Concept

▶ Watch: Live demo success and RASK's practical value (36:00)

The talk featured two significant demonstrations that underscored the capabilities of Rask and its implementation, NLCC.

The first demonstration showcased NLCC's self-hosting capabilities and integration with NLlibC. Sakai presented a Makefile sequence:

1. Build NLCC with GCC: The initial NLCC compiler is built using a standard, external compiler (GCC).
2. Build NLlibC with NLCC: The newly built NLCC then compiles NLlibC, Sakai's custom standard C library.
3. Self-Host NLCC: NLCC, now capable of compiling C code, recompiles its own source code. This step is crucial, as it proves the compiler's robustness and ability to handle its own complex C codebase.
4. Run Test Programs: Finally, the self-compiled NLCC is used to compile and run a suite of test programs.

While Sakai noted that the execution of these self-compiled programs was "dramatically slow" compared to GCC-compiled counterparts, the critical takeaway was that it worked. This demonstrated the functional maturity and self-sufficiency of the NLCC compiler and its accompanying library, validating its core premise of being a complete, albeit simplified, toolchain.

The second, and perhaps most impactful, demonstration was the live porting of NLCC to a Renesas V850 architecture. Sakai started with the existing X86 assembly generation code for NLCC. He then proceeded, in real-time, to adapt this code to generate V850 assembly. He explained that he had done some preparatory work (about 1 hour for the base port and 1 hour for debugging) prior to the conference, but the actual on-stage porting took only about 15 minutes.

During this live coding session, Sakai performed the following key steps:

• Register Mapping: He identified V850's common registers (e.g., R10, R11 for general purpose, SP for stack pointer, LP for link pointer) and updated the compiler's internal mappings from X86 equivalents (e.g., EAX, EDX, ESP).
• Instruction Translation: He translated core X86 instructions to their V850 counterparts. For instance, loading an immediate value into a register on V850 required two instructions (MOV_HI for the upper part and MOV_EA for the lower part), which he carefully added. Stack manipulation used V850's ADI (Add Immediate) instruction. Memory access was done via LDW (Load Word) and STW (Store Word) instructions.
• Function Call Conventions: He updated the function call sequence, specifying that parameters are passed in registers R6 through R9 and function calls use the JL (Jump and Link) instruction, which stores the return address in the LP register.
• Reliance on Built-ins: This was a highlight. For complex operations like multiplication, division, XOR, bit shifts, negation, and various conditional comparisons (e.g., greater than, less than or equal to), Sakai deleted the architecture-specific assembly generation code. By doing so, he explicitly instructed NLCC to fall back on its software-based built-in implementations, significantly reducing the amount of V850-specific assembly he needed to write.
• Prologue/Epilogue Adaptation: He adapted the standardized function prologue and epilogue to save and restore V850's LP and non-volatile registers.

Despite a few minor syntax errors during the live coding (which he quickly debugged), the final Hello World program successfully compiled and executed on a V850 emulator. This live demonstration profoundly validated the "one-day compiler" claim, showing that a deep understanding of compiler internals combined with a simplified design can enable remarkably fast adaptation to new CPU architectures.

Finally, Sakai briefly demonstrated a simple stack protector as a proof-of-concept for security feature integration. This experimental protector was implemented by modifying the function prologue to bit-flip the return address before storing it on the stack, and then reversing the bit-flip in the function epilogue before returning. While simple (described as "cryptography that is a bit lacking"), it illustrated how easily custom security logic can be injected directly into the compiled code using Rask's flexible and simplified architecture definition.

Defensive Implications

▶ Watch: Q&A: Easiest CPU architecture to support (RISC-V) (37:00)

The Rask compiler and its NLCC implementation present several significant defensive implications, particularly for the security of embedded systems and the broader field of compiler-based security.

1. Rapid Integration of Custom Security Features: The most direct implication is the ability to quickly integrate bespoke security measures into compiled code for a wide array of CPU architectures. Traditional compiler modification for security features (e.g., Control-Flow Integrity (CFI), Address Space Layout Randomization (ASLR), advanced stack protections) is a highly specialized and time-consuming task, often requiring deep familiarity with the compiler's complex internals and specific ISA nuances. Rask's simplified approach drastically lowers this barrier, allowing security researchers and embedded developers to rapidly prototype and deploy custom protections, even on obscure or proprietary hardware that lacks extensive toolchain support. This means that instead of waiting for upstream compiler support, defenders can proactively implement their own safeguards.

1. Tailored Security for Embedded Systems: Embedded systems often operate under stringent resource constraints and have unique threat models. Rask enables the creation of highly specialized compilers where security, rather than raw performance, is the primary design goal. This allows for the integration of security mechanisms that might be deemed too costly or complex for general-purpose compilers but are critical for specific embedded applications. For instance, simpler, verifiable stack protectors or memory region access checks can be embedded directly into the generated assembly, providing a foundational layer of defense.

1. Enhanced Supply Chain Security: By enabling developers to build and control their own compiler toolchains more easily, Rask offers a potential avenue for improving supply chain security. Organizations can audit and verify their custom compilers more thoroughly than complex general-purpose compilers, reducing the risk of malicious code injection or backdoors introduced at the compiler level. This is particularly relevant in critical infrastructure or national security contexts where trust in the entire software development ecosystem is paramount.

1. Compiler as a Security Layer: Sakai's talk reinforces the concept of the compiler as a crucial layer for security enforcement, alongside the CPU, OS kernel, and standard libraries. Understanding how code is translated from high-level languages to machine instructions provides unique opportunities to inject, verify, and enforce security policies. Rask makes this layer more accessible, allowing security practitioners to leverage it for implementing multi-layered defenses that coordinate across the entire software stack.

1. Prototyping and Research Platform: For security researchers, Rask provides an excellent platform for experimenting with novel compiler-based security techniques. The ease of modifying and porting the compiler allows for rapid iteration on new defensive strategies, testing their efficacy across different architectures without the prohibitive overhead of learning and modifying complex, production-grade compilers. This can accelerate the development of new security primitives and their eventual adoption.

1. Education and Awareness: The "one-day compiler" concept itself demystifies compiler development. By showing how a functional compiler can be built and ported with a relatively small set of rules, it encourages more security professionals to understand the compiler layer. This increased awareness can lead to better vulnerability analysis, more effective exploit mitigation strategies, and the development of more robust software.

While Rask prioritizes portability and simplicity over raw performance (leading to larger and slower binaries), in many security-critical embedded contexts, the ability to rapidly integrate verifiable security features often outweighs the need for peak performance. This trade-off makes Rask a valuable tool in the defender's arsenal.

Key Takeaways

• Simplified Compiler Design: The Rask approach drastically reduces the complexity of compiler development by limiting the generated assembly instruction set to a minimal, common subset.
• Rapid Architecture Porting: New CPU architectures can be supported by defining as few as 64 assembly patterns, enabling adaptation within approximately one day, as demonstrated by the live porting to Renesas V850 in 15 minutes.
• Built-in Operations for Portability: Complex operations (e.g., multiplication, division, XOR) are handled by software-based "built-ins" if specific hardware instructions are not defined, prioritizing portability over raw execution speed.
• Functional Self-Hosting Toolchain: The NLCC implementation of Rask is capable of self-hosting and integrates with NLlibC, creating a complete, albeit simplified, C compilation environment.
• Compiler as a Security Integration Layer: Rask demonstrates that custom security features (e.g., a simple stack protector via return address bit-flipping) can be easily integrated directly into the compiled code by modifying the compiler's output patterns.
• Multi-Layered Security Perspective: The talk emphasizes that effective security requires understanding and coordination across CPU, OS kernel, libraries, and the compiler itself, with Rask making the compiler layer more accessible for defensive innovation.

About the Speaker(s)

Hiroaki Sakai is a Global Jisshi Engineer, known for his deep expertise in CPU architectures and compiler design. He is the author of several technical books, including "Dai Neketsu Assembler Mon" (Great Passionate Assembler Gate), a comprehensive guide to understanding diverse assembly languages. Sakai developed the "Nekketsu Assembler Method" (Passionate Assembler Method), a unique technique for inferring a CPU's instruction set architecture (ISA) by analyzing compiler output, rather than solely relying on official documentation. His extensive practical experience includes reading over 60 different CPU assemblers and successfully running "Hello World" programs on more than 30 CPU simulators. This profound, hands-on understanding of how compilers interact with a vast array of ISAs forms the foundational knowledge that led to his innovative Rask compiler project.