Lightweight Software Kernels and Hardware Extensions for Efficient Sparse Deep Neural Networks on Microcontrollers

Lightweight Software Kernels and Hardware Extensions for Efficient Sparse Deep Neural Networks on Microcontrollers

Speakers: Francesco Daghero, Daniele Jahier Pagliari, Francesco Conti, Luca Benini, Alessio Burrello

Conference: MLSys 2025

YouTube: https://www.youtube.com/watch?v=None

Duration: 1154.0s

Overview

This talk, presented by Francesco Daghero and his colleagues, delves into critical advancements for deploying deep learning models on highly constrained microcontrollers (MCUs). The core challenge addressed is the inherent computational and memory intensity of deep neural networks (DNNs) when targeting ultra-low-power platforms operating within power envelopes of mere tens of milliwatts. The research introduces a dual-pronged approach: developing highly optimized software-only kernels and an exceptionally lightweight hardware extension, both specifically designed to accelerate N:M semi-structured pruning for common sparse operators like convolutions and fully connected layers.

The significance of this work lies in its ability to enable sophisticated AI capabilities directly on edge devices, where power consumption, memory footprint, and cost are paramount. By demonstrating substantial latency reductions (up to 3.1X end-to-end) with negligible accuracy drops and minimal hardware overhead, the presented solutions pave the way for a new generation of intelligent, battery-powered IoT and embedded systems. This talk is crucial for anyone involved in edge AI, embedded systems design, or optimizing ML workloads for extreme resource constraints, offering practical strategies to overcome the formidable barriers of deploying modern DNNs in the tiniest of compute environments.

Background

▶ Watch: Introduction: Challenges of DL on microcontrollers (0:00)

The proliferation of deep learning across various domains has led to an increasing demand for deploying these powerful models on resource-constrained edge devices. However, modern DNNs are typically characterized by a vast number of parameters and high computational requirements, making their direct deployment on microcontrollers (MCUs), which operate under strict power, memory, and area budgets, a significant challenge. These platforms often have memory capacities in the kilobytes to a few megabytes and power envelopes in the tens of milliwatts, leading to issues like high memory footprint, increased latency, and excessive energy consumption.

To mitigate these challenges, various optimization techniques have emerged, including quantization (reducing the precision of weights and activations) and pruning (removing redundant connections or weights). This work specifically focuses on weight pruning, and more precisely, N:M semi-structured pruning. In this widely adopted pattern, weights are examined, and the less important ones are removed. The remaining N elements are stored for every M elements, along with their offsets, effectively compressing the model. The advantages of N:M pruning are compelling: it leads to a lower memory footprint, reduces the number of multiply-accumulate (MAC) operations, and generally incurs only minimal accuracy drops. However, a significant drawback is the introduction of irregular workloads. The sparse, non-uniform access patterns to weights and activations can disrupt the efficient, predictable data flows that dense computations enjoy, often leading to performance penalties on conventional hardware.

Existing solutions for accelerating sparse workloads on MCUs can be broadly categorized into two groups. Software-only solutions typically offer the benefit of zero hardware area overhead, making them attractive for cost-sensitive or fixed-architecture MCUs. However, current software-only approaches often yield limited speedups and generally require very high sparsity ratios to become beneficial, struggling with moderately sparse models. On the other hand, hardware-based solutions, such as dedicated accelerators or additional hardware blocks, can deliver considerable speedups. The major impediment for MCUs, however, is the associated large area overhead and potentially significant increase in manufacturing cost, which directly contradicts the core design principles of these resource-constrained devices. This talk addresses this critical gap, aiming to accelerate common sparse operators on MCUs through either highly optimized software kernels or a hardware extension designed with an extremely limited area overhead, thus bridging the performance gap without incurring the typical costs of hardware acceleration.

Key Findings

▶ Watch: Target platform and dense convolution baseline (3:45)

The research presented by Daghero et al. delivers several pivotal findings that collectively push the boundaries of efficient sparse DNN inference on microcontrollers:

1. High-Performance Sparse Kernels: The team successfully developed highly optimized kernels for common operations like convolutions and fully connected layers that are specifically tailored for N:M semi-structured sparsity. Critically, these kernels, even in their pure software implementation, can outperform dense baselines in terms of latency at moderate to high sparsity ratios (e.g., 1:8 and 1:16). This demonstrates that significant performance gains can be achieved without any hardware modifications, particularly when memory savings are also considered.

1. Novel Lightweight Hardware Extension: A key contribution is the design and implementation of an inexpensive hardware extension, featuring a custom instruction dubbed Xdecimate. This instruction is engineered to drastically reduce the overhead associated with unpacking sparse weight offsets at runtime. The hardware integration is remarkably efficient, incurring only a 5% area overhead relative to a single RISC-V core and less than 1% of the total System-on-Chip (SoC) area.

1. Substantial Performance Boost from Hardware: The Xdecimate instruction provides a profound performance uplift. It reduces the instruction count in the innermost loop by a factor of 10 and makes the hardware-accelerated kernels 1.8 times faster than their software-only counterparts. Furthermore, with this extension, even moderately sparse configurations (e.g., 1:4 sparsity) can outperform the dense baseline, with overall speedups reaching up to 4 times faster than the dense baseline for certain layers.

1. End-to-End Deployment with AI Compiler: The work includes the extension of an existing TVM-based AI compiler, named Match, to be sparsity-aware. This compiler features updated pattern recognition, a sparsity-aware tiling engine, and a hardware cost model, enabling seamless end-to-end deployment of sparse models on heterogeneous edge SoCs. This compiler outputs optimized C code, facilitating practical application.

1. Significant End-to-End Latency Reduction and Memory Savings: Benchmarking on common datasets (CIFAR-10, CIFAR-100) and models (ResNet-18, a bit small) demonstrates impressive end-to-end results. The solutions achieve latency reductions of up to 2X for the 'a bit small' model and 3.1X for ResNet-18 in terms of MACs per cycle. Concurrently, memory reduction is significant, and accuracy drops are kept to a minimum, typically less than 1.5% even for aggressive 1:16 sparsity, showcasing the resilience of the approach.

1. Superior Area-Efficiency Compared to State-of-the-Art: A crucial comparative analysis against other works accelerating sparsity on microcontrollers highlights the superior area-efficiency of this approach. While achieving comparable speedups, the proposed hardware extension does so with a dramatically lower area overhead (e.g., 4.31x speedup with 5% core area overhead versus other solutions achieving similar speedups with 44% area overhead). This makes the solution particularly attractive for cost- and area-sensitive MCU deployments.

Technical Deep Dive

▶ Watch: Introducing Xdecimate hardware extension for sparsity (7:50)

The technical core of this work revolves around optimizing sparse deep neural network inference on a specific RISC-V-based microcontroller architecture, the PULP (Parallel Ultra Low Power) family. This platform is characterized by a cluster of eight RISC-V cores, each equipped with an extended instruction set for deep learning, including 8-bit SIMD (Single Instruction, Multiple Data) hardware loops and post-increment loads. This custom instruction set, predating the official RISC-V Vector ISA, utilizes general-purpose registers for SIMD operations, effectively providing a specialized compute engine.

The starting point for optimization is the PULP-NN library, an open-source collection of highly optimized dense kernels, conceptually similar to ARM's CMSIS-NN. For dense convolutions, PULP-NN employs an output stationary approach. This means the computation iterates over output pixels, performing a partial "into-call" (an inner loop accumulation) for two output pixels at a time due to limited memory. The loops proceed over output channels and then input filters, with the innermost loop responsible for loading weights, activations, and executing dot products using the custom SIMD instructions. The design ensures optimal unrolling to maximize register usage, a critical factor for performance on embedded processors.

Transitioning from dense to sparse (specifically N:M) computations introduces several challenges. Firstly, the N:M sparsity pattern, where N non-zero elements are stored for every M elements, does not guarantee that the offsets of these non-zero elements are shared across different output channels. This loss of offset commonality prevents efficient unrolling over output channels, diminishing potential performance gains. Secondly, the offsets themselves are packed into less than 8 bits for memory efficiency. This necessitates an unpacking overhead at runtime to determine the correct memory addresses for loading sparse data, adding extra instructions and cycles to the critical path.

To address these issues, the researchers developed a two-pronged solution:

1. Software-Only Sparse Kernels:

For the software-only approach, the core strategy maintains the output stationarity and dense into-call structure. However, the innermost loop is modified significantly. Instead of directly loading contiguous dense weights, it now involves:

• Loading the compressed sparse weights.
• Unpacking the offsets: This step reads the packed offset information to identify the positions of the N non-zero elements within the M-element block.
• Loading elements based on offsets: Using the unpacked offsets, the corresponding N non-zero activation elements are loaded into the SIMD-capable vector registers (or general-purpose registers used for SIMD).
• Performing SIMD dot product: The SIMD instruction then executes the dot product on these loaded sparse activations and weights.

While this software-only approach successfully saves memory and, for higher sparsity ratios (1:8, 1:16), even outperforms the dense baseline in latency, analysis revealed a significant bottleneck: the numerous instructions dedicated solely to offset unpacking. This overhead, particularly noticeable at lower sparsity (1:4), limited the overall speedup.

1. Hardware Extension: The Xdecimate Instruction:

To overcome the unpacking bottleneck, an innovative, lightweight hardware extension was designed, introducing a custom instruction called Xdecimate. This instruction is specifically tailored to streamline the sparse data loading and unpacking process.

• Functionality: Given input buffers (B1, B2, representing partial activation blocks) and a set of offsets, the Xdecimate instruction, when called, simultaneously loads and unpacks the correct byte (corresponding to a non-zero element) into a designated vector buffer register (e.g., VB1). Subsequent calls to Xdecimate fill other vector registers (e.g., VB2, then VB1 at a second position), effectively populating the SIMD registers with the scattered non-zero elements in a single cycle. This means the instruction uncompresses the offsets and loads the correct byte into the target register within a single clock cycle.
• Hardware Implementation: The Xdecimate instruction extends the existing instruction decode, execute, and write-back stages of the RISC-V pipeline. The critical aspect of its design is its minimal impact on the core's area. This hardware block occupies only 5% of the area of a single RISC-V core, and less than 1% of the overall SoC area, making it an extremely cost-effective addition for MCUs.
• Performance Impact: The introduction of Xdecimate dramatically reduces the instruction count in the innermost loop by 10 instructions, leading to a 1.8x speedup compared to the software-only sparse kernel. This efficiency boost ensures that even the 1:4 sparse version of the kernel, which was slower in the software-only implementation, now significantly outperforms the dense baseline.

Finally, to enable end-to-end deployment, the research team extended their TVM-based AI compiler, Match. This compiler is designed for heterogeneous edge system-on-chip deployment and incorporates several sparsity-aware features:

• Updated Pattern Recognition: To identify and optimize for N:M sparsity patterns.
• Sparsity-Aware Tiling Engine: To efficiently partition and schedule sparse operations, considering memory constraints and data locality.
• Hardware Cost Model: To accurately estimate the performance of both software-only and hardware-accelerated sparse kernels, guiding optimal deployment decisions.

The output of the Match compiler is optimized C code, ready for execution on the target PULP MCUs. This integrated compiler support is crucial for abstracting away the low-level complexities of sparse kernel implementation and hardware interaction, making the solution accessible for practical deep learning model deployment.

Experimental Setup & Results

▶ Watch: Xdecimate instruction functionality and low area overhead (9:30)

The experimental evaluation of the proposed lightweight software kernels and hardware extensions for sparse DNNs on microcontrollers was conducted rigorously to demonstrate their efficacy across various models, datasets, and sparsity levels.

Hardware Target:

The primary target hardware was a RISC-V System-on-Chip (SoC) from the PULP (Parallel Ultra Low Power) family. This specific SoC features a cluster of eight RISC-V cores. Critically, it includes 128 kilobytes (KB) of L1 memory and 1.6 megabytes (MB) of L2 memory, highlighting the extremely constrained memory environment typical of microcontrollers. This setup allowed for a realistic assessment of the memory and computational efficiency of the proposed solutions.

Datasets and Models:

Benchmarking was performed on two popular image classification datasets:

• CIFAR-10
• CIFAR-100

Two distinct deep neural network models were used:

• ResNet-18: A widely recognized convolutional neural network, representing a more complex model.
• "a bit small": A smaller, more compact model, likely tailored for edge deployment, demonstrating the applicability to highly resource-constrained scenarios.

Sparsity Ratios:

The experiments focused on N:M semi-structured pruning, specifically evaluating configurations where N=1. The sparsity ratios tested were:

• M=4 (1:4 sparsity)
• M=8 (1:8 sparsity)
• M=16 (1:16 sparsity)

The models used in the evaluation were pre-pruned during training, referencing a specific paper for the pruning methodology.

Layer-by-Layer Benchmarks:

Initial benchmarks focused on individual layers to isolate the performance of the sparse kernels. The results showed:

• Software-only Kernels: For 1:4 sparsity, the software-only sparse kernel was less performant than the dense baseline (shown in green in the presentation, indicating a higher instruction count/latency), though it offered memory savings. However, for higher sparsity levels (1:8 and 1:16), the software-only kernels consistently outperformed the dense baseline in terms of latency.
• Hardware-Accelerated Kernels (Xdecimate): The kernels leveraging the Xdecimate instruction demonstrated significant improvements. They were always faster than the dense baseline, even at 1:4 sparsity. In the most favorable cases, these hardware-extended kernels achieved up to 4 times faster execution compared to the dense baseline for specific layers. This highlights the effectiveness of offloading the offset unpacking overhead to a dedicated hardware instruction.

End-to-End Results:

Beyond individual layer performance, the study presented end-to-end results, measuring the overall model execution.

• Multiply-Accumulates per Cycle (MACs/cycle): A key metric for computational efficiency, MACs/cycle showed significant acceleration. Both the software-only and hardware-accelerated instruction set versions substantially improved MACs/cycle. The ResNet-18 model showed even greater acceleration compared to the 'a bit small' model.
• Latency Reduction: End-to-end latency was reduced by up to 2X for the 'a bit small' model and up to 3.1X for ResNet-18. These are substantial improvements for real-time inference on MCUs.
• Accuracy: A critical concern with pruning is accuracy degradation. The results demonstrated that accuracy drops were minimal, less than 1.5% even for the aggressive 1:16 sparsity. This indicates that the N:M pruning strategy, combined with the efficient execution, is "quite resilient" to accuracy loss.
• Memory Reduction: The pruning technique led to a significant memory reduction, which is vital for MCUs with limited on-chip and off-chip memory.

State-of-the-Art Comparison:

A direct comparison was made against other existing works that accelerate sparsity on microcontrollers. This comparison considered various sparsity ratios and, crucially, the hardware area overhead of the proposed solutions.

• The presented work achieved comparable speedups relative to dense kernels at similar sparsity levels as other state-of-the-art solutions.
• However, the key differentiator was area efficiency. While other solutions might offer similar speedups, they often came with a substantial area penalty (e.g., one compared solution had a 44% area overhead relative to a single core). In stark contrast, the proposed hardware extension achieved a 4.31x speedup with only a 5% area overhead relative to a single core. This makes the solution exceptionally efficient in terms of area, which is a paramount concern for MCU design and cost. For the full SoC, the area overhead was even less than 1%.

In summary, the experimental results unequivocally demonstrate that both the optimized software kernels and, particularly, the hardware-accelerated kernels with the Xdecimate instruction, provide significant performance and memory benefits without compromising model accuracy, all while maintaining an extremely low hardware footprint.

Practical Implications

▶ Watch: Performance benchmarks: up to 4x faster than dense (12:00)

The advancements presented in this talk hold profound practical implications for the burgeoning field of edge AI and the deployment of machine learning on highly constrained devices. By tackling the core challenges of memory, latency, and energy consumption in microcontrollers (MCUs), this work offers tangible benefits for practitioners, infrastructure teams, model builders, and deployers.

For Practitioners and Deployers:

• Enabling Complex AI on the Edge: The most significant implication is the ability to deploy more complex and accurate deep neural network models on ultra-low-power, battery-operated devices that were previously too resource-limited. This opens up new possibilities for on-device intelligence in IoT, wearables, smart sensors, and other embedded applications where cloud connectivity is intermittent or undesirable due to latency, privacy, or energy concerns.
• Reduced Total Cost of Ownership: By allowing models to run efficiently on lower-cost, lower-power MCUs, the total cost of ownership for AI-enabled edge devices can be significantly reduced. This includes both the bill-of-materials (BOM) cost of the hardware and the operational costs associated with power consumption.
• Flexible Deployment Strategies: The dual offering of highly optimized software-only kernels and a lightweight hardware extension provides flexibility. For existing MCUs with fixed hardware, the software kernels offer immediate performance gains for sparse models. For new hardware designs or custom silicon, the Xdecimate instruction provides a powerful, area-efficient acceleration option.
• Longevity and Battery Life: The substantial reductions in latency and computational instructions directly translate to lower energy consumption per inference, thereby extending the battery life of portable and autonomous edge devices.

For Infrastructure Teams and Hardware Designers:

• Guidance for Custom Hardware Design: The detailed analysis of the Xdecimate instruction and its minimal area overhead (5% of a core, <1% of SoC) provides a strong case study and blueprint for designing highly efficient, application-specific hardware accelerators for sparse workloads. This demonstrates that significant performance benefits can be achieved with very small hardware footprints, challenging the notion that hardware acceleration always implies large, expensive blocks.
• Sparsity-Aware Toolchain Integration: The extension of the TVM-based Match compiler with sparsity-aware features (pattern recognition, tiling engine, cost model) is crucial. It highlights the necessity of co-designing hardware/software optimizations with the entire ML deployment toolchain. This reduces the burden on developers, allowing them to focus on model design rather than intricate low-level optimizations.

For Model Builders:

• Incentive for Pruning: The demonstrated performance gains and minimal accuracy drops (less than 1.5% for 1:16 sparsity) provide a strong incentive for model builders to adopt N:M semi-structured pruning as a standard optimization technique for edge deployments. Knowing that the target hardware/software stack can efficiently execute these sparse models empowers them to design more compact and efficient networks.
• Understanding Tradeoffs: Model builders can better understand the tradeoffs between sparsity levels, accuracy, and the resulting performance on target hardware. For instance, a 1:4 sparse model might achieve higher accuracy but require the hardware extension for optimal performance, whereas a 1:16 sparse model might run efficiently even with software-only kernels.

Tradeoffs and Limitations:

• Platform Specificity: The current implementation of the hardware extension is tailored for the RISC-V PULP architecture and its custom SIMD instruction set. While the underlying principles (e.g., accelerating offset unpacking) are general, direct portability to other MCU architectures (e.g., ARM Cortex-M) would require re-implementation or different hardware extensions specific to those platforms.
• Compiler Dependency: While the Match compiler simplifies deployment, it also introduces a dependency on this specific toolchain. Broader adoption might require integrating similar sparsity-aware optimizations into more generalized ML compilers.
• Sparsity Pattern: The work focuses on N:M semi-structured sparsity. While this is a popular and efficient pattern, other sparsity patterns (e.g., unstructured, block sparsity) might require different or additional optimization techniques.
• Training Complexity: While the models are assumed to be pre-pruned, the process of training and pruning models to achieve optimal sparsity without significant accuracy loss can still add complexity to the model development workflow.

In conclusion, this research provides a comprehensive and highly effective solution for deploying sparse deep neural networks on microcontrollers. By offering a spectrum of software-to-hardware optimizations, it addresses critical constraints and paves the way for a more pervasive and efficient integration of AI into the fabric of everyday embedded systems.

Key Takeaways

• N:M Semi-Structured Pruning is Effective for MCUs: The research validates N:M semi-structured pruning as a powerful technique to reduce memory footprint and computational load for DNNs on microcontrollers, achieving significant memory savings and competitive performance.
• Software-Only Sparse Kernels Offer Immediate Gains: Highly optimized software kernels alone can outperform dense baselines in latency for higher sparsity ratios (e.g., 1:8, 1:16), providing a zero-overhead solution for existing MCU platforms.
• Xdecimate Hardware Instruction Drastically Improves Performance: A novel, lightweight hardware instruction, Xdecimate, efficiently handles sparse data unpacking, leading to up to 4x speedup over dense baselines and 1.8x acceleration over software-only sparse kernels, even for lower sparsity (1:4).
• Exceptional Area-Efficiency for Hardware Acceleration: The Xdecimate hardware extension adds only 5% area overhead to a single RISC-V core (less than 1% of the SoC), making it a highly cost-effective and practical solution for resource-constrained microcontrollers, outperforming state-of-the-art alternatives in efficiency.
• End-to-End Deployment with Minimal Accuracy Loss: Integration with a sparsity-aware AI compiler (Match) enables end-to-end deployment, achieving impressive latency reductions (up to 3.1X) on models like ResNet-18 with negligible accuracy drops (less than 1.5% for 1:16 sparsity).
• Enabling Advanced AI for Extreme Edge Devices: This work provides critical tools and methodologies for deploying complex deep learning models on ultra-low-power, memory-constrained microcontrollers, expanding the possibilities for intelligent IoT and embedded systems.

About the Speaker(s)

The talk "Lightweight Software Kernels and Hardware Extensions for Efficient Sparse Deep Neural Networks on Microcontrollers" was presented by Francesco Daghero, with contributions from Daniele Jahier Pagliari, Francesco Conti, Luca Benini, and Alessio Burrello. While specific biographical details were not provided in the transcript, the collective expertise of the speakers is evident in the depth and innovation of the research. Their work focuses on the challenging domain of bringing advanced AI/ML capabilities to highly resource-constrained edge devices, specifically microcontrollers. The detailed technical content, ranging from custom instruction set architecture extensions to compiler optimizations and comprehensive benchmarking, reflects a strong background in embedded systems, computer architecture, and deep learning optimization for low-power platforms. Their affiliation with the MLSys 2025 conference signifies their standing as leading researchers in the field of machine learning systems.