MEADOW: Memory-efficient Dataflow and Data Packing for Low Power Edge LLMs

MEADOW: Memory-efficient Dataflow and Data Packing for Low Power Edge LLMs

Speakers: Abhishek Moitra, Arkapravo Ghosh, Shrey Agrawal, Karthik Swaminathan, Priyadarshini Panda

Conference: MLSys 2025

YouTube: https://www.youtube.com/watch?v=None

Overview

The rapid scaling of Large Language Models (LLMs) has unlocked a vast array of applications, from sophisticated chatbots to real-time translation systems. However, deploying these increasingly massive models on resource-constrained edge devices presents significant challenges, primarily due to their immense computational and memory demands. This talk by Abhishek Moitra and collaborators from IBM Research introduces MEADOW, a novel framework designed to address the memory bottleneck inherent in deploying LLMs on low-power edge hardware, specifically Field-Programmable Gate Arrays (FPGAs).

MEADOW proposes two orthogonal optimization strategies: a lossless weight packing method and a Token Parallel Head Sequential (TPHS) dataflow. Unlike traditional approaches that often rely on aggressive quantization, MEADOW's weight packing preserves model accuracy while significantly reducing the memory footprint and fetch latency of model weights. Concurrently, the TPHS dataflow re-architects the attention computation to maximize on-chip data reuse, thereby minimizing costly off-chip data transfers. This work is crucial for enabling the proliferation of advanced AI capabilities to edge devices where power and memory bandwidth are severely limited, offering a path to deploy models like LLaMA 1.3 billion parameters with practical latency.

Background

▶ Watch: Introduction to LLM scaling and edge challenges (0:00)

Over the past few years, LLMs have witnessed an unprecedented scale-up, with their parameter counts and operational complexity growing by millions and trillions, respectively. This exponential growth has been largely driven by advancements in hardware following Moore's Law and the availability of high-bandwidth memory (HBM) solutions on high-performance accelerators. However, the paradigm shifts dramatically when targeting low-power edge devices.

LLM inference typically involves two distinct stages: the pre-fill stage and the decode stage. During pre-fill, the model processes an entire input prompt, which can consist of many tokens, to establish context. This stage often involves large intermediate outputs. In contrast, the decode stage generates tokens one by one, making its intermediate outputs much smaller.

Traditional hardware acceleration for deep learning, especially on GPUs, often relies on General Matrix Multiply (GEMM) operations. The GEMM dataflow involves fetching weights and inputs from off-chip DRAM to the compute core, performing matrix multiplication, and then storing the outputs back to DRAM. This process leads to substantial on and off-chip memory access, creating a significant data movement bottleneck, particularly for the large matrices involved in LLMs. While HBMs provide high bandwidth to mitigate this, their high power consumption makes them unsuitable for edge deployments where strict power constraints are paramount.

When LLMs are deployed on low-power edge devices like the Zynq ZCU 102 FPGA, the memory bottleneck becomes acutely apparent. In the pre-fill stage, the large intermediate outputs necessitate frequent transfers between DRAM and the compute core, leading to the attention computation consuming a major portion of the latency. This is primarily due to the repeated fetching of weights and storage/fetching of intermediate outputs. For the decode stage, while intermediate outputs are smaller, the latency is still dominated by fetching the KV caches (key and value caches for attention) and weight matrices. This highlights a critical challenge: on edge devices, memory access, not raw compute, is often the primary performance bottleneck, demanding a shift in optimization strategies from compute-centric to memory-centric approaches.

Key Findings

▶ Watch: Analyzing LLM latency on low-power FPGAs (4:30)

The MEADOW framework introduces a dual-pronged approach that effectively tackles the memory bottlenecks prevalent in edge LLM deployment, yielding significant performance improvements without compromising model accuracy. The core contributions and findings are:

1. Lossless Weight Compression: MEADOW proposes a novel lossless weight packing strategy that is distinct from traditional quantization. By identifying and encoding redundant patterns within the weight matrices, it significantly reduces the size of the weights that need to be fetched from memory. This method, combined with a frequency-aware re-indexing technique (akin to Huffman encoding), leads to a denser packing and substantially minimizes the weight fetch latency. An ablation study demonstrated that this packing, when applied to an already 4-bit quantized model, could achieve an additional 2.6 times reduction in latency, primarily by increasing the occurrence of low-precision index values, thus enabling more efficient memory access. Crucially, this method avoids the accuracy loss typically associated with post-training quantization.

1. Token Parallel Head Sequential (TPHS) Dataflow: To optimize the attention computation and reduce on/off-chip data transfers, MEADOW introduces the TPHS dataflow. This approach re-architects how attention heads are processed and how data flows between layers. By ensuring that the outputs of one layer are immediately utilized by the subsequent layer and by employing specialized processing elements (PEs) for pipelined operations, TPHS minimizes the need to store and refetch large intermediate outputs from off-chip DRAM. This strategy is particularly effective in reducing the latency of attention computation during the pre-fill stage.

1. End-to-End Latency Reduction: When evaluated on a Zynq ZCU 102 FPGA with LLaMA 125 million and 1.3 billion parameter models, MEADOW achieved an overall 1.5 times reduction in end-to-end latency (encompassing both pre-fill and decode stages) compared to an 8-bit quantized GEMM baseline. This substantial speedup is attributed to the combined effects of the TPHS dataflow and the weight packing strategy.

1. Component-Specific Gains: The TPHS dataflow was found to be highly effective in reducing the attention computation latency, particularly during the pre-fill stage where large intermediate outputs are common. The lossless weight packing further contributed an additional 1.8 times reduction in latency by accelerating the fetching of model weights.

1. Memory-Bound Optimization: A key insight from MEADOW's evaluation is its effectiveness in memory-bound scenarios. Unlike many prior works that assume high-bandwidth memory (like HBM) and thus focus on compute-centric optimizations, MEADOW excels in environments where memory access is the primary bottleneck. This underscores the importance of memory-aware design for low-power edge systems and suggests that the optimal dataflow strategy (e.g., TPHS vs. GEMM) depends critically on where the system resides on the roofline model – memory-bound or compute-bound.

Technical Deep Dive

▶ Watch: Detailed explanation of lossless weight compression (7:00)

The MEADOW framework’s technical prowess lies in its two orthogonal yet complementary optimization techniques: a novel lossless weight compression scheme and a tailored dataflow for attention computation.

Lossless Weight Compression

The weight compression method in MEADOW targets the inherent redundancies within the large weight matrices of LLMs to reduce their memory footprint without any loss of precision. The process unfolds in several steps:

1. Chunking: An N x M sized weight matrix is first divided into smaller chunks, each containing C elements. This allows for localized redundancy detection.
2. Unique Matrix Creation: For each chunk, the system identifies repeating vectors or patterns. These unique patterns are then compiled into a unique matrix. This unique matrix serves as a dictionary of frequently occurring weight sub-patterns.
3. Index Encoding: Instead of storing the original, full weight matrix, MEADOW encodes it using index values that point to entries in the unique matrix. Essentially, the original weight matrix is represented as a combination of the unique matrix and a corresponding index matrix. This means that for inference, instead of fetching the entire N x M weight matrix, the system fetches the smaller unique matrix and the index matrix, significantly reducing data transfer volume.
4. Frequency-Aware Re-indexing: To further enhance packing density, a frequency-aware re-indexing step is applied. This technique is conceptually similar to Huffman encoding, where more frequently occurring index values are assigned shorter bit representations. By increasing the occurrences of these low-precision index values, the overall bit-width required to represent the index matrix is reduced. This leads to a denser packing of the index matrix, which translates directly to fewer cycles required to fetch the weight matrices from memory. The core motivation for this approach, as highlighted by the speakers, is to bypass the accuracy degradation and computational overhead associated with traditional post-training quantization, offering a lossless alternative for weight size reduction.

MEADOW Architecture and Token Parallel Head Sequential (TPHS) Dataflow

The MEADOW architecture is designed to support the Token Parallel Head Sequential (TPHS) dataflow efficiently on FPGA hardware. It features a tiled architecture with several specialized components:

1. Off-chip DRAM: This serves as the main memory for storing model parameters and intermediate activations.
2. Compute Array: This is the heart of the processing, containing different types of processing elements (PEs) tailored for specific operations:

• Broadcasting PEs: These PEs are crucial for operations where a single value needs to be distributed across multiple computations, such as in softmax-V operations. They enable a fine-grained, pipelined computation.
• Parallel PEs: These handle general parallel computation tasks, distributing workload across the array.

1. Pipeline Softmax Units: These dedicated units are designed to execute softmax and other preceding operations in a pipelined manner, maximizing throughput and reducing idle cycles.
2. Layer Normalization and Non-linear Operation Units: Specialized hardware units are included to efficiently perform these common LLM operations.
3. Weight BRAM (Block RAM): On-chip BRAMs are used to store the compressed weight matrices, providing much faster access than off-chip DRAM.
4. Unpack Module (WILLYU): This crucial module sits between the Weight BRAM and the PEs. During execution, it dynamically unpacks the compressed weights from the BRAM and spatially maps them to the appropriate processing elements in real-time. This ensures that the benefits of weight packing are realized during actual computation.

The TPHS dataflow itself is central to MEADOW's efficiency. Its key principle is maximizing data reuse and minimizing off-chip data transfers. Instead of fetching data from DRAM for each new operation, TPHS ensures that the outputs generated by one layer or sub-operation are immediately utilized as inputs by the next layer or subsequent operation. This "pipeline" effect is particularly effective for the attention computation, where intermediate outputs can be substantial. The use of broadcasting PEs allows for the softmax-V operations to be computed in a highly pipelined and fine-grained manner. By keeping data on-chip for as long as possible and orchestrating a seamless flow between computational stages, TPHS significantly reduces the latency associated with memory access and transfer, especially for the attention mechanism's intermediate outputs and KV caches.

Experimental Setup & Results

▶ Watch: Ablation study: weight packing latency reduction (9:30)

To validate the efficacy of the MEADOW framework, a series of experiments were conducted, focusing on its performance on a representative low-power edge device.

Hardware Platform: The primary experimental platform was the Zynq ZCU 102 FPGA. This choice is critical as FPGAs represent a prevalent class of programmable edge hardware, characterized by strict power budgets and limited memory bandwidth compared to high-end GPUs.

Models Evaluated: The experiments utilized two popular Large Language Models from the LLaMA family:

• LLaMA 125 million parameters
• LLaMA 1.3 billion parameters

These models were chosen to demonstrate MEADOW's scalability across different model sizes relevant for edge deployment.

Baselines: MEADOW's performance was rigorously compared against a standard GEMM (General Matrix Multiply) baseline. For a fair comparison, the baseline LLMs were implemented with 8-bit quantization, a common practice for reducing model size and accelerating inference on hardware.

MEADOW Implementation: In MEADOW's implementation, the same LLaMA models were also subjected to 8-bit quantization. However, on top of this, MEADOW incorporated its two core innovations: the Token Parallel Head Sequential (TPHS) dataflow and the lossless weight packing strategy. This setup allowed for direct comparison of the benefits derived from MEADOW's unique contributions.

Key Metrics: The primary performance metric was end-to-end latency, which includes the cumulative time taken for both the pre-fill and decode stages of LLM inference.

Headline Results:

• Overall Latency Reduction: MEADOW achieved an impressive 1.5 times reduction in end-to-end latency for both LLaMA 125M and 1.3B models when compared to the 8-bit quantized GEMM baseline on the Zynq ZCU 102 FPGA.
• Contribution Breakdown:
• The TPHS dataflow was identified as the primary contributor to latency reduction, particularly in optimizing the attention computation during the pre-fill stage. This stage, characterized by large intermediate outputs, greatly benefits from TPHS's on-chip data reuse strategy.
• The lossless weight packing strategy provided a further, significant boost, contributing an additional 1.8 times reduction in latency. This highlights the effectiveness of minimizing weight fetch overhead.
• Ablation Study for Weight Packing: A specific ablation study demonstrated the power of the weight packing method. When applied to an already 4-bit quantized weight matrix, the combination of packing and frequency-aware re-indexing led to an almost 2.6 times reduction in latency. This was attributed to the denser packing achieved by increasing the prevalence of low-precision values in the weight representation, thereby reducing the cycles needed to fetch weights.

Comparison with Prior State-of-the-Art (SOTA):

The talk emphasized that MEADOW's strength lies in its ability to minimize memory access overhead, a critical factor for edge devices. Many prior SOTA LLM optimization strategies, while effective, often assumed the availability of high-bandwidth memory (HBM) – a luxury not afforded to low-power edge platforms. Consequently, these prior works frequently focused on compute-bound bottlenecks. MEADOW, by contrast, explicitly addresses the memory-bound nature of edge inference, allowing it to outperform solutions that do not prioritize memory efficiency as rigorously.

Roofline Model Implications:

The experimental findings reinforced the importance of the roofline model in guiding optimization choices. The presenters noted that the TPHS dataflow is particularly advantageous in memory-bound situations, where memory bandwidth is the limiting factor. Conversely, in compute-bound scenarios (e.g., on high-end GPUs with HBM), traditional GEMM operations might still be preferred because they effectively distribute computational load across many processing elements within a layer. This nuanced understanding underscores that there is no one-size-fits-all solution, and the optimal strategy depends on the underlying hardware's performance characteristics.

Practical Implications

▶ Watch: MEADOW framework summary and experimental overview (14:00)

The MEADOW framework offers several compelling practical implications for various stakeholders involved in the deployment of AI/ML models, particularly in the burgeoning field of edge AI.

For Practitioners and Deployers: MEADOW provides a tangible pathway to deploy larger and more capable LLMs, such as the LLaMA 1.3 billion parameter model, on power-constrained edge devices like FPGAs. This unlocks new possibilities for on-device intelligence, enabling applications like real-time, privacy-preserving chatbots or translation services without reliance on cloud connectivity. The achieved 1.5x end-to-end latency reduction translates directly into a more responsive user experience for edge applications. It means that developers are no longer strictly limited to highly compressed, smaller models that might sacrifice accuracy or capabilities.

For Infrastructure Teams and Hardware Designers: This work highlights a critical shift in focus for edge ML infrastructure. Instead of solely chasing higher computational throughput (FLOPS), the emphasis must increasingly be on memory-centric optimizations. For edge devices, memory bandwidth and access latency are often the true bottlenecks, not just the raw processing power. Infra teams can leverage MEADOW's insights to design or select hardware architectures that prioritize efficient data movement, on-chip memory utilization, and specialized dataflow engines over brute-force compute. The WILLYU unpack module and specialized PEs within MEADOW demonstrate how custom hardware logic can be designed to support memory-efficient operations.

For Model Builders and Algorithm Developers: MEADOW introduces a valuable lossless weight compression method that complements or can even serve as an alternative to quantization. For scenarios where maintaining model accuracy is paramount, but memory footprint reduction is also necessary, this technique offers a powerful tool. It allows model builders to explore redundancy in weights without the accuracy trade-offs often associated with aggressive quantization. This could lead to new avenues for designing intrinsically memory-efficient model architectures from the ground up.

Tradeoffs and Limitations: While highly effective, MEADOW's approach also comes with certain tradeoffs. The custom dataflow and specialized hardware units (like broadcasting PEs and the unpack module) imply a higher design complexity compared to simply running a quantized model on a standard GEMM engine. Integrating such custom logic onto an FPGA requires significant hardware design expertise. Furthermore, while excellent for memory-bound scenarios, MEADOW acknowledges that for compute-bound systems (e.g., high-end GPUs with abundant HBM), a standard GEMM approach might still be more performant due to its ability to extensively parallelize computations. The current work focuses on FPGAs; its direct applicability and efficiency on other edge platforms, such as custom ASICs or microcontrollers, would require further investigation and potentially platform-specific adaptations. The overhead of the unpack module, while designed to be efficient, is still a component in the critical path.

In essence, MEADOW provides a blueprint for making LLMs truly ubiquitous by overcoming the memory wall on low-power edge devices, pushing the boundaries of what's possible for on-device AI.

Key Takeaways

• Deploying Large Language Models (LLMs) on low-power edge devices is primarily bottlenecked by memory bandwidth and access latency, not just raw computational power.
• MEADOW introduces a novel, lossless weight packing strategy that effectively compresses model weights by exploiting redundancies and using frequency-aware re-indexing, significantly reducing weight fetch latency without compromising accuracy.
• The Token Parallel Head Sequential (TPHS) dataflow is a key innovation, optimizing attention computation by maximizing on-chip data reuse and minimizing costly off-chip data transfers, especially during the pre-fill stage.
• The combined MEADOW framework achieves a substantial 1.5 times reduction in end-to-end latency for LLaMA 125M and 1.3B models on a Zynq ZCU 102 FPGA compared to an 8-bit quantized GEMM baseline.
• The effectiveness of dataflow optimizations is context-dependent: TPHS is highly beneficial in memory-bound scenarios typical of edge devices, while traditional GEMM may be preferred in compute-bound situations with high-bandwidth memory.
• Future advancements in edge machine learning must prioritize memory-aware hardware architectures and dataflow optimizations to enable the widespread deployment of increasingly complex LLMs.

About the Speaker(s)

The talk was presented by Abhishek Moitra, who collaborated with Arkapravo Ghosh, Shrey Agrawal, Karthik Swaminathan, and Priyadarshini Panda on this research. The work was conducted in collaboration with IBM Research, indicating a strong foundation in industrial research and development. The team's expertise lies in addressing the challenges of deploying large-scale AI models on resource-constrained hardware, with a particular focus on memory efficiency and novel dataflow strategies for low-power edge implementations.