Efficient On-Device Machine Learning with a Biologically-Plausible Forward-Only Algorithm

Efficient On-Device Machine Learning with a Biologically-Plausible Forward-Only Algorithm

Speakers: Baichuan Huang, PhD Candidate, Lund University; Amir Aminifar, Professor, Lund University

Conference: MLSys 2025

YouTube: https://www.youtube.com/watch?v=None

Overview

This talk introduces BioFO (Biologically Plausible Forward-Only Algorithm), a novel approach to training neural networks designed to address the significant energy consumption and biological implausibility inherent in traditional backpropagation (BP) methods. Presented by Baichuan Huang from Lund University, the research highlights the urgent need for more sustainable and efficient machine learning paradigms, particularly for on-device applications. With the global average temperature rising and large language models (LLMs) like GPT-4 consuming staggering amounts of energy—48 times more than GPT-3 for training—the environmental impact of AI development is becoming a critical concern.

BioFO draws inspiration from the human brain, which operates with remarkable energy efficiency (around 20 watts) compared to today's power-hungry AI models. The core innovation lies in its "forward-only" nature, which eliminates the need for error backpropagation across long distances in the network. This design choice tackles several long-standing biological implausibility issues of backpropagation, such as the locking, non-locality, weight transport, and frozen activities problems, which contribute to its computational and memory inefficiencies.

The significance of BioFO extends beyond theoretical biological plausibility. The algorithm demonstrates superior performance compared to other forward-only algorithms in terms of classification accuracy, memory efficiency, convergence rate, and energy consumption, while maintaining performance close to backpropagation. This makes BioFO a compelling alternative for deploying and training machine learning models on resource-constrained edge devices and for fostering a more sustainable future for AI development.

Background

▶ Watch: Introduction and problem: AI's energy consumption (0:00)

The rapid advancements in machine learning, particularly with the proliferation of large language models, have brought unprecedented capabilities but also significant challenges. A primary concern is the escalating energy consumption associated with training these colossal models. As highlighted in the talk, the energy requirements for models like GPT-4 are astronomical, consuming approximately 48 times more energy than GPT-3 during training, leading to a 13-fold increase in carbon dioxide emissions. This trend is unsustainable and underscores a critical need for more energy-efficient learning algorithms.

In stark contrast to these energy-intensive AI models, the human brain operates with remarkable efficiency, consuming only about 20 watts while continuously learning and adapting. This biological efficiency has spurred research into biologically plausible algorithms that can mimic the brain's learning mechanisms to develop more sustainable AI. The dominant algorithm for training artificial neural networks today is backpropagation (BP). While incredibly effective, BP faces several fundamental issues, particularly concerning its biological plausibility and computational overhead.

Let's dissect the process of backpropagation using a four-layer network example (input X, label Y):

1. Forward Pass: Input X is fed through the network, layer by layer, generating activations (H1, H2, etc.) and ultimately an output.
2. Error Calculation: The network's output is compared with the true label Y to compute an error.
3. Backward Pass: This error is then propagated backward through the network, from the last layer to the hidden layers, to calculate gradients for each weight. These gradients are subsequently used to update the weights.

The problems with backpropagation, particularly from a biological perspective, are multifaceted:

• Locking Issue: This refers to the sequential nature of BP. A new sample cannot begin its forward pass until the previous sample has completed its entire forward pass, backward pass, and weight update. This synchronization requirement makes parallel processing challenging and inefficient, especially for continuous learning.
• Non-Locality Problem: To compute the gradient for weights in an early layer (e.g., W2), the error signal must propagate all the way back from the final output layer. This means the gradient calculation for a specific weight is not localized to the information available at that neuron or layer, requiring global knowledge.
• Weight Transport Problem: In the forward pass, a weight matrix (e.g., W3) is used to compute activations. In the backward pass, its transpose (W3^T) is often required to propagate error signals. The existence of distinct forward and backward weight matrices (or the need for weight transposition) is not observed in biological neural networks, where synaptic strengths are typically symmetric or at least not requiring a transposed version for feedback.
• Frozen Activities: To compute gradients during the backward pass, all activations generated during the forward pass (e.g., H1, H2) must be stored. This requirement for storing intermediate activations and parameters significantly increases the memory footprint, especially for deep networks, making on-device training or training very large models memory-intensive.

These inherent limitations of backpropagation—both in terms of biological plausibility and practical resource consumption—motivate the exploration of alternative learning algorithms like the biologically plausible forward-only methods, which aim to overcome these challenges and pave the way for more efficient and sustainable AI.

Key Findings

▶ Watch: Introducing BioFO: Biologically plausible forward-only algorithm (4:10)

The research on BioFO presents several compelling findings that position it as a significant advancement in efficient and biologically plausible machine learning:

• Superior Classification Performance: BioFO consistently outperforms other state-of-the-art forward-only algorithms, including DRTP, Papita, and Hinton's Forward-Forward (FF) algorithm, across a diverse range of datasets. These datasets include common benchmarks like MNIST, CIFAR-10, CIFAR-100, and Mini-ImageNet, as well as real-world wearable application datasets such as CHB-MIT and MIT-BH, where complexity and energy consumption are critical constraints.
• Performance Competitive with Backpropagation: Despite its forward-only nature and biological plausibility advantages, BioFO achieves classification performance remarkably close to that of traditional backpropagation. This demonstrates that efficiency and biological inspiration do not necessarily come at the cost of accuracy.
• Significant Memory Efficiency: A major contribution of BioFO is its substantial reduction in training memory requirements. By eliminating the need to store all intermediate activations from the forward pass for backward gradient computation, BioFO saves at least three times the training memory compared to backpropagation in a four-layer network. This makes it highly suitable for memory-constrained environments like edge devices.
• Faster Convergence Rate: BioFO exhibits a faster convergence rate compared to other forward-only algorithms like Papita and FF. This means models trained with BioFO can reach a stable performance level more quickly, reducing overall training time.
• Enhanced Energy Efficiency: Measured on a Jetson Nano (a popular platform for on-device AI), BioFO significantly outperforms other forward-only algorithms in terms of energy consumption. This is a direct consequence of its local learning mechanism and reduced computational overhead, making it ideal for sustainable and battery-powered applications.
• Architectural Flexibility: The algorithm is easily extensible to various network architectures. While it performs well with fully connected networks, its performance is further improved when extended to locally connected networks and dramatically enhanced when applied to convolutional networks. This adaptability, facilitated by a sparsity mask matrix, allows BioFO to leverage powerful architectural designs, even if the biological plausibility of specific elements like weight sharing in CNNs is an ongoing discussion.

These key findings collectively demonstrate BioFO's potential to bridge the gap between high-performance machine learning and the demands of energy efficiency and biological plausibility, offering a viable path for the next generation of AI systems.

Technical Deep Dive

▶ Watch: BioFO's superior classification performance (6:30)

The core innovation of BioFO lies in its departure from the global error backpropagation mechanism of traditional neural networks, instead embracing a biologically plausible forward-only learning paradigm. This approach directly addresses the four major biological implausibility issues of backpropagation: locking, non-locality, weight transport, and frozen activities.

Let's break down the technical aspects of BioFO:

1. Forward-Only Learning Mechanism:

Unlike backpropagation, which computes a single global loss and propagates its gradient backward through all layers, BioFO adopts a local learning strategy. For each hidden layer in the network, a separate, local learning objective is established.

Consider a multi-layer network with input X and layers generating activations H1, H2, ....

• Local Auxiliary Classifiers: For each hidden layer's activation (e.g., H1), BioFO introduces an auxiliary classifier. This classifier does not operate on the network's final output but rather on the intermediate activation of its respective layer.
• Fixed Random Projection Matrix (B): Before being fed into the auxiliary classifier, the activation H1 is projected through a fixed random projection matrix (B). This matrix B is typically initialized randomly and remains constant during training. The projection H1 * B transforms the activation into a "logic" space suitable for classification.
• Local Loss Function: For each auxiliary classifier, a local loss function is computed. The speaker clarified that this typically involves a cross-entropy loss function between the projected "logic" (from H1 * B) and the true label Y.
• Local Gradient Update: Crucially, this local loss is then used to compute gradients and update only the weights directly feeding into that specific hidden layer (e.g., W1 for H1). This means W1 is updated based on a loss derived directly from H1, rather than waiting for an error signal to propagate from the final output layer. This process is then repeated sequentially for subsequent layers (e.g., H2 uses its own B and local loss to update W2).

This local, layer-wise learning mechanism directly addresses the biological implausibility issues:

• No Locking: Since each layer can learn and update its weights based on locally available information and the global ground truth, there's less reliance on the completion of a full forward-backward pass for the entire network. This allows for more asynchronous and parallel learning.
• Non-Locality: Gradients are generated locally for the weights of a specific layer, eliminating the need for error signals to travel from the final output layer.
• No Weight Transport: The algorithm does not require the transpose of weight matrices for backward passes, as there are no traditional backward passes in the BP sense.
• No Frozen Activities: Intermediate activations from previous layers do not need to be stored for a subsequent backward pass. Once an activation H_i has been used to calculate its local loss and update W_i, it can be discarded, significantly reducing memory overhead.

2. Architectural Flexibility with Sparsity Mask Matrix:

BioFO incorporates a sparsity mask matrix to enable its application to various network architectures beyond simple fully connected layers.

• Fully Connected Networks: If all elements in the mask are positive, the network behaves as a standard fully connected network.
• Locally Connected Networks: By defining specific patterns within the mask, BioFO can be extended to locally connected networks, where neurons only connect to a limited receptive field in the previous layer.
• Convolutional Networks: The most significant architectural extension is to convolutional networks. This is achieved by incorporating weight sharing mechanisms within the sparsity mask. While the speaker acknowledges that weight sharing in convolutional networks is often considered "not biologically plausible" in strict terms, its inclusion drastically improves performance (as seen in the experimental results). This highlights a pragmatic trade-off between strict biological fidelity and practical effectiveness in deep learning.

3. Loss Function Detail:

An audience question specifically inquired about the loss function. The speaker clarified that for each local auxiliary classifier, the output (logic) from the fixed random projection is combined with the true label Y to compute a cross-entropy loss function. This means that each layer effectively tries to classify the input based on its current representation, guided by the global ground truth Y.

In essence, BioFO re-imagines neural network training by decentralizing the learning process. Instead of a single, global error signal dictating updates across the entire network, each layer learns locally, informed by the overall task objective (via the ground truth label Y) but without the computational and biological overhead of backpropagation. This leads to a more efficient, memory-friendly, and biologically inspired learning algorithm.

Experimental Setup & Results

▶ Watch: Significant memory efficiency gains (7:05)

The evaluation of BioFO was comprehensive, involving a variety of datasets, comparisons against leading baselines, and measurements across critical performance metrics, including classification accuracy, memory usage, convergence speed, and energy consumption.

1. Datasets:

The researchers utilized a mix of standard academic benchmarks and specialized real-world datasets:

• Common Image Classification Datasets:
• MNIST: A dataset of handwritten digits, often used for initial algorithm validation.
• CIFAR-10: A dataset of 32x32 color images across 10 classes.
• CIFAR-100: Similar to CIFAR-10 but with 100 classes, posing a greater challenge.
• Mini-ImageNet: A subset of the larger ImageNet dataset, providing a more complex and larger-scale image classification task.
• Real-World Wearable Application Datasets:
• CHB-MIT: Likely related to medical or physiological signal processing from wearable devices.
• MIT-BH: Another dataset from wearable applications, chosen specifically because these applications face strict constraints on computational complexity and energy consumption.

2. Baselines:

BioFO's performance was benchmarked against several key algorithms:

• Other Forward-Only Algorithms:
• DRTP (Direct Random Target Projection): An existing forward-only method that uses random projections.
• Papita: Another biologically inspired forward-only learning algorithm.
• Forward-Forward (FF): Proposed by Geoffrey Hinton, a prominent forward-only algorithm that uses a local goodness function.
• Backpropagation (BP): The conventional and most widely used training algorithm for neural networks, serving as the gold standard for classification performance.

3. Hardware:

To accurately assess energy efficiency, experiments were conducted on a Jetson Nano. This embedded system is representative of the types of resource-constrained, on-device platforms where BioFO's efficiency advantages would be most impactful.

4. Metrics:

The evaluation focused on the following key metrics:

• Classification Performance: Measured primarily by error rate (lower values indicate better performance).
• Memory Efficiency: Quantified as the reduction in training memory footprint compared to backpropagation.
• Convergence Rate: Assessed by plotting the test error (or similar metric) against training epochs or time, indicating how quickly the algorithm reaches a stable performance level.
• Energy Consumption: Calculated by multiplying the power consumption of the hardware during training by the convergence time (lower values are better).

5. Headline Numbers & Results:

• Classification Performance:
• Across all four common datasets (MNIST, CIFAR-10, CIFAR-100, Mini-ImageNet), BioFO consistently achieved lower error rates than DRTP, Papita, and FF.
• Critically, BioFO's classification performance was observed to be very close to that of backpropagation, indicating that it can achieve high accuracy without BP's drawbacks.
• Memory Efficiency:
• For a typical four-layer network, BioFO demonstrated a saving of at least three times the training memory compared to backpropagation. This significant reduction is a direct benefit of not needing to store intermediate activations for the backward pass.
• Convergence Rate:
• Using CIFAR-10 as an example, BioFO was shown to converge faster than both Papita and FF. While DRTP might show a different convergence curve, it ultimately resulted in a higher error, indicating worse overall classification performance.
• Energy Efficiency:
• On the Jetson Nano, BioFO outperformed all other forward-only algorithms in terms of total energy consumption, reinforcing its suitability for power-constrained environments.
• Architectural Scalability:
• The experiments highlighted the critical role of network architecture.
• Starting with a simple fully connected (FC) network, the CIFAR-10 error served as a baseline.
• Extending BioFO to a locally connected network resulted in a "little bit improved" classification performance.
• However, when extended to a convolutional network, the classification performance improved significantly, with the error "dropping a lot." This underscores the finding that for bio-inspired algorithms, the underlying architecture is extremely important, even if some architectural elements (like weight sharing in CNNs) might be debated for their strict biological plausibility.

These results provide strong evidence for BioFO's efficacy, demonstrating its ability to deliver high performance efficiently and sustainably across diverse applications.

Practical Implications

▶ Watch: Summary of challenges and BioFO's contributions (10:10)

The development and validation of BioFO carry significant practical implications for various stakeholders in the machine learning ecosystem, particularly in the context of on-device AI and sustainable computing.

1. For Practitioners and Infra Teams:

• Enabling On-Device Learning: BioFO's substantial memory and energy efficiency make it an ideal candidate for on-device machine learning (ML) and edge AI. This means models can be trained or continuously updated directly on resource-constrained devices like smartphones, wearables, IoT sensors, and embedded systems, without relying on constant cloud connectivity or powerful data centers. This opens up possibilities for personalized models, privacy-preserving learning, and real-time adaptation.
• Reduced Hardware Requirements: The lower memory footprint could translate into less demanding hardware specifications for training setups. For infrastructure teams, this might mean being able to train models on cheaper, lower-power GPUs or even specialized edge AI accelerators, potentially reducing capital expenditure and operational costs.
• Sustainable AI Development: By offering a more energy-efficient alternative to backpropagation, BioFO contributes to reducing the carbon footprint of AI. This aligns with growing industry and societal demands for sustainable AI, allowing companies to develop powerful models with a smaller environmental impact.
• Asynchronous and Decentralized Training: The local learning nature of BioFO could facilitate more asynchronous and decentralized training paradigms, potentially simplifying the orchestration of distributed training jobs and improving fault tolerance.

2. For Model Builders and Deployers:

• New Capabilities for Edge Devices: Model builders can now envision and develop more complex models that can perform learning tasks directly on edge devices, where previously only inference was feasible. This could lead to innovative applications in areas like predictive maintenance, smart healthcare, and autonomous systems.
• Improved Model Adaptability: On-device training capabilities allow models to adapt to local data distributions or user-specific preferences over time, leading to more robust and personalized user experiences without retraining the entire model from scratch in the cloud.
• Simplified Deployment and Updates: For deployers, pushing updates to models on edge devices might become more straightforward if the devices themselves can perform incremental learning efficiently, reducing the need for full model re-deployments.

3. Tradeoffs and Limitations:

• Biological Plausibility vs. Performance: While BioFO is "biologically plausible" in addressing BP's core issues, there's a nuanced discussion. The speaker noted that while convolutional networks significantly improve performance, their weight-sharing mechanism is debated for its biological plausibility. This highlights a common tension in bio-inspired AI: how closely should we adhere to biological mechanisms versus pragmatic engineering choices for performance? For practical applications, performance often takes precedence, but the biological inspiration remains valuable for guiding architectural and algorithmic innovations.
• Scalability to Large Language Models (LLMs) and Transformers: A key limitation and an active area of research is the scalability of forward-only algorithms, including BioFO, to more complicated multi-architectures such as decoder-based transformers used in LLMs. The speaker acknowledged this as a critical challenge but expressed optimism, citing recent advancements in zero-order gradient generation algorithms that show potential for fine-tuning large language models without full backpropagation. This suggests that while BioFO itself might not directly scale to GPT-scale models today, its underlying principles are contributing to a broader research trend that aims to achieve this.
• Ground Truth Labels: An audience member raised a valid point about the biological plausibility of using "ground truth" labels in a local learning context, as the brain might not always have access to such explicit labels. The speaker acknowledged this as a valid question not specifically addressed in this work, which focused on the four identified issues with backpropagation. This implies that while BioFO makes strides in certain aspects of biological plausibility, the full spectrum of how biological systems learn remains an open research question.

In summary, BioFO offers a compelling pathway to more efficient, sustainable, and on-device machine learning. While challenges remain, particularly concerning scalability to the largest and most complex contemporary AI architectures, the foundational advancements presented pave the way for a new generation of AI systems that are both powerful and responsible.

Key Takeaways

• BioFO (Biologically Plausible Forward-Only Algorithm) effectively addresses key biological implausibility issues of backpropagation, including the locking, non-locality, weight transport, and frozen activities problems.
• The algorithm achieves significant efficiency gains, saving at least 3x training memory and outperforming other forward-only methods in energy consumption, making it highly suitable for resource-constrained on-device ML.
• BioFO maintains classification performance competitive with traditional backpropagation across diverse datasets, including MNIST, CIFAR-10, CIFAR-100, Mini-ImageNet, and real-world wearable data.
• Its core mechanism involves local learning through auxiliary classifiers, which project intermediate layer activations via a fixed random matrix, allowing for local cross-entropy loss calculation and weight updates without global error backpropagation.
• The choice of network architecture is crucial for BioFO's performance, with convolutional networks demonstrating significant improvements, although the strict biological plausibility of weight sharing remains a topic of discussion.
• While challenges exist in scaling forward-only algorithms to very large, complex architectures like transformers for LLMs, ongoing research into zero-order gradient generation algorithms indicates promising potential for future advancements in this area.

About the Speaker(s)

Baichuan Huang is a PhD Candidate at Lund University in Sweden. His research focuses on developing efficient and biologically plausible machine learning algorithms, particularly for on-device applications, as demonstrated by his work on the BioFO algorithm.

Amir Aminifar is a Professor at Lund University in Sweden and is a co-author of the presented paper. His expertise lies in related areas of efficient computing and machine learning, contributing to the foundational research behind BioFO.