Going forward-forward in distributed deep learning

Official URL: http://dx.doi.org/10.1007/978-3-032-00347-8_11

We introduce a new approach in distributed learning, build- ing on Hinton’s Forward-Forward (FF) algorithm to speed up the train- ing of neural networks in distributed environments without losing ac- curacy. Unlike traditional methods that rely on forward and backward passes, the FF algorithm employs a dual forward pass strategy, eliminat- ing the dependency among layers required during the backpropagation period, which prevents efficient parallelization of the training process. Although the original FF algorithm focused on its ability to match the performance of the backpropagation algorithm, this work aims to re- duce the training time with pipeline parallelism. We propose three novel pipelined FF algorithms that speed up training 3.75 times on the MNIST dataset while maintaining accuracy when training a four-layer network with four compute nodes. These results show that FF is highly paral- lelizable and its potential in large-scale distributed/federated systems to enable faster training for larger and more complex models.