Memory Efficient Backpropagation Algorithms for Training Deep Networks on Edge Devices

Abstract

The fast adoption of edge intelligence-based applications has led to the need for memory-efficient training techniques that can be executed within constrained edge devices. This research seeks to devise a memory-efficient backpropagation algorithm for training deep neural networks on edge devices. The backpropagation algorithm will ensure efficient training processes without increasing memory, computations, or energy costs. The backpropagation algorithm will involve selective activation checkpointing, gradient computation, and adaptive precision optimization for minimizing the amount of intermediate activation storage in the training process. Lightweight CNN architectures will be analyzed using benchmark datasets such as CIFAR-10, MNIST, Fashion-MNIST, and Edge Sensor datasets. With regard to the proposed framework, there were reductions in terms of memory usage from 2450 MB down to 1210 MB, representing memory savings of about 50%. As far as training times are concerned, there were reductions in terms of minutes used from 128 down to 88 minutes, with energy consumption also being reduced from 78 W down to 47 W. With all these reductions, however, accuracy rates remained very high at 95.2% on the CIFAR-10 dataset and 99.1% on the MNIST datasets. Training speed reductions between 26.5% and 33.2% validated computational efficiency of the model.