Optimization Algorithms in Deep Learning Models for Improving the Forecasting Accuracy in Sequential Datasets with Application in the South African Stock Market Index: A Review

Abstract

In this paper we review different popular optimization algorithms for machine
learning models, we then evaluate the model performance and convergence
rates for each optimizer using a multilayer fully connected neural networks.
Using sequential dataset of index returns (time-series data) spanning over of 20-
years, we demonstrate Adam and RMSprop optimizers can efficiently solve
practical deep learning problems dealing with sequential datasets. We use the
same parameter initialization when comparing different optimization
algorithms. The hyper-parameters, such as learning rate and momentum, are
searched over a dense grid and the results are reported using the best hyperparameter
setting.