APTx: Better Activation Function than MISH, SWISH, and ReLU’s Variants used in Deep Learning

Abstract

Activation Functions introduce non-linearity in the deep neural networks. This
nonlinearity helps the neural networks learn faster and efficiently from the
dataset. In deep learning, many activation functions are developed and used
based on the type of problem statement. ReLU’s variants, SWISH, and MISH are
goto activation functions. MISH function is considered having similar or even
better performance than SWISH, and much better than ReLU. In this paper, we
propose an activation function named APTx which behaves similar to MISH, but
requires lesser mathematical operations to compute. The lesser computational
requirements of APTx does speed up the model training, and thus also reduces
the hardware requirement for the deep learning model.