Whitened gradient descent, a new updating method for optimizers in deep neural networks

Optimizers are the vital components of deep neural networks that perform weight updates. This paper introduces a new updating method for optimizers based on a gradient descent called the whitened gradient descent (WGD). This method is easy to implement, and can be used in every optimizer based on the gradient descent algorithm. It does not increase the training time of the network significantly. This method smooths the training curve, and improves the classification metrics. In order to evaluate the proposed algorithm, we perform 48 different tests on two datasets, Cifar100 and Animals-10, using three network structures including densenet121, resnet18, and resnet50. The experiments show that using the WGD method in the gradient descent-based optimizers improves the classification results significantly. For example, integrating WGD in the RAdam optimizer increases the accuracy of DenseNet from 87.69% to 90.02% on the Animals-10 dataset.