Convergence Analysis of Multiplicative Weight Noise Injection During Training

Injecting weight noise during training has been proposed for almost two decades as a simple technique to improve fault tolerance and generalization of a multilayer perceptron (MLP). However, little has been done regarding their convergence behaviors. Therefore, we presents in this paper the convergence proofs of two of these algorithms for MLPs. One is based on combining injecting multiplicative weight noise and weight decay (MWN-WD) during training. The other is based on combining injecting additive weight noise and weight decay (AWN-WD) during training. Let m be the number of hidden nodes of a MLP, a be the weight decay constant and S_b be the noise variance. It is showed that the convergence of MWN-WD algorithm is with probability one if a >; √(S_b)m. While the convergence of the AWN-WD algorithm is with probability one if a >; 0.