Overcoming the local-minimum problem in training multilayer perceptrons by gradual deconvexification

A method of training neural networks using the risk-averting error (RAE) criterion J_λ (w), which was presented in IJCNN 2001, has the capability to avoid nonglobal local minima, but suffers from a severe limitation on the magnitude of the risk-sensitivity index λ. To eliminating the limitation, an improved method using the normalized RAE (NRAE) C_λ (w) was proposed in ISNN 2012, but it requires a selection of a proper λ, whose range may be dependent on the application. A new training method called the gradual deconvexification (GDC) is proposed in this paper. It starts with a very large λ and gradually decreases it in the training process until a global minimum of C_λ (w) or a good generalization capability is achieved. GDC training method was tested on a large number of numerical examples and produced a very good result in each test.