Title :
LMS learning algorithms: misconceptions and new results on converence
Author :
Wang, Zi-Qin ; Manry, Michael T. ; Schiano, Jeffrey L.
Author_Institution :
FAS Technol., Dallas, TX, USA
fDate :
1/1/2000 12:00:00 AM
Abstract :
The Widrow-Hoff delta rule is one of the most popular rules used in training neural networks. It was originally proposed for the ADALINE, but has been successfully applied to a few nonlinear neural networks as well. Despite its popularity, there exist a few misconceptions on its convergence properties. We consider repetitive learning (i.e., a fixed set of samples are used for training) and provide an in-depth analysis in the least mean square (LMS) framework. Our main result is that contrary to common belief, the nonbatch Widrow-Hoff rule does not converge in general. It converges only to a limit cycle
Keywords :
backpropagation; convergence; least mean squares methods; neural nets; Widrow-Hoff delta rule; convergence properties; least mean square framework; least mean square learning algorithms; nonlinear neural networks; repetitive learning; Backpropagation algorithms; Convergence; Helium; Least squares approximation; Limit-cycles; Multi-layer neural network; Neural networks; Neurons; Nonhomogeneous media; Signal processing algorithms;
Journal_Title :
Neural Networks, IEEE Transactions on
