LMS learning algorithms: misconceptions and new results on converence

Author

Wang, Zi-Qin ; Manry, Michael T. ; Schiano, Jeffrey L.

Author_Institution

FAS Technol., Dallas, TX, USA

Volume

11

Issue

1

fYear

2000

fDate

1/1/2000 12:00:00 AM

Firstpage

47

Lastpage

56

Abstract

The Widrow-Hoff delta rule is one of the most popular rules used in training neural networks. It was originally proposed for the ADALINE, but has been successfully applied to a few nonlinear neural networks as well. Despite its popularity, there exist a few misconceptions on its convergence properties. We consider repetitive learning (i.e., a fixed set of samples are used for training) and provide an in-depth analysis in the least mean square (LMS) framework. Our main result is that contrary to common belief, the nonbatch Widrow-Hoff rule does not converge in general. It converges only to a limit cycle

Keywords

backpropagation; convergence; least mean squares methods; neural nets; Widrow-Hoff delta rule; convergence properties; least mean square framework; least mean square learning algorithms; nonlinear neural networks; repetitive learning; Backpropagation algorithms; Convergence; Helium; Least squares approximation; Limit-cycles; Multi-layer neural network; Neural networks; Neurons; Nonhomogeneous media; Signal processing algorithms;

fLanguage

English

Journal_Title

Neural Networks, IEEE Transactions on

Publisher

ieee

ISSN

1045-9227

Type

jour

DOI

10.1109/72.822509

Filename

822509