Neural network training schemes for non-linear adaptive filtering and modelling

There are a wide variety of cost functions, techniques for estimating their gradient, and adaptive algorithms for updating the coefficients of neural networks used as nonlinear adaptive filters. The authors discuss the various algorithms which result from various choices of criteria and of gradient estimation techniques. New algorithms are introduced, and the relations between the present work, the real-time recurrent learning algorithm, and the teacher forcing technique are discussed. The authors show that the training algorithms suggested recently for feedback networks are very closely related to and in some cases identical to the algorithms used classically for adapting recursive filters