Mean squared error analysis of analog neural networks subject to drifting targets and noise

In previous work we studied the tracking behavior of neural networks with binary outputs subject to drifting targets and noise. This paper extends this work by considering the tracking behavior of analog output neurons when subjected to additive noise and slowly drifting target weights. The target weights are described by a stochastic difference equation with weights changing slowly with time. The tracker weights follow the least mean square (LMS) gradient descent algorithm and at each update are given a noise corrupted value of the output of the target network. When inputs are Gaussian and the activation used is the Gaussian error function (closely approximates the standard sigmoidal activation function) the analysis is tractable. The dynamics of target and tracking networks are described by a set of stochastic difference equations. We obtain an approximation of the mean squared generalization error by linearizing the nonlinear difference equations and using simple probabilistic arguments. We consider the single neuron case and some specific multi-layer neural networks.