Performance of analog neural networks subject to drifting targets and noise

This paper analyzes the tracking behavior of analog neurons when subjected to additive noise and slowly drifting target weights. The performance of single analog neurons and multilayer analog neural networks are studied. We consider a system identification model with a target and tracking neural network. The target weights are described by a stochastic difference equation with weights changing slowly with time. The tracker weights follow the least mean square (LMS) gradient descent algorithm and at each update are given a noise corrupted value of the output of the target network. When inputs are Gaussian and the activation used is the Gaussian error function, g(x)=1/√(2π)∫_-x^xe(-t^2/2 )dt. The analysis is tractable. The dynamics of target and tracking networks are described by a set of stochastic difference equations. We obtain an approximation of the mean squared generalization error by linearizing the nonlinear difference equations and using simple probabilistic arguments. Analytical results are then confirmed with simulation results