On-line tracking abilities of neural networks with graded responses

This paper analyzes the tracking performance of neural networks with graded analog responses when the weights of a target network change slowly with time. We first study the performance of a tracker consisting of a single neuron and then discuss two-layer networks. The target network weights are described by a stochastic difference equation with weights changing slowly with time. The tracker weights follow the Least Mean Square (LMS) gradient descent algorithm. We use a Gaussian error function: g(x)=1/√(2π)∫_-x^xe(-t² /2)dt, as this activation function makes the analysis tractable and the function closely approximates the standard sigmoidal nonlinearity (g(x)=tanh(x)). For a weight drift of rate y and appropriately chosen step size μ the mean squared generalization error is proportional to γ²/μ. The paper then formulates an approach to analyzing a two-layer soft committee machine and concludes by discussing extensions to this research