A convergence analysis for neural networks with constant learning rates and non-stationary inputs

A novel deterministic approach to the convergence analysis of (stochastic) temporal neural networks is presented. The link between the two is a new concept of time-average invariance (TAI) which is a property of deterministic signals but with applications to stochastic signals. With this new concept, the conventional ODE method can be extended to the case of constant learning rate. With weaker conditions, not requiring mutually independence, it is shown that a temporal neural network is ε-convergent to x⁰, if its associated (autonomous) equations are asymptotically stable at x⁰. This result is then extended to the case of perturbed TAI signals. A temporal neural network for blind signal separation is used as an example