Training neural networks with time-varying optimization

Training neural networks is a process of optimization and in many practical applications this process is usually time-dependent. Time-varying optimization proposed in this paper is just a process of tracking the time-varying optimum of a time-dependent objective function. Several techniques are proposed for solving time-varying optimization problems. One of them ensure the tracking converge exponentially and the Newton-Raphson algorithm is a special case of it. Theoretical analysis and computer experiments show that the training of neural networks is substantially speeded up using time-varying optimization techniques.