Scaling properties of on-line learning with momentum

We study online learning with momentum term for nonlinear learning rules. Through introduction of auxiliary variables, we show that the learning process can still be described by a first-order Markov process. For small learning parameters η and momentum parameters α close to 1 (we consider the case α=1-√(η/λ) for small η), Van Kampen´s expansion can be applied in a straightforward manner. We obtain evolution equations for the average network state and the fluctuations around this average. These evolution equations depend (after rescaling of time and fluctuations) only on λ=η/(1-α)²: all combinations (η,α) with the same value of λ give rise to similar graphs. For small λ, i.e., η≪(1-α)², learning with momentum term is equivalent to learning without momentum term with rescaled learning parameter η˜=η/(1-α). Simulations with the nonlinear Oja learning rule confirm our theoretical results