Title :
A theoretical comparison of batch-mode, on-line, cyclic, and almost-cyclic learning
Author :
Heskes, Tom ; Wiegerinck, Wim
Author_Institution :
Dept. of Med. Phys. & Biophys., Nijmegen Univ., Netherlands
fDate :
7/1/1996 12:00:00 AM
Abstract :
We study and compare different neural network learning strategies: batch-mode learning, online learning, cyclic learning, and almost-cyclic learning. Incremental learning strategies require less storage capacity than batch-mode learning. However, due to the arbitrariness in the presentation order of the training patterns, incremental learning is a stochastic process; whereas batch-mode learning is deterministic. In zeroth order, i.e., as the learning parameter /spl eta/ tends to zero, all learning strategies approximate the same ordinary differential equation for convenience referred to as the "ideal behavior". Using stochastic methods valid for small learning parameters /spl eta/, we derive differential equations describing the evolution of the lowest-order deviations from this ideal behavior. We compute how the asymptotic misadjustment, measuring the average asymptotic distance from a stable fixed point of the ideal behavior, scales as a function of the learning parameter and the number of training patterns. Knowing the asymptotic misadjustment, we calculate the typical number of learning steps necessary to generate a weight within order /spl epsiv/ of this fixed point, both with fixed and time-dependent learning parameters. We conclude that almost-cyclic learning (learning with random cycles) is a better alternative for batch-mode learning than cyclic learning (learning with a fixed cycle).
Keywords :
differential equations; learning (artificial intelligence); neural nets; almost-cyclic learning; asymptotic misadjustment; batch-mode learning; deterministic process; fixed learning parameters; learning parameter; neural network learning strategies; online learning; ordinary differential equation; stable fixed point; stochastic process; storage capacity; time-dependent learning parameters; Biophysics; Computer networks; Differential equations; Hardware; Neural networks; Physics computing; Stochastic processes;
Journal_Title :
Neural Networks, IEEE Transactions on