Comparing parameterless learning rate adaptation methods

Since the popularization of the backpropagation learning rule for training multilayer neural networks, many improvements and extensions of it have been proposed. Adaptive learning rate techniques are certainly among the most well-known of such improvements, promising a significant increase in learning speed and, in a case when no new tunable parameters are introduced, eliminating the trial-and-error process of finding a suitable learning rate. Hence, in order to compare the most promising of these, five methods without tunable parameters have been selected. Both the online and batch versions of standard backpropagation are also integrated into the study as points of reference. However, in order to compare the convergence speed of different learning rules, a better complexity measure is needed than the commonly used `number of training iterations´. Hence, a refined complexity measure is introduced here and used in the comparison of the seven chosen methods