TRTRL: A Localized Resource-Efficient Learning Algorithm for Recurrent Neural Netowrks

This paper introduces an efficient, low-complexity online learning algorithm for recurrent neural networks. The approach is based on the real-time recurrent learning (RTRL) algorithm, whereby the sensitivity set of each neuron is reduced to weights associated either with its input or ouput links. As a consequence, storage requirements are reduced from O(N³) to O(N²) and the computational complexity is reduced to O(N²). Despite the radical reduction in resource requirements, it is shown through simulation results that the overall performance degradation is rather minor. Moreover, the scheme lends itself to parallel hardware realization by virtue of the localized property that is inherent to the approach.