Encoding sequential structure: experience with the real-time recurrent learning algorithm

It is shown that recurrent nets trained with the RTRL (real-time recurrent learning) algorithm are able to learn tasks that Elman nets appear unable to learn. Moreover, they learn a more stringent form of the task that does not require knowledge of the string boundaries to be used. This is of potential importance in cases in which this information is not known beforehand but must be learned by the network, such as speech recognition. Although the recurrent nets learn the prediction tasks, they do so with great difficulty, requiring many string presentations to reach criteria. This is a significant problem because of the large amount of computation required by the recurrent algorithm. Some of the tasks have taken 30 hours of CPU time on a Sun-4/280 to be learned. This time could be greatly reduced if advantage could be taken of the very high degree of parallelism in the RTRL algorithm.<>