Comparing Different Recurrent Neural Architectures on a Specific Task from Vanishing Gradient Effect Perspective

The objective of this paper is to compare the performances of different recurrent neural systems when applied to a specific task, namely the latching problem. It is a benchmark for evaluating the impact of the vanishing gradient effect, arising when the neural network under study is trained through gradient based learning algorithms. Three distinct architectures have been addressed, in different configurations: the fully recurrent neural network (fRNN), the recurrent multiscale network (RMN) and the echo state network (ESN), all already known in literature, but never considered together from this perspective. As expected, ESNs seem to be immune to the vanishing gradient problem, whose effect is conversely strong in the case of fRNN and partially (but significantly) mitigated when RMNs are used