Learning the deterministically constructed Echo State Networks

Echo State Networks (ESNs) have shown great promise in the applications of non-linear time series processing because of their powerful computational ability and efficient training strategy. However, the nature of randomization in the structure of the reservoir causes it be poorly understood and leaves room for further improvements for specific problems. A deterministically constructed reservoir model, Cycle Reservoir with Jumps (CRJ), shows superior generalization performance to standard ESN. However, the weights that govern the structure of the reservoir (reservoir weights) in CRJ model are obtained through exhaustive grid search which is very computational intensive. In this paper, we propose to learn the reservoir weights together with the linear readout weights using a hybrid optimization strategy. The reservoir weights are trained through nonlinear optimization techniques while the linear readout weights are obtained through linear algorithms. The experimental results demonstrate that the proposed strategy of training the CRJ network tremendously improves the computational efficiency without jeopardizing the generalization performance, sometimes even with better generalization performance.