Approximation theory and recurrent networks

It is shown that a given trajectory sequence with the corresponding time steps can be represented by a discrete-time connected recurrent neural net. The result is generalized to an approximation of a differentiable trajectory on a compact time interval. It is shown that fully recurrent neural nets of sigmoid type units can approximate a large class of continuous real functions of time. This implies that fully recurrent neural networks can be universal approximators of trajectories. This fundamental principle of constructing a set of linearly independent vectors can be used to obtain the weights which serve for constructing such networks either directly or by providing a good initial guess for iterative learning algorithms. The estimation of network size is given