Constructive neural networks: some practical considerations

Based on a Hilbert space point of view, we proposed in our previous work a novel objective function for training new hidden units in a constructive feedforward neural network. Moreover, we proved that if the hidden unit functions satisfy the universal approximation property, the network so constructed incrementally, using the proposed objective function and with input weight freezing, still preserves the universal approximation property with respect to L² performance criteria. In this paper, we provide experimental support for the feasibility of using this objective function. Experiments are performed on two chaotic time series with encouraging results. In passing, we also demonstrate that engineering problems are not to be neglected in practical implementations. We identify the problem of plateau, and then show that by suitably transforming the objective function and modifying the quickprop algorithm, significant improvement can be obtained