A theoretically sound learning algorithm for constructive neural networks

In this paper, we analyse the problem of learning in constructive neural networks from a Hilbert space point of view. A novel objective function for training new hidden units using a greedy approach is derived. More importantly, we prove that a network so constructed incrementally still preserves the universal approximation property with respect to L² performance criteria. While theoretical results obtained so far on the universal approximation capabilities of multilayer feedforward networks only provide existence proofs, our results move one step further by providing a theoretically sound procedure for constructive approximation while still preserving the universal approximation property