A dynamic neural network architecture by sequential partitioning of the input space

A sequential approach to neural network training is presented. The network is presented with each item of data only once, and its architecture is dynamically adjusted during training. At the arrival of each example, a decision whether to increase the complexity of the network is made, based on three heuristic criteria. These criteria measure the position of the new item of data with respect to the information currently stored in the network. Each hidden unit in the network is trained in closed form by means of a recursive least squares algorithm. A local covariance matrix of the data is maintained at each node, and the closed form solution is recursively updated. The performance of the algorithm is illustrated on a small-scale problem, using two-dimensional speech data. The sequential nature of the algorithm has an efficient hardware implementation in the form of systolic arrays, and the incremental training idea has better biological plausibility