A heuristic approach to structural and parametric change in artificial neural networks

Selection of the right connectivity is one open issue in neural network design. This paper describes a method that, assuming a variant of the common synaptic model, allows for simultaneous weight and structure updating during the training phase. The method effectively trims those connections that are not essential and, unlike traditional pruning techniques, it does not require any subjectively interpretable saliency measure. Detailed implications are provided for the case of discrete-time recurrent networks and the particular case of feedforward perceptrons trained by gradient-descent methods. Preliminary experiments in three real-world classification tasks show favorable results with a considerable reduction in the number of effective connections.