Push-pull separability objective for supervised layer-wise training of neural networks

Deep architecture neural networks have been shown to generalise well for many classification problems, however, outside the empirical evidence, it is not entirely clear how deep representation benefits these problems. This paper proposes a supervised cost function for an individual layer in a deep architecture classifier that improves data separability. From this measure, a training algorithm for a multi-layer neural network is developed and evaluated against backpropagation and deep belief net learning. The results confirm that the proposed supervised training objective leads to appropriate internal representation with respect to the classification task, especially for datasets where unsupervised pre-conditioning is not effective. Separability of the hidden layers offers new directions and insights in the quest to illuminate the black box model of deep architectures.