A novel sparse auto-encoder for deep unsupervised learning

This paper proposes a novel sparse variant of auto-encoders as a building block to pre-train deep neural networks. Compared with sparse auto-encoders through KL-divergence, our method requires fewer hyper-parameters and the sparsity level of the hidden units can be learnt automatically. We have compared our method with several other unsupervised leaning algorithms on the benchmark databases. The satisfactory classification accuracy (97.92% on MNIST and 87.29% on NORB) can be achieved by a 2-hidden-layer neural network pre-trained using our algorithm, and the whole training procedure (including pre-training and fine-tuning) takes far less time than the state-of-art results.