Sparse maximum entropy deep belief nets

In this paper, we present a sparse maximum entropy (SME) learning algorithm for deep belief net (DBN). The SME algorithm aims to maximize the entropy and encourage sparsity of the model. Compared with the conventional maximum likelihood (ML) learning, the proposed SME algorithm enables DBN to be more unbiased to data distributions and robust to overfitting issues, and accordingly provide a better generalization capability. MNIST and NORB data sets were used to evaluated the proposed SME algorithm. Experimental results show that SME-trained DBN outperforms ML-trained DBN on both data sets.