How to initialize the CNN for small datasets: Extracting discriminative filters from pre-trained model

In this paper, we study how to initialize the convolutional neural network (CNN) model for training on a small dataset. Specially, we try to extract discriminative filters from the pre-trained model for a target task. On the basis of relative entropy and linear reconstruction, two methods, Minimum Entropy Loss (MEL) and Minimum Reconstruction Error (MRE), are proposed. The CNN models initialized by the proposed MEL and MRE methods are able to converge fast and achieve better accuracy. We evaluate MEL and MRE on the CIFAR10, CIFAR100, SVHN, and STL-10 public datasets. The consistent performances demonstrate the advantages of the proposed methods.