Simplifying OCR neural networks with oracle learning

Often the best model to solve a real world problem is relatively complex. The article presents oracle learning, a method using a larger model as an oracle to train a smaller model on unlabeled data in order to obtain: (1) a simpler acceptable model and (2) improved results over standard training methods on a similarly sized smaller model. In particular, this paper looks at oracle learning as applied to multilayer perceptrons trained using standard backpropagation. For optical character recognition, oracle learning results in an 11.40% average decrease in error over direct training while maintaining 98.95% of the initial oracle accuracy.