An improved RBM based on Bayesian Regularization

Restricted Boltzmann Machine is a fundamental method in deep learning networks. Training and generalization is an ill-defined problem in that many different networks may achieve the training goal; however each will respond differently to an unknown input. Traditional approaches include stopping the training early and/or restricting the size of the network These approaches ameliorate the problem of over-fitting where the network learns the patterns presented but is unable to generalize. Bayesian regularization addresses these issues by requiring the weights of the network to attain a minimum magnitude. This ensures that non-contributing weights are reduced significantly and the resulting network represents the essence of the inter-relations of the training. Bayesian Regularization simply introduces an additional term to the objective function. This term comprises the sum of the squares of the weights. The optimization process therefore not only achieves the objective of the original cost (i.e. the minimization of an error metric) but it also ensures that this objective is achieved with minimum-magnitude weights. We have introduced Bayesian Regularization in the training of Restricted Boltzmann Machines and have applied this method in experiments of hand-written numbers classification. Our experiments showed that by adding Bayesian regularization in the training of RBMs, we were able to improve the generalization capabilities of the trained network by reducing its recognition errors by more than 1.6%.