Parallel tempering with equi-energy moves for training of restricted boltzmann machines

Training RBMs is laborious due to the difficulty of sampling from model´s distribution. Although using Parallel Tempering (PT) alleviates the problem to some extent, it will result in low swap acceptance ratio when the states´ energies of neighboring chains are very different. In this paper, we propose a novel PT algorithm based on the principle of swapping between chains with the same level of energy. This new algorithm partitions the state space obtained by a population of Gibbs sampling chains into several energy rings. In each ring, states have similar energies and swapping of each pair of states are conducted with a probability. Experiments on a toy dataset as well as the MNIST dataset shown that the new algorithm keeps high swap acceptance ration and results in better likelihood scores compared to several training methods.