Mining invariance in Restricted Boltzmann Machine via Information Geometry

Obtaining invariant result between data and its variants under some kinds of transformation is a useful property in machine learning and computer vision. Previous researchers have empirically shown that Deep Belief Network (DBN) has some degree of invariance, but it still lacks a sound theoretical explanation. In this paper, we study the invariance of Restricted Boltzmann Machine (RBM), which is the building block of DBN, from its stationary distribution via Imformation Geometry (IG) theory. This is different from previous works which focused on the state of latent variables (as features) in the hidden layer of RBM. We show theoretically and empirically that single layer Boltzmann Machine (SBM) has invariance when data and its variants are similar in local information. We also empirically show that RBM has better invariance degree comparing with SBM. We expect our results can inspire future explanation for the invariance of DBN.