Information criterion for variational Bayes learning in regular and singular cases

Variational Bayes learning gives the accurate statistical estimation as Bayes learning with smaller computational cost. However, it has been difficult to estimate its generalization loss, because learning machines used in variational Bayes are not regular but singular, resulting that the conventional information criteria such as AIC, BIC, or DIC can not be applied. In this paper, we propose a new information criterion for variational Bayes learning, which is the unbiased estimator of the generalization loss for both cases when the posterior distribution is regular and singular. We show the theoretical support of the proposed information criterion, and its effectiveness is illustrated by numerical experiments.