A position paper on statistical inference techniques which integrate neural network and Bayesian network models

Some statistical methods which have been shown to have direct neural network analogs are surveyed here; we discuss sampling, optimization and representation methods which make them feasible when applied in conjunction with, or in place of, neural networks. We present the foremost of these, the Gibbs sampler, both in its successful role as a convergence heuristic derived from statistical physics and under its probabilistic learning interpretation. We then review various manifestations of Gibbs sampling in Bayesian learning; its relation to “traditional” simulated annealing; specializations and instances such as EM; and its application as a model construction technique for the Bayesian network formalism. Next, we examine the ramifications of advances in Markov chain Monte Carlo methods for learning by backpropagation. Finally we consider how the Bayesian network formalism informs the causal reasoning interpretation of some neural networks, and how it prescribes optimizations for efficient random sampling in Bayesian learning applications