DocumentCode :
2202441
Title :
A tutorial on stochastic approximation algorithms for training Restricted Boltzmann Machines and Deep Belief Nets
Author :
Swersky, Kevin ; Chen, Bo ; Marlin, Benjamin ; De Freitas, Nando
Author_Institution :
Dept. of Comput. Sci., Univ. of British Columbia, Vancouver, BC, Canada
fYear :
2010
fDate :
Jan. 31 2010-Feb. 5 2010
Firstpage :
1
Lastpage :
10
Abstract :
In this study, we provide a direct comparison of the Stochastic Maximum Likelihood algorithm and Contrastive Divergence for training Restricted Boltzmann Machines using the MNIST data set. We demonstrate that Stochastic Maximum Likelihood is superior when using the Restricted Boltzmann Machine as a classifier, and that the algorithm can be greatly improved using the technique of iterate averaging from the field of stochastic approximation. We further show that training with optimal parameters for classification does not necessarily lead to optimal results when Restricted Boltzmann Machines are stacked to form a Deep Belief Network. In our experiments we observe that fine tuning a Deep Belief Network significantly changes the distribution of the latent data, even though the parameter changes are negligible.
Keywords :
approximation theory; belief networks; iterative methods; maximum likelihood estimation; stochastic processes; contrastive divergence; deep belief network; iterate averaging; maximum likelihood algorithm; restricted Boltzmann machines; stochastic approximation; Approximation algorithms; Backpropagation; Collaborative work; Computer science; Filtering; Guidelines; Logic circuits; Sampling methods; Stochastic processes; Tutorial;
fLanguage :
English
Publisher :
ieee
Conference_Titel :
Information Theory and Applications Workshop (ITA), 2010
Conference_Location :
San Diego, CA
Print_ISBN :
978-1-4244-7012-9
Electronic_ISBN :
978-1-4244-7014-3
Type :
conf
DOI :
10.1109/ITA.2010.5454138
Filename :
5454138
Link To Document :
بازگشت