Training Restricted Boltzmann Machines with auxiliary function approach

Restricted Boltzmann Machines (RBMs) are neural network models for unsupervised learning, but have recently found a wide range of applications as feature extractors for supervised learning algorithms. They have also received a lot of attention recently after being proposed as building blocks for deep belief networks. The success of these models raises the issue of how best to train them. At present, the most popular training algorithm for RBMs is the Contrastive Divergence (CD) learning algorithm. The aim of this paper is to seek for a new optimization algorithm tailored for training RBMs in the hope of obtaining a faster algorithm than the CD algorithm. We propose deriving a new training algorithm for RBMs based on an auxiliary function approach. Through an experiment on parameter training of an RBM, we confirmed that the present algorithm converged faster and to a better solution than the CD algorithm.

[1]  亀岡 弘和,et al.  Statistical approach to multipitch analysis , 2007 .

[2]  Hirokazu Kameoka,et al.  I-Divergence-based dereverberation method with auxiliary function approach , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[3]  Hirokazu Kameoka,et al.  Efficient algorithms for multichannel extensions of Itakura-Saito nonnegative matrix factorization , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[4]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[5]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[6]  Paul Smolensky,et al.  Information processing in dynamical systems: foundations of harmony theory , 1986 .

[7]  Hirokazu Kameoka,et al.  Complex NMF: A new sparse representation for acoustic signals , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[8]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[9]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[10]  Hirokazu Kameoka,et al.  Maximum reconstruction probability training of Restricted Boltzmann machines with auxiliary function approach , 2014, 2014 IEEE International Workshop on Machine Learning for Signal Processing (MLSP).

[11]  Hirokazu Kameoka,et al.  Auxiliary function approach to parameter estimation of constrained sinusoidal model for monaural speech separation , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[12]  Hirokazu Kameoka,et al.  Robust speech dereverberation based on non-negativity and sparse nature of speech spectrograms , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[13]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[14]  H. Kameoka,et al.  Convergence-guaranteed multiplicative algorithms for nonnegative matrix factorization with β-divergence , 2010, 2010 IEEE International Workshop on Machine Learning for Signal Processing.

[15]  Nobutaka Ono,et al.  Stable and fast update rules for independent vector analysis based on auxiliary function technique , 2011, 2011 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA).