Restricted Boltzmann Machine with Adaptive Local Hidden Units

Deep belief network DBN shows the ability to learn hierarchical feature representation from image datasets which mimics the hierarchical organization of the mammal visual cortex. DBN is composed of a stack of Restricted Boltzmann Machines RBM which serves as feature extractors. A number of variants of RBM have been proposed to learn feature representations similar to gabor filters. They require extracting small image patches first. As images vary among different datasets, it is preferable to learn the patch size or a proper region of interest. We propose a variant of RBM with adaptive local hidden units ALRBM by adding a distance function to the connection weights between visible and hidden units. Experiments on hand-written digits and human faces show that our algorithm has the ability to learn region-based local feature representations adapting to the content of the images automatically.

[1]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[2]  R. L. Valois,et al.  The orientation and direction selectivity of cells in macaque visual cortex , 1982, Vision Research.

[3]  Marc'Aurelio Ranzato,et al.  Sparse Feature Learning for Deep Belief Networks , 2007, NIPS.

[4]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[5]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[6]  Honglak Lee,et al.  Sparse deep belief net model for visual area V2 , 2007, NIPS.

[7]  Marc'Aurelio Ranzato,et al.  Efficient Learning of Sparse Representations with an Energy-Based Model , 2006, NIPS.

[8]  Allen Y. Yang,et al.  Robust Face Recognition via Sparse Representation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[10]  Andrew Y. Ng,et al.  The Importance of Encoding Versus Training with Sparse Coding and Vector Quantization , 2011, ICML.

[11]  D. Hubel,et al.  Receptive fields and functional architecture of monkey striate cortex , 1968, The Journal of physiology.

[12]  Honglak Lee,et al.  Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations , 2009, ICML '09.

[13]  Cordelia Schmid,et al.  Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).