Combining normal sparse into discriminative deep belief networks

Training deep models are time consuming and face many local minima. For dealing with this problem, we can use DBN (Deep Belief Network) with Contrastive Divergence (CD). Sparse representations are more efficient. This paper aims to find the best structure of incorporating sparsity into discriminative DBN. We use a DBN architecture with 784 units as input, two layers each with 500 hidden units, one layer with 2000 hidden units, and 10 units as the final output. We argue that combining sparsity and discriminative DBN may increase the accuracy, but no previous studies suggest the best structure or configuration of that combination that can give the best accuracy. We took three stages of experiments to find the best configuration, namely preliminary, intermediate, and final stages. Each analysis of each stage serves as a background for consideration of the next experiment We use normal sparse for generative DBN and discriminative DBN. Experimental studies on MNIST dataset show that the best structure or scenario to combine normal sparse into deep belief networks is as follows: input — generative (CD) — generative (CD) — normal sparse discriminative (CD).

[1]  Yoshua Bengio,et al.  Classification using discriminative restricted Boltzmann machines , 2008, ICML '08.

[2]  Honglak Lee,et al.  Sparse deep belief net model for visual area V2 , 2007, NIPS.

[3]  Mohammad Mehdi Homayounpour,et al.  A brief survey on deep belief networks and introducing a new object oriented MATLAB toolbox (DeeBNet) , 2014, ArXiv.

[4]  Hervé Glotin,et al.  Sparse Penalty in Deep Belief Networks: Using the Mixed Norm Constraint , 2013, ArXiv.

[5]  Mohammad Mehdi Homayounpour,et al.  Normal sparse Deep Belief Network , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[6]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[7]  Tijmen Tieleman,et al.  Training restricted Boltzmann machines using approximations to the likelihood gradient , 2008, ICML '08.

[8]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[9]  Ragheb Walid,et al.  Handwritten digit recognition using sparse deep architectures , 2014, 2014 9th International Conference on Intelligent Systems: Theories and Applications (SITA-14).

[10]  Yan Liu,et al.  Discriminative deep belief networks for visual data classification , 2011, Pattern Recognit..

[11]  Zhi-Hua Zhou,et al.  Special edition on semi-supervised learning for visual content analysis and understanding , 2011, Pattern Recognit..

[12]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[13]  Mohammad Mehdi Homayounpour,et al.  Deep Belief Network Training Improvement Using Elite Samples Minimizing Free Energy , 2015, Int. J. Pattern Recognit. Artif. Intell..