A regularization-reinforced DBN for digital recognition

The problem of over fitting in DBN is extensively focused on since different networks may respond differently to an unknown input. In this study, a regularization-reinforced deep belief network (RrDBN) is proposed to improve generalization ability. In RrDBN, a special regularization-reinforced term is developed to make the weights in the unsupervised training process to attain a minimum magnitude. Then, the non-contributing weights are reduced and the resultant network can represent the inter-relations of the input–output characteristics. Therefore, the optimization process is able to obtain the minimum-magnitude weights of RrDBN. Moreover, contrastive divergence is introduced to increase RrDBN’s convergence speed. Finally, RrDBN is applied to hand-written numbers classification and water quality prediction. The results of the experiments show that RrDBN can improve the recognition performance with less recognition errors than other existing methods.

[1]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[2]  Derek C. Rose,et al.  Deep Machine Learning - A New Frontier in Artificial Intelligence Research [Research Frontier] , 2010, IEEE Computational Intelligence Magazine.

[3]  José M. Alonso,et al.  Special Issue on Computational Intelligence Software Guest Editorial , 2016, IEEE Comput. Intell. Mag..

[4]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[5]  Hermann Ney,et al.  A Deep Learning Approach to Machine Transliteration , 2009, WMT@EACL.

[6]  Mohamed Bouamar,et al.  Evaluation of the performances of ANN and SVM techniques used in water quality classification , 2007, 2007 14th IEEE International Conference on Electronics, Circuits and Systems.

[7]  Dong Yu,et al.  Large vocabulary continuous speech recognition with context-dependent DBN-HMMS , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[8]  Alan F. Murray,et al.  Continuous restricted Boltzmann machine with an implementable training algorithm , 2003 .

[9]  Tijmen Tieleman,et al.  Training restricted Boltzmann machines using approximations to the likelihood gradient , 2008, ICML '08.

[10]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[11]  Yann LeCun,et al.  The mnist database of handwritten digits , 2005 .

[12]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[13]  Jun-Fei Qiao,et al.  A structure optimisation algorithm for feedforward neural network construction , 2013, Neurocomputing.

[14]  Nitish Srivastava,et al.  Improving neural networks by preventing co-adaptation of feature detectors , 2012, ArXiv.

[15]  Yee Whye Teh,et al.  Unsupervised Discovery of Nonlinear Structure Using Contrastive Backpropagation , 2006, Cogn. Sci..

[16]  Yoshua Bengio,et al.  Greedy Layer-Wise Training of Deep Networks , 2006, NIPS.

[17]  Honglak Lee,et al.  Sparse deep belief net model for visual area V2 , 2007, NIPS.

[18]  Ian R. Fasel,et al.  Deep Belief Networks for Real-Time Extraction of Tongue Contours from Ultrasound During Speech , 2010, 2010 20th International Conference on Pattern Recognition.

[19]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[20]  Siripun Sanguansintukul,et al.  Water quality classification using neural networks: Case study of canals in Bangkok, Thailand , 2009, 2009 International Conference for Internet Technology and Secured Transactions, (ICITST).

[21]  K. Patan.  Stability Analysis and the Stabilization of a Class of Discrete-Time Dynamic Neural Networks , 2007, IEEE Transactions on Neural Networks.

[22]  Geoffrey E. Hinton,et al.  Learning representations of back-propagation errors , 1986 .

[23]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[24]  Geoffrey E. Hinton,et al.  Binary coding of speech spectrograms using a deep auto-encoder , 2010, INTERSPEECH.

[25]  Geoffrey E. Hinton,et al.  Acoustic Modeling Using Deep Belief Networks , 2012, IEEE Transactions on Audio, Speech, and Language Processing.

[26]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[27]  Ahmed El-Shafie,et al.  Prediction of johor river water quality parameters using artificial neural networks , 2009 .

[28]  Linwang Yuan,et al.  Boundary Restricted Non-Overlapping Sphere Tree for Unified Multidimensional Solid Object Index: Boundary Restricted Non-Overlapping Sphere Tree for Unified Multidimensional Solid Object Index , 2012 .

[29]  Geoffrey E. Hinton,et al.  Using fast weights to improve persistent contrastive divergence , 2009, ICML '09.

[30]  Chen Yu,et al.  Chinese Relation Extraction Based on Deep Belief Nets , 2012 .

[31]  Jie Li,et al.  Understanding the dropout strategy and analyzing its effectiveness on LVCSR , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[32]  Mübeccel Demirekler,et al.  Undesirable effects of output normalization in multiple classifier systems , 2003, Pattern Recognit. Lett..

[33]  Kunikazu Kobayashi,et al.  Time series forecasting using a deep belief network with restricted Boltzmann machines , 2014, Neurocomputing.

[34]  K. Kunisch,et al.  Iterative choices of regularization parameters in linear inverse problems , 1998 .