A novel adaptive learning deep belief network based on automatic growing and pruning algorithms

Abstract In this study, a novel adaptive learning deep belief network (ALDBN) with a series of growing and pruning algorithms is proposed to dynamically adjust its structure when ALDBN is utilized for extracting features. Specifically, a neuron growing algorithm is designed considering the individual and macroscopical impacts on each neuron to detect unstable hidden neurons, and a new hidden neuron will be added around each unstable neuron to compensate for the inadequacy of the local structure for feature extraction. Moreover, the relations of network depth and information entropy with respect to the normal distribution of each weight between hidden layers are revealed. On basis of the relations revealed, a layer growing algorithm is designed considering the obedience rate of the normal distribution to control the number of hidden layers. In addition, a neuron pruning algorithm using the standard deviation of neuron activation probability is integrated in ALDBN to prune the redundant neurons with low discriminative ability. We first give the theoretical proof for the convergence of ALDBN, which is crucial to its stability. To exhibit its performance, parameter sensitivity analysis is then provided to investigate the effects of two key parameters in ALDBN. Finally, we compare ALDBN with five state-of-art methods on three benchmark datasets, and the comparative experimental results demonstrate that ALDBN outperforms the other five competitors in terms of the accuracies of common test, cross-validated test and holdout test.

[1]  Junfei Qiao,et al.  A self-organizing deep belief network for nonlinear system modeling , 2018, Appl. Soft Comput..

[2]  Masatoshi Okutomi,et al.  Coupled convolution layer for convolutional neural network , 2018, Neural Networks.

[3]  Nian Liu,et al.  Improved deep belief networks and multi-feature fusion for leaf identification , 2016, Neurocomputing.

[4]  Yu Xue,et al.  Text classification based on deep belief network and softmax regression , 2016, Neural Computing and Applications.

[5]  Takumi Ichimura,et al.  An adaptive learning method of Deep Belief Network by layer generation algorithm , 2016, 2016 IEEE Region 10 Conference (TENCON).

[6]  Olgierd Hryniewicz,et al.  On generalized versions of central limit theorems for IF-events , 2016, Inf. Sci..

[7]  Shen Furao,et al.  Forecasting exchange rate using deep belief networks and conjugate gradient method , 2015, Neurocomputing.

[8]  Zhang Yi,et al.  A Novel Deep Learning-Based Collaborative Filtering Model for Recommendation System , 2019, IEEE Transactions on Cybernetics.

[9]  Qi Zhang,et al.  Detection of Magnetic Anomaly Signal Based on Information Entropy of Differential Signal , 2018, IEEE Geoscience and Remote Sensing Letters.

[10]  Lijun Liu,et al.  Relaxed conditions for convergence of batch BPAP for feedforward neural networks , 2015, Neurocomputing.

[11]  H. L Rong,et al.  A Vehicle Type Recognition Method based on Sparse Auto Encoder , 2015 .

[12]  Rajdeep Chatterjee,et al.  A novel machine learning based feature selection for motor imagery EEG signal classification in Internet of medical things environment , 2019, Future Gener. Comput. Syst..

[13]  Takumi Ichimura,et al.  A Structural Learning Method of Restricted Boltzmann Machine by Neuron Generation and Annihilation Algorithm , 2016, ICONIP.

[14]  Wei Li,et al.  A new deep auto-encoder using multiscale reconstruction errors and weight update correlation , 2021, Inf. Sci..

[15]  Yangyang Li,et al.  Parallel design of sparse deep belief network with multi-objective optimization , 2020, Inf. Sci..

[16]  Hidetoshi Murakami,et al.  Model based on skew normal distribution for square contingency tables with ordinal categories , 2014, Comput. Stat. Data Anal..

[17]  Hsin Chen,et al.  An Adaptable Continuous Restricted Boltzmann Machine in VLSI for Fusing the Sensory Data of an Electronic Nose , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[18]  Fei Wang,et al.  Efficient tree classifiers for large scale datasets , 2018, Neurocomputing.

[19]  Junfei Qiao,et al.  Identification and modeling of nonlinear dynamical systems using a novel self-organizing RBF-based approach , 2012, Autom..

[20]  Hao Chen,et al.  Cross-Validated Locally Polynomial Modeling for 2-D/3-D Gaze Tracking With Head-Worn Devices , 2020, IEEE Transactions on Industrial Informatics.

[21]  Wen Yu,et al.  Randomized algorithms for nonlinear system identification with deep learning modification , 2016, Inf. Sci..

[22]  Yan Liu,et al.  Relaxed conditions for convergence analysis of online back-propagation algorithm with L2 regularizer for Sigma-Pi-Sigma neural network , 2018, Neurocomputing.

[23]  Zhang Yi,et al.  Graph Regularized Restricted Boltzmann Machine , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[24]  Junfei Qiao,et al.  A Self-Organizing Fuzzy Neural Network Based on a Growing-and-Pruning Algorithm , 2010, IEEE Transactions on Fuzzy Systems.

[25]  Savitha Ramasamy,et al.  Online RBM: Growing Restricted Boltzmann Machine on the fly for unsupervised representation , 2020, Appl. Soft Comput..

[26]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[27]  Andrew Beng Jin Teoh,et al.  Stacking-based deep neural network: Deep analytic network on convolutional spectral histogram features , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[28]  Wei Wu,et al.  Convergence analysis of online gradient method for BP neural networks , 2011, Neural Networks.

[29]  Fernando Pereira dos Santos,et al.  Supervised and unsupervised relevance sampling in handcrafted and deep learning features obtained from image collections , 2019, Appl. Soft Comput..

[30]  Xi Chen,et al.  Estimation of the Volume of the Left Ventricle From MRI Images Using Deep Neural Networks , 2017, IEEE Transactions on Cybernetics.

[31]  Amit Verma,et al.  Deep learning based enhanced tumor segmentation approach for MR brain images , 2019, Appl. Soft Comput..

[32]  Mark D. McDonnell,et al.  Deep extreme learning machines: supervised autoencoding architecture for classification , 2016, Neurocomputing.

[33]  Martin A. Riedmiller,et al.  Deep auto-encoder neural networks in reinforcement learning , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[34]  Ahmet Murat Ozbayoglu,et al.  Deep Learning for Financial Applications : A Survey , 2020, Appl. Soft Comput..

[35]  Giuseppe De Pietro,et al.  Deep neural network for hierarchical extreme multi-label text classification , 2019, Appl. Soft Comput..

[36]  Kazunori Iwata,et al.  Extending the Peak Bandwidth of Parameters for Softmax Selection in Reinforcement Learning , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[37]  Omar Naifar,et al.  Comments on "Lyapunov stability theorem about fractional system without and with delay" , 2016, Commun. Nonlinear Sci. Numer. Simul..

[38]  Masato Okada,et al.  Dynamical analysis of contrastive divergence learning: Restricted Boltzmann machines with Gaussian visible units , 2016, Neural Networks.

[39]  Zhiqiang Geng,et al.  A new deep belief network based on RBM with glial chains , 2018, Inf. Sci..

[40]  Yu-Lin He,et al.  Determining the optimal temperature parameter for Softmax function in reinforcement learning , 2018, Appl. Soft Comput..

[41]  Wei-Chang Yeh,et al.  A Squeezed Artificial Neural Network for the Symbolic Network Reliability Functions of Binary-State Networks , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[42]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.