A new growing pruning deep learning neural network algorithm (GP-DLNN)

During the last decade, a significant research progress has been drawn in both the theoretical aspects and the applications of Deep Learning Neural Networks. Besides their spectacular applications, optimal architectures of these neural networks may speed up the learning process and exhibit better generalization results. So far, many growing and pruning algorithms have been proposed by many researchers to deal with the optimization of standard Feedforward Neural Network architectures. However, applying both the growing and the pruning on the same net may lead a good model for a big data set and hence good selection results. This work is devoted to propose a new Growing and pruning Learning algorithm for Deep Neural Networks. This new algorithm is presented and applied on diverse medical data sets. It is shown that this algorithm outperforms various other artificial intelligent techniques in terms of accuracy and simplicity of the resulting architecture.

[1]  Chandan Chakraborty,et al.  Efficient deep learning model for mitosis detection using breast histopathology images , 2017, Comput. Medical Imaging Graph..

[2]  Shingo Mabu,et al.  Enhancing the generalization ability of neural networks through controlling the hidden layers , 2009, Appl. Soft Comput..

[3]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[4]  Nassir Navab,et al.  AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images , 2016, IEEE Trans. Medical Imaging.

[5]  Jun-Fei Qiao,et al.  A structure optimisation algorithm for feedforward neural network construction , 2013, Neurocomputing.

[6]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[7]  Hossam Faris,et al.  Training feedforward neural networks using multi-verse optimizer for binary classification problems , 2016, Applied Intelligence.

[8]  Ronald M. Summers,et al.  Deep Learning in Medical Imaging: Overview and Future Promise of an Exciting New Technique , 2016 .

[9]  Deok-Hwan Kim,et al.  Solving local minima problem with large number of hidden nodes on two-layered feed-forward artificial neural networks , 2008, Neurocomputing.

[10]  Michael T. Manry,et al.  An integrated growing-pruning method for feedforward network training , 2008, Neurocomputing.

[11]  Bogdan M. Wilamowski,et al.  A Hybrid Constructive Algorithm for Single-Layer Feedforward Networks Learning , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[12]  Christophe Garcia,et al.  An Online Backpropagation Algorithm with Validation Error-Based Adaptive Learning Rate , 2007, ICANN.

[13]  Qing Song,et al.  Adaptive learning rate of SpikeProp based on weight convergence analysis , 2015, Neural Networks.

[14]  Xin Yao,et al.  A constructive algorithm for training cooperative neural network ensembles , 2003, IEEE Trans. Neural Networks.

[15]  Alfred Jean Philippe Lauret,et al.  A node pruning algorithm based on a Fourier amplitude sensitivity test method , 2006, IEEE Transactions on Neural Networks.

[16]  James T. Kwok,et al.  Objective functions for training new hidden units in constructive neural networks , 1997, IEEE Trans. Neural Networks.

[17]  Yih-Fang Huang,et al.  Bounds on the number of hidden neurons in multilayer perceptrons , 1991, IEEE Trans. Neural Networks.

[18]  Laxmidhar Behera,et al.  On Adaptive Learning Rate That Guarantees Convergence in Feedforward Networks , 2006, IEEE Transactions on Neural Networks.

[19]  F. Fnaiech,et al.  Breast cancer diagnosis based on joint variable selection and Constructive Deep Neural Network , 2018, 2018 IEEE 4th Middle East Conference on Biomedical Engineering (MECBME).

[20]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[21]  K. C. Ho,et al.  A formal selection and pruning algorithm for feedforward artificial neural network optimization , 1999, IEEE Trans. Neural Networks.

[22]  R.J.F. Dow,et al.  Neural net pruning-why and how , 1988, IEEE 1988 International Conference on Neural Networks.

[23]  Haibo He,et al.  Learning from Imbalanced Data , 2009, IEEE Transactions on Knowledge and Data Engineering.

[24]  Ge Wang,et al.  A Perspective on Deep Imaging , 2016, IEEE Access.

[25]  Dianhui Wang,et al.  Global Convergence of Online BP Training With Dynamic Learning Rate , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[26]  B. Chandra,et al.  Deep learning with adaptive learning rate using laplacian score , 2016, Expert Syst. Appl..

[27]  Nader Fnaiech,et al.  Feedforward Neural Networks Pruning Algorithms , 2011, Intelligent Systems.

[28]  C. L. Philip Chen,et al.  Broad Learning System: An Effective and Efficient Incremental Learning System Without the Need for Deep Architecture , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[29]  Xin Yao,et al.  A New Constructive Algorithm for Architectural and Functional Adaptation of Artificial Neural Networks , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[30]  R. Setiono,et al.  Effective neural network pruning using cross-validation , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[31]  Ryad A. Zemouri,et al.  An evolutionary building algorithm for Deep Neural Networks , 2017, 2017 12th International Workshop on Self-Organizing Maps and Learning Vector Quantization, Clustering and Data Visualization (WSOM).

[32]  Yuan Lan,et al.  Constructive hidden nodes selection of extreme learning machine for regression , 2010, Neurocomputing.

[33]  Xin Yao,et al.  A New Adaptive Merging and Growing Algorithm for Designing Artificial Neural Networks , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[34]  Lars Kai Hansen,et al.  Structure learning by pruning in independent component analysis , 2008, Neurocomputing.

[35]  Ying Zhang,et al.  Fast unit pruning algorithm for feedforward neural network design , 2008, Appl. Math. Comput..

[36]  S. S. Sridhar,et al.  A Novel Constructive Neural Network Architecture Based on Improved Adaptive Learning Strategy for Pattern Classification , 2011, SocProS.

[37]  SchmidhuberJürgen.  Deep learning in neural networks , 2015 .

[38]  P. Karimaghaee,et al.  A fast convergence algorithm for BPNN based on optimal control theory based learning rate , 2011, The 2nd International Conference on Control, Instrumentation and Automation.

[39]  Asifullah Khan,et al.  Two-phase deep convolutional neural network for reducing class skewness in histopathological images based breast cancer detection , 2017, Comput. Biol. Medicine.

[40]  Amaury Lendasse,et al.  OP-ELM: Optimally Pruned Extreme Learning Machine , 2010, IEEE Transactions on Neural Networks.

[41]  Luiz Eduardo Soares de Oliveira,et al.  Breast cancer histopathological image classification using Convolutional Neural Networks , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[42]  Jihoon Yang,et al.  Constructive Neural-Network Learning Algorithms for Pattern Classification , 2000 .

[43]  James T. Kwok,et al.  Constructive algorithms for structure learning in feedforward neural networks for regression problems , 1997, IEEE Trans. Neural Networks.

[44]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[45]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[46]  Matthew D. Zeiler.  ADADELTA: An Adaptive Learning Rate Method , 2012, ArXiv.

[47]  T. Kathirvalavakumar,et al.  A Novel Pruning Algorithm for Optimizing Feedforward Neural Network of Classification Problems , 2011, Neural Processing Letters.

[48]  Thomas R. Shultz,et al.  A dual-phase technique for pruning constructive networks , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[49]  Nico Karssemeijer,et al.  Large scale deep learning for computer aided detection of mammographic lesions , 2017, Medical Image Anal..

[50]  T. Kathirvalavakumar,et al.  Pruning algorithms of neural networks — a comparative study , 2013, Central European Journal of Computer Science.

[51]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  Zongben Xu,et al.  Universal Approximation of Extreme Learning Machine With Adaptive Growth of Hidden Nodes , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[53]  Rudy Setiono,et al.  A Penalty-Function Approach for Pruning Feedforward Neural Networks , 1997, Neural Computation.

[54]  Andries Petrus Engelbrecht,et al.  A new pruning heuristic based on variance analysis of sensitivity information , 2001, IEEE Trans. Neural Networks.

[55]  Xin Zhou,et al.  Deep neural networks: A promising tool for fault characteristic mining and intelligent diagnosis of rotating machinery with massive data , 2016 .

[56]  Junfei Qiao,et al.  A Self-Organizing Fuzzy Neural Network Based on a Growing-and-Pruning Algorithm , 2010, IEEE Transactions on Fuzzy Systems.

[57]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[58]  Xin-She Yang,et al.  Cuckoo search: recent advances and applications , 2013, Neural Computing and Applications.

[59]  Bao-Gang Hu,et al.  Two-Phase Construction of Multilayer Perceptrons Using Information Theory , 2009, IEEE Transactions on Neural Networks.

[60]  Xiao-Hua Yu,et al.  A new pruning algorithm for neural network dimension analysis , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[61]  Noureddine Zerhouni,et al.  Autonomous and adaptive procedure for cumulative failure prediction , 2011, Neural Computing and Applications.

[62]  J. Czerniak,et al.  Application of rough sets in the presumptive diagnosis of urinary system diseases , 2003 .

[63]  Amir Hussain,et al.  Applications of Deep Learning and Reinforcement Learning to Biological Data , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[64]  Leonardo Franco,et al.  C-Mantec: A novel constructive neural network algorithm incorporating competition between neurons , 2012, Neural Networks.

[65]  Rajesh Parekh,et al.  Constructive Neural Network Learning Algorithms for Multi-Category Real-Valued Pattern Classification , 1997 .

[66]  Leonardo Franco,et al.  Constructive Neural Networks , 2009, Constructive Neural Networks.

[67]  Daniel W. C. Ho,et al.  A new training and pruning algorithm based on node dependence and Jacobian rank deficiency , 2006, Neurocomputing.

[68]  Fernando José Von Zuben,et al.  A constructive algorithm to synthesize arbitrarily connected feedforward neural networks , 2012, Neurocomputing.

[69]  C. Chakraborty,et al.  An Advanced Deep Learning Approach for Ki-67 Stained Hotspot Detection and Proliferation Rate Scoring for Prognostic Evaluation of Breast Cancer , 2017, Scientific Reports.

[70]  Narasimhan Sundararajan,et al.  An efficient sequential learning algorithm for growing and pruning RBF (GAP-RBF) networks , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[71]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[72]  Qing Song,et al.  Robust learning in SpikeProp , 2017, Neural Networks.

[73]  Gustavo Carneiro,et al.  A deep learning approach for the analysis of masses in mammograms with minimal user intervention , 2017, Medical Image Anal..

[74]  Xin Yao,et al.  A review of evolutionary artificial neural networks , 1993, Int. J. Intell. Syst..

[75]  Pascal Vincent,et al.  Contractive Auto-Encoders: Explicit Invariance During Feature Extraction , 2011, ICML.

[76]  Jacek M. Zurada,et al.  Deep Learning of Part-Based Representation of Data Using Sparse Autoencoders With Nonnegativity Constraints , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[77]  Fangju Ai.  A new pruning algorithm for Feedforward Neural Networks , 2011, The Fourth International Workshop on Advanced Computational Intelligence.

[78]  Stefano Tubaro,et al.  Deep Convolutional Neural Networks for pedestrian detection , 2015, Signal Process. Image Commun..

[79]  Wenqing Sun,et al.  Enhancing deep convolutional neural network scheme for breast cancer diagnosis with unlabeled data , 2017, Comput. Medical Imaging Graph..

[80]  Daniel S. Yeung,et al.  Hidden neuron pruning of multilayer perceptrons using a quantified sensitivity measure , 2006, Neurocomputing.

[81]  Oscar Fontenla-Romero,et al.  A review of adaptive online learning for artificial neural networks , 2016, Artificial Intelligence Review.

[82]  Yoshua Bengio,et al.  Maxout Networks , 2013, ICML.

[83]  Anant Madabhushi,et al.  A Deep Convolutional Neural Network for segmenting and classifying epithelial and stromal regions in histopathological images , 2016, Neurocomputing.

[84]  F. Fnaiech,et al.  A modified version of a formal pruning algorithm based on local relative variance analysis , 2004, First International Symposium on Control, Communications and Signal Processing, 2004..

[85]  Himansu Sekhar Behera,et al.  A novel nature inspired firefly algorithm with higher order neural network: Performance analysis , 2016 .

[86]  Junfei Qiao,et al.  Constructive algorithm for fully connected cascade feedforward neural networks , 2016, Neurocomputing.

[87]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[88]  Giovanna Castellano,et al.  An iterative pruning algorithm for feedforward neural networks , 1997, IEEE Trans. Neural Networks.

[89]  Junfei Qiao,et al.  An adaptive growing and pruning algorithm for designing recurrent neural network , 2017, Neurocomputing.

[90]  Yuanjie Zheng,et al.  Breast Cancer Multi-classification from Histopathological Images with Structured Deep Learning Model , 2017, Scientific Reports.

[91]  Changyin Sun,et al.  Active Learning From Imbalanced Data: A Solution of Online Weighted Extreme Learning Machine , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[92]  Masafumi Hagiwara,et al.  A simple and effective method for removal of hidden units and weights , 1994, Neurocomputing.

[93]  Gregory J. Wolff,et al.  Optimal Brain Surgeon and general network pruning , 1993, IEEE International Conference on Neural Networks.

[94]  O. Mangasarian,et al.  Multisurface method of pattern separation for medical diagnosis applied to breast cytology. , 1990, Proceedings of the National Academy of Sciences of the United States of America.

[95]  M. Bialko,et al.  Training of artificial neural networks using differential evolution algorithm , 2008, 2008 Conference on Human System Interactions.

[96]  Junfei Qiao,et al.  A node pruning algorithm for feedforward neural network based on neural complexity , 2010, 2010 International Conference on Intelligent Control and Information Processing.

[97]  Paulo Cortez,et al.  Particle swarms for feedforward neural network training , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[98]  Mohamed Cheriet,et al.  The Combined Statistical Stepwise and Iterative Neural Network Pruning Algorithm , 2009, Intell. Autom. Soft Comput..

[99]  Luca Maria Gambardella,et al.  Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks , 2013, MICCAI.