Modified Backpropagation with Added White Gaussian Noise in Weighted Sum for Convergence Improvement

Abstract The impact of the noise injection in backpropagation is studied since last two decades. In this manuscript, a brief review is done on noise injection techniques used in the backpropagation. Here, the modified backpropagation algorithm is proposed, in which the white Gaussian noise is added in the weighted sum entity of the backpropagation. The experimentation is carried out on different standard benchmark problems such as 2 bit parity and iris dataset. It is observed that the proposed modified backpropagation requires less number of epochs as compared to the standard backpropagation for the convergence when applied on 2 bit parity and Iris dataset.

[1]  Arjen van Ooyen,et al.  Improving the convergence of the back-propagation algorithm , 1992, Neural Networks.

[2]  Stephen Grossberg,et al.  Recurrent neural networks , 2013, Scholarpedia.

[3]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[4]  Eiji Hayashi,et al.  Development of Cloud Action for Seamless Robot Using Backpropagation Neural Network , 2017, J. Robotics Netw. Artif. Life.

[5]  Weiwei Sun,et al.  A Novel Approach to Subpixel Land-Cover Change Detection Based on a Supervised Back-Propagation Neural Network for Remotely Sensed Images With Different Resolutions , 2017, IEEE Geoscience and Remote Sensing Letters.

[6]  Wei Wu,et al.  Boundedness and convergence of batch back-propagation algorithm with penalty for feedforward neural networks , 2012, Neurocomputing.

[7]  Andrew Chi-Sing Leung,et al.  Convergence and Objective Functions of Some Fault/Noise-Injection-Based Online Learning Algorithms for RBF Networks , 2010, IEEE Transactions on Neural Networks.

[8]  Guoqiang Peter Zhang,et al.  A neural network ensemble method with jittered training data for time series forecasting , 2007, Inf. Sci..

[9]  Petri Koistinen,et al.  Kernel regression and backpropagation training with noise , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[10]  Amit Gupta,et al.  Weight decay backpropagation for noisy data , 1998, Neural Networks.

[11]  Yves Grandvalet,et al.  Noise Injection: Theoretical Prospects , 1997, Neural Computation.

[12]  Kiyotoshi Matsuoka,et al.  Noise injection into inputs in back-propagation learning , 1992, IEEE Trans. Syst. Man Cybern..

[13]  Alan F. Murray,et al.  Synaptic weight noise during multilayer perceptron training: fault tolerance and training improvements , 1993, IEEE Trans. Neural Networks.

[14]  Chuan Wang,et al.  Training neural networks with additive noise in the desired signal , 1999, IEEE Trans. Neural Networks.

[15]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[16]  Christopher M. Bishop,et al.  Current address: Microsoft Research, , 2022 .

[17]  Alberto Tesi,et al.  On the Problem of Local Minima in Backpropagation , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Minoru Fukumi,et al.  A new back-propagation algorithm with coupled neuron , 1991, International 1989 Joint Conference on Neural Networks.

[19]  Jacek M. Zurada,et al.  Deterministic convergence of conjugate gradient method for feedforward neural networks , 2011, Neurocomputing.

[20]  Claudio Moraga,et al.  Multilayer Feedforward Neural Network Based on Multi-valued Neurons (MLMVN) and a Backpropagation Learning Algorithm , 2006, Soft Comput..

[21]  Chi-Chung Cheung,et al.  Magnified gradient function with deterministic weight modification in adaptive learning , 2004, IEEE Transactions on Neural Networks.

[22]  Rudy Setiono,et al.  Use of a quasi-Newton method in a feedforward neural network construction algorithm , 1995, IEEE Trans. Neural Networks.

[23]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[24]  Andrew Chi-Sing Leung,et al.  Objective Functions of Online Weight Noise Injection Training Algorithms for MLPs , 2011, IEEE Transactions on Neural Networks.

[25]  Guozhong An,et al.  The Effects of Adding Noise During Backpropagation Training on a Generalization Performance , 1996, Neural Computation.

[26]  Byung Joo Kim.  Applying Conjugate Gradient Method Based Backpropagation Algorithm to Chemical Processing System , 2017 .

[27]  Ehud D. Karnin,et al.  A simple procedure for pruning back-propagation trained neural networks , 1990, IEEE Trans. Neural Networks.

[28]  Pengfei Huang,et al.  An improved back propagation neural network prediction model for subsurface drip irrigation system , 2017, Comput. Electr. Eng..

[29]  Yong Liu,et al.  Modeling effects of alloying elements and heat treatment parameters on mechanical properties of hot die steel with back-propagation artificial neural network , 2017 .