Multilayer Perceptrons: Other Learning Techniques

Training of feedforward networks can be viewed as an unconstrained optimization problem. BP is slow to converge when the error surface is flat along a weight dimension. Second-order optimization techniques have a strong theoretical basis and provide significantly faster convergence.

[1]  Kenji Fukumizu,et al.  Adaptive Method of Realizing Natural Gradient Learning for Multilayer Perceptrons , 2000, Neural Computation.

[2]  Samir Kouro,et al.  Unidimensional Modulation Technique for Cascaded Multilevel Converters , 2009, IEEE Transactions on Industrial Electronics.

[3]  Hao Yu,et al.  Neural Network Learning Without Backpropagation , 2010, IEEE Transactions on Neural Networks.

[4]  Andrew Chi-Sing Leung,et al.  On the Kalman filtering method in neural network training and pruning , 1999, IEEE Trans. Neural Networks.

[5]  Etienne Barnard,et al.  Optimization for training neural nets , 1992, IEEE Trans. Neural Networks.

[6]  John L. Nazareth.  Differentiable optimization and equation solving , 2003 .

[7]  Michael T. Manry,et al.  A neural network training algorithm utilizing multiple sets of linear equations , 1996, Conference Record of The Thirtieth Asilomar Conference on Signals, Systems and Computers.

[8]  Tom Heskes,et al.  On Natural Learning and Pruning in Multilayered Perceptrons , 2000, Neural Computation.

[9]  Antanas Verikas,et al.  Training neural networks by stochastic optimisation , 2000, Neurocomputing.

[10]  Francesco Piazza,et al.  On the complex backpropagation algorithm , 1992, IEEE Trans. Signal Process..

[11]  Francesco Palmieri,et al.  MEKA-a fast, local algorithm for training feedforward neural networks , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[12]  M. N. Shanmukha Swamy,et al.  Complex EKF neural network for adaptive equalization , 2000, 2000 IEEE International Symposium on Circuits and Systems. Emerging Technologies for the 21st Century. Proceedings (IEEE Cat No.00CH36353).

[13]  Robert M. Burton,et al.  Event-dependent control of noise enhances learning in neural networks , 1992, Neural Networks.

[14]  Hao Yu,et al.  Improved Computation for Levenberg–Marquardt Training , 2010, IEEE Transactions on Neural Networks.

[15]  Chin-Teng Lin,et al.  A second-order learning algorithm for multilayer networks based on block Hessian matrix , 1998, Neural Networks.

[16]  Chao Zhang,et al.  Convergence of Batch Split-Complex Backpropagation Algorithm for Complex-Valued Neural Networks , 2009 .

[17]  Sammy Siu,et al.  Sensitivity Analysis of the Split-Complex Valued Multilayer Perceptron Due to the Errors of the i.i.d. Inputs and Weights , 2007, IEEE Transactions on Neural Networks.

[18]  Stavros J. Perantonis,et al.  Two highly efficient second-order algorithms for training feedforward networks , 2002, IEEE Trans. Neural Networks.

[19]  Kwok-Wo Wong,et al.  Generalized RLS approach to the training of neural networks , 2006, IEEE Trans. Neural Networks.

[20]  Cris Koutsougeras,et al.  Complex domain backpropagation , 1992 .

[21]  Hideaki Sakai,et al.  A real-time learning algorithm for a multilayered neural network based on the extended Kalman filter , 1992, IEEE Trans. Signal Process..

[22]  Jaewook Lee.  Attractor-based trust-region algorithm for efficient training of multilayer perceptrons , 2003 .

[23]  Léon Personnaz,et al.  A recursive algorithm based on the extended Kalman filter for the training of feedforward neural models , 1998, Neurocomputing.

[24]  Stavros J. Perantonis,et al.  Training feedforward neural networks with the Dogleg method and BFGS Hessian updates , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[25]  Nikolai S. Rubanov.  The layer-wise method and the backpropagation hybrid approach to learning a feedforward neural network , 2000, IEEE Trans. Neural Networks Learn. Syst..

[26]  H.S.M. Beigi.  Neural network learning through optimally conditioned quadratically convergent methods requiring NO LINE SEARCH , 1993, Proceedings of 36th Midwest Symposium on Circuits and Systems.

[27]  Mark E. Oxley,et al.  Comparative Analysis of Backpropagation and the Extended Kalman Filter for Training Multilayer Perceptrons , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Sharad Singhal,et al.  Training feed-forward networks with the extended Kalman algorithm , 1989, International Conference on Acoustics, Speech, and Signal Processing,.

[29]  Danilo P. Mandic,et al.  A normalised complex backpropagation algorithm , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[30]  Paramasivan Saratchandran,et al.  A new learning algorithm with logarithmic performance index for complex-valued neural networks , 2009, Neurocomputing.

[31]  Daesik Hong,et al.  Nonlinear blind equalization schemes using complex-valued multilayer feedforward neural networks , 1998, IEEE Trans. Neural Networks.

[32]  Gregory J. Wolff,et al.  Optimal Brain Surgeon and general network pruning , 1993, IEEE International Conference on Neural Networks.

[33]  Stefano Fanelli,et al.  A new class of quasi-Newtonian methods for optimal learning in MLP-networks , 2003, IEEE Trans. Neural Networks.

[34]  George W. Irwin,et al.  Fast parallel off-line training of multilayer perceptrons , 1997, IEEE Trans. Neural Networks.

[35]  Michael Fairbank,et al.  Efficient Calculation of the Gauss-Newton Approximation of the Hessian Matrix in Neural Networks , 2012, Neural Computation.

[36]  Andrew Luk,et al.  Fast Convergent Generalized Back-Propagation Algorithm with Constant Learning Rate , 2004, Neural Processing Letters.

[37]  Okyay Kaynak,et al.  Computing Gradient Vector and Jacobian Matrix in Arbitrarily Connected Neural Networks , 2008, IEEE Transactions on Industrial Electronics.

[38]  Vijanth S. Asirvadam,et al.  A memory optimal BFGS neural network training algorithm , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[39]  Leszek Rutkowski,et al.  A fast training algorithm for neural networks , 1998 .

[40]  Chris Bishop,et al.  Exact Calculation of the Hessian Matrix for the Multilayer Perceptron , 1992, Neural Computation.

[41]  Sagar V. Kamarthi,et al.  Accelerating neural network training using weight extrapolations , 1999, Neural Networks.

[42]  Roberto Battiti,et al.  First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method , 1992, Neural Computation.

[43]  José R. Dorronsoro,et al.  Natural conjugate gradient training of multilayer perceptrons , 2006, Neurocomputing.

[44]  A. Perry.  A Modified Conjugate Gradient Algorithm for Unconstrained Nonlinear Optimization , 1975 .

[45]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[46]  David Zhang,et al.  Parameter by Parameter Algorithm for Multilayer Perceptrons , 2006, Neural Processing Letters.

[47]  Paolo Campolucci,et al.  Complex-valued neural networks with adaptive spline activation function for digital-radio-links nonlinear equalization , 1999, IEEE Trans. Signal Process..

[48]  Eugenius Kaszkurewicz,et al.  Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method , 2004, Neural Networks.

[49]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[50]  Tülay Adali,et al.  Approximation by Fully Complex Multilayer Perceptrons , 2003, Neural Computation.

[51]  C. M. Reeves,et al.  Function minimization by conjugate gradients , 1964, Comput. J..

[52]  Jorge J. Moré,et al.  The Levenberg-Marquardt algo-rithm: Implementation and theory , 1977 .

[53]  George W. Irwin,et al.  A Variable Memory Quasi-Newton Training Algorithm , 1999, Neural Processing Letters.

[54]  Avinoam Perry,et al.  Technical Note - A Modified Conjugate Gradient Algorithm , 1978, Oper. Res..

[55]  Michael T. Manry,et al.  Fast training of neural networks for remote sensing , 1994 .

[56]  Tohru Nitta,et al.  An Extension of the Back-Propagation Algorithm to Complex Numbers , 1997, Neural Networks.

[57]  Kiyoshi Nishiyama,et al.  H∞-learning of layered neural networks , 2001, IEEE Trans. Neural Networks.

[58]  John S. Shawe-Taylor,et al.  Linear programming algorithm for neural networks , 1990, Neural Networks.

[59]  Farid U. Dowla,et al.  Backpropagation Learning for Multilayer Feed-Forward Neural Networks Using the Conjugate Gradient Method , 1991, Int. J. Neural Syst..

[60]  Lijun Liu,et al.  Convergence Analysis of Three Classes of Split-Complex Gradient Algorithms for Complex-Valued Recurrent Neural Networks , 2010, Neural Computation.

[61]  Mohammad Bagher Menhaj,et al.  Training feedforward networks with the Marquardt algorithm , 1994, IEEE Trans. Neural Networks.

[62]  Martin Fodslette Møller,et al.  A scaled conjugate gradient algorithm for fast supervised learning , 1993, Neural Networks.

[63]  Yutaka Fukuoka,et al.  A modified back-propagation method to avoid false local minima , 1998, Neural Networks.

[64]  Shixin Cheng,et al.  Dynamic learning rate optimization of the backpropagation algorithm , 1995, IEEE Trans. Neural Networks.

[65]  Mahmood R. Azimi-Sadjadi,et al.  Fast learning process of multilayer neural networks using recursive least squares method , 1992, IEEE Trans. Signal Process..

[66]  Lawrence D. Jackel,et al.  Handwritten Digit Recognition with a Back-Propagation Network , 1989, NIPS.

[67]  Jonas Sjöberg,et al.  Efficient training of neural nets for nonlinear adaptive filtering using a recursive Levenberg-Marquardt algorithm , 2000, IEEE Trans. Signal Process..

[68]  Roberto Battiti,et al.  Learning with first, second, and no derivatives: A case study in high energy physics , 1994, Neurocomputing.

[69]  Andrew Chi-Sing Leung,et al.  Two regularizers for recursive least squared algorithms in feedforward multilayered neural networks , 2001, IEEE Trans. Neural Networks.

[70]  M. J. D. Powell,et al.  Restart procedures for the conjugate gradient method , 1977, Math. Program..

[71]  Yixin Chen,et al.  TREAT: a trust-region-based error-aggregated training algorithm for neural networks , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[72]  L. Dixon.  Conjugate Gradient Algorithms: Quadratic Termination without Linear Searches , 1975 .

[73]  C. Charalambous,et al.  Conjugate gradient algorithm for efficient training of artifi-cial neural networks , 1990 .

[74]  Kazumi Saito,et al.  Partial BFGS Update and Efficient Step-Length Calculation for Three-Layer Neural Networks , 1997, Neural Computation.

[75]  Patrick van der Smagt.  Minimisation methods for training feedforward neural networks , 1994, Neural Networks.

[76]  Jukka Saarinen,et al.  Complex RPROP-algorithm for neural network equalization of GSM data bursts , 2004, Neurocomputing.

[77]  Edward W. Kamen,et al.  A local linearized least squares algorithm for training feedforward neural networks , 2000, IEEE Trans. Neural Networks Learn. Syst..

[78]  Michael T. Manry,et al.  An efficient hidden layer training method for the multilayer perceptron , 2006, Neurocomputing.

[79]  Okyay Kaynak,et al.  An algorithm for fast convergence in training neural networks , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[80]  E. Polak,et al.  Computational methods in optimization : a unified approach , 1972 .

[81]  George Cybenko,et al.  Ill-Conditioning in Neural Network Training Problems , 1993, SIAM J. Sci. Comput..

[82]  Nazif Tepedelenlioglu,et al.  A fast new algorithm for training feedforward neural networks , 1992, IEEE Trans. Signal Process..

[83]  Thomas P. Vogl,et al.  Rescaling of variables in back propagation learning , 1991, Neural Networks.

[84]  M Maarten Steinbuch,et al.  Uncertainty modelling and structured singular-value computation applied to an electromechanical system , 1992 .

[85]  Chin-Sung Liu,et al.  Quadratic optimization method for multilayer neural networks with local error-backpropagation , 1999, Int. J. Syst. Sci..

[86]  Frank Bärmann,et al.  On a class of efficient learning algorithms for neural networks , 1992, Neural Networks.

[87]  Theodoula N. Grapsa,et al.  Self-scaled conjugate gradient training algorithms , 2009, Neurocomputing.

[88]  Marc Toussaint,et al.  Rprop Using the Natural Gradient , 2005 .

[89]  Henry Leung,et al.  The complex backpropagation algorithm , 1991, IEEE Trans. Signal Process..

[90]  Mostafa Kaveh,et al.  Conjugate gradient learning algorithms for multilayer perceptrons , 1989, Proceedings of the 32nd Midwest Symposium on Circuits and Systems,.

[91]  Youmin Zhang,et al.  A Fast U-d Factorization-based Learning Algorithm with Applications to Nonlinear System Modeling and Identification , 2022 .

[92]  Paul Kang-Hoh Phua,et al.  Parallel nonlinear optimization techniques for training neural networks , 2003, IEEE Trans. Neural Networks.

[93]  Sammy Siu,et al.  Analysis of the Initial Values in Split-Complex Backpropagation Algorithm , 2008, IEEE Transactions on Neural Networks.

[94]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[95]  Bhaskar D. Rao,et al.  A generalized learning paradigm exploiting the structure of feedforward neural networks , 1996, IEEE Trans. Neural Networks.

[96]  Ya-Xiang Yuan,et al.  A Nonlinear Conjugate Gradient Method with a Strong Global Convergence Property , 1999, SIAM J. Optim..

[97]  Lee A. Feldkamp,et al.  Decoupled extended Kalman filter training of feedforward layered networks , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[98]  Ying Zhang,et al.  Boundedness and Convergence of Split-Complex Back-Propagation Algorithm with Momentum and Penalty , 2013, Neural Processing Letters.

[99]  Kwok-Wo Wong,et al.  A pruning method for the recursive least squared algorithm , 2001, Neural Networks.

[100]  Don R. Hush,et al.  Error surfaces for multilayer perceptrons , 1992, IEEE Trans. Syst. Man Cybern..

[101]  Martin Fieldhouse,et al.  Review: Book review: Linear programming , 1964, Comput. J..

[102]  David F. Shanno,et al.  Conjugate Gradient Methods with Inexact Searches , 1978, Math. Oper. Res..

[103]  S. Ergezinger,et al.  An accelerated learning algorithm for multilayer perceptrons: optimization layer by layer , 1995, IEEE Trans. Neural Networks.

[104]  Tülay Adali,et al.  Fully Complex Multi-Layer Perceptron Network for Nonlinear Signal Processing , 2002, J. VLSI Signal Process..