An efficient learning algorithm for finding multiple solutions based on fixed-point homotopy method

This paper describes an efficient learning algorithm based on fixed-point homotopy method. The proposed algorithm has the ability to train the neural networks with high success rates for the initial guesses compared with other typical second-order training algorithms. Furthermore, the method proposed here not only has the widely convergent property but also find out multiple solutions. The validity of the proposed algorithm for the standard multilayer neural networks is demonstrated through the computer simulations. As a result, it is confirmed that our algorithm is efficient and practical for the learning of the multilayer feedforward neural networks.

[1]  T.,et al.  Training Feedforward Networks with the Marquardt Algorithm , 2004 .

[2]  Andrew Chi-Sing Leung,et al.  Two regularizers for recursive least squared algorithms in feedforward multilayered neural networks , 2001, IEEE Trans. Neural Networks.

[3]  H. Asai,et al.  An efficient learning algorithm with second-order convergence for multilayer neural networks , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[4]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[5]  Virginia L. Stonick,et al.  Homotropy approaches for the analysis and solution of neural network and other nonlinear systems of equations , 1995 .

[6]  C. Tomita.  Note on learning algorithm based on Newton homotopy method for feedforward neural networks , 2004 .

[7]  Roberto Battiti,et al.  First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method , 1992, Neural Computation.

[8]  John Moody,et al.  Note on generalization, regularization and architecture selection in nonlinear learning systems , 1991, Neural Networks for Signal Processing Proceedings of the 1991 IEEE Workshop.

[9]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[10]  Yasuaki Inoue,et al.  An efficient algorithm for finding multiple DC solutions based onthe SPICE-oriented Newton homotopy method , 2002, IEEE Trans. Comput. Aided Des. Integr. Circuits Syst..

[11]  Geoffrey E. Hinton,et al.  Learning representations of back-propagation errors , 1986 .

[12]  Virginia L. Stonick,et al.  On a natural homotopy between linear and nonlinear single-layer networks , 1996, IEEE Trans. Neural Networks.

[13]  Naoki Kinoshita,et al.  A new learning algorithm without explicit error backpropagation , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[14]  Yasuaki Inoue,et al.  An efficient algorithm for finding multiple DC solutions based on Spice oriented Newton homotopy method , 2001, ISCAS 2001. The 2001 IEEE International Symposium on Circuits and Systems (Cat. No.01CH37196).

[15]  Kiyotaka Yamamura,et al.  A fixed-point homotopy method for solving modified nodal equations , 1999 .