New globally convergent training scheme based on the resilient propagation algorithm

In this paper, a new globally convergent modification of the Resilient Propagation-Rprop algorithm is presented. This new addition to the Rprop family of methods builds on a mathematical framework for the convergence analysis that ensures that the adaptive local learning rates of the Rprop's schedule generate a descent search direction at each iteration. Simulation results in six problems of the PROBEN1 benchmark collection show that the globally convergent modification of the Rprop algorithm exhibits improved learning speed, and compares favorably against the original Rprop and the Improved Rprop, a recently proposed Rrpop modification.

[1]  M. N. Vrahatisa,et al.  A class of gradient unconstrained minimization algorithms with adaptive stepsize , 1999 .

[2]  Xiaohui Liu,et al.  Classification of Protein Localisation Patterns via Supervised Neural Network Learning , 2003, IDA.

[3]  George D. Magoulas,et al.  Effective Backpropagation Training with Variable Stepsize , 1997, Neural Networks.

[4]  B. André,et al.  A genomic view of yeast membrane transporters. , 2001, Current opinion in cell biology.

[5]  George D. Magoulas,et al.  Improving the Convergence of the Backpropagation Algorithm Using Learning Rate Adaptation Methods , 1999, Neural Computation.

[6]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[7]  Martin Fodslette Møller,et al.  A scaled conjugate gradient algorithm for fast supervised learning , 1993, Neural Networks.

[8]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[9]  Lutz Prechelt,et al.  PROBEN 1 - a set of benchmarks and benchmarking rules for neural network training algorithms , 1994 .

[10]  M. N. Vrahatisa,et al.  From linear to nonlinear iterative methods , 2003 .

[11]  Lalit M. Patnaik,et al.  Target detection through image processing and resilient propagation algorithms , 2000, Neurocomputing.

[12]  M. Kanehisa,et al.  Expert system for predicting protein localization sites in gram‐negative bacteria , 1991, Proteins.

[13]  Martin A. Riedmiller,et al.  A direct adaptive method for faster backpropagation learning: the RPROP algorithm , 1993, IEEE International Conference on Neural Networks.

[14]  T.,et al.  Training Feedforward Networks with the Marquardt Algorithm , 2004 .

[15]  Jorge Nocedal,et al.  Theory of algorithms for unconstrained optimization , 1992, Acta Numerica.

[16]  Raúl Rojas,et al.  Speeding-up backpropagation-a comparison of orthogonal techniques , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[17]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[18]  Christian Igel,et al.  Empirical evaluation of the improved Rprop learning algorithms , 2003, Neurocomputing.

[19]  G. Yarrington.  Molecular Cell Biology , 1987, The Yale Journal of Biology and Medicine.

[20]  P. Wolfe.  Convergence Conditions for Ascent Methods. II: Some Corrections , 1971 .

[21]  Roberto Battiti,et al.  First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method , 1992, Neural Computation.

[22]  Philip E. Gill,et al.  Practical optimization , 1981 .

[23]  Tamás D. Gedeon,et al.  Simulated annealing and weight decay in adaptive learning: the SARPROP algorithm , 1998, IEEE Trans. Neural Networks.

[24]  P. Wolfe.  Convergence Conditions for Ascent Methods. II , 1969 .

[25]  N. W. Davis,et al.  The complete genome sequence of Escherichia coli K-12. , 1997, Science.

[26]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[27]  M. Kanehisa,et al.  A knowledge base for predicting protein localization sites in eukaryotic cells , 1992, Genomics.

[28]  Paul Horton,et al.  Better Prediction of Protein Cellular Localization Sites with the it k Nearest Neighbors Classifier , 1997, ISMB.

[29]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[30]  Monica Riley,et al.  Physiological genomics of Escherichia coli protein families. , 2002, Physiological genomics.

[31]  G. Kane.  Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol 1: Foundations, vol 2: Psychological and Biological Models , 1994 .

[32]  BattitiRoberto.  First- and second-order methods for learning , 1992 .