Perceptron training algorithms designed using discrete-time control Liapunov functions

Perceptrons, proposed in the seminal paper McCulloch-Pitts of 1943, have remained of interest to neural network community because of their simplicity and usefulness in classifying linearly separable data and can be viewed as implementing iterative procedures for ''solving'' linear inequalities. Gradient descent and conjugate gradient methods, normally used for linear equalities, can be used to solve linear inequalities by simple modifications that have been proposed in the literature but not been analyzed completely. This paper applies a recently proposed control-inspired approach to the design of iterative steepest descent and conjugate gradient algorithms for perceptron training in batch mode, by regarding certain parameters of the training/algorithm as controls and then using a control Liapunov technique to choose appropriate values of these parameters.

[1]  Eugenius Kaszkurewicz,et al.  Support vector classifiers via gradient systems with discontinuous righthand sides , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[2]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machines , 2002 .

[3]  Bernhard Schölkopf,et al.  New Support Vector Algorithms , 2000, Neural Computation.

[4]  R. P. Jagadeesh Chandra Bose,et al.  Adaptive conjugate gradient algorithm for perceptron training , 2006, Neurocomputing.

[5]  Mohamad H. Hassoun,et al.  Adaptive Ho-Kashyap rules for perceptron training , 1992, IEEE Trans. Neural Networks.

[6]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[7]  G. Nagaraja,et al.  Solution of linear inequalities: A conjugate gradient algorithm for linear constraints , 1974 .

[8]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[9]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[10]  R. L. Kashyap,et al.  An Algorithm for Linear Inequalities and its Applications , 1965, IEEE Trans. Electron. Comput..

[11]  Nils J. Nilsson,et al.  The Mathematical Foundations of Learning Machines , 1990 .

[12]  Davide Anguita,et al.  Circuital implementation of support vector machines , 1998 .

[13]  Yousef Saad,et al.  Iterative methods for sparse linear systems , 2003 .

[14]  Eugenius Kaszkurewicz,et al.  Control Perspectives on Numerical Algorithms And Matrix Problems (Advances in Design and Control) (Advances in Design and Control 10) , 2006 .

[15]  Amit Bhaya,et al.  Adaptive filtering algorithms designed using control Liapunov functions , 2006, IEEE Signal Processing Letters.

[16]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[17]  C. M. Reeves,et al.  Function minimization by conjugate gradients , 1964, Comput. J..

[18]  Eugenius Kaszkurewicz,et al.  Iterative methods as dynamical systems with feedback control , 2003, 42nd IEEE International Conference on Decision and Control (IEEE Cat. No.03CH37475).