The No-Prop algorithm: A new learning algorithm for multilayer neural networks

A new learning algorithm for multilayer neural networks that we have named No-Propagation (No-Prop) is hereby introduced. With this algorithm, the weights of the hidden-layer neurons are set and fixed with random values. Only the weights of the output-layer neurons are trained, using steepest descent to minimize mean square error, with the LMS algorithm of Widrow and Hoff. The purpose of introducing nonlinearity with the hidden layers is examined from the point of view of Least Mean Square Error Capacity (LMS Capacity), which is defined as the maximum number of distinct patterns that can be trained into the network with zero error. This is shown to be equal to the number of weights of each of the output-layer neurons. The No-Prop algorithm and the Back-Prop algorithm are compared. Our experience with No-Prop is limited, but from the several examples presented here, it seems that the performance regarding training and generalization of both algorithms is essentially the same when the number of training patterns is less than or equal to LMS Capacity. When the number of training patterns exceeds Capacity, Back-Prop is generally the better performer. But equivalent performance can be obtained with No-Prop by increasing the network Capacity by increasing the number of neurons in the hidden layer that drives the output layer. The No-Prop algorithm is much simpler and easier to implement than Back-Prop. Also, it converges much faster. It is too early to definitively say where to use one or the other of these algorithms. This is still a work in progress.

[1]  Hongming Zhou,et al.  Extreme Learning Machine for Regression and Multiclass Classification , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[2]  Bernard Widrow.  Adaptive inverse control , 1990, Defense, Security, and Sensing.

[3]  Philipp Slusallek,et al.  Introduction to real-time ray tracing , 2005, SIGGRAPH Courses.

[4]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[5]  Andrew Beng Jin Teoh,et al.  An online AUC formulation for binary classification , 2012, Pattern Recognit..

[6]  Bernard Widrow,et al.  Adaptive Signal Processing , 1985 .

[7]  Joseph W. Goodman,et al.  Backpropagation and Its Application to Handwritten Signature Verification , 1988, NIPS.

[8]  Guang-Bin Huang,et al.  Extreme learning machine: a new learning scheme of feedforward neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[9]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[10]  Chee Kheong Siew,et al.  Incremental extreme learning machine with fully complex hidden nodes , 2008, Neurocomputing.

[11]  Bernard Widrow,et al.  30 years of adaptive neural networks: perceptron, Madaline, and backpropagation , 1990, Proc. IEEE.

[12]  Luciano Boquete,et al.  EOG-based eye movements codification for human computer interaction , 2012, Expert Syst. Appl..

[13]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[14]  Kwontaeg Choi,et al.  Incremental face recognition for large-scale social network services , 2012, Pattern Recognit..

[15]  Bernard Widrow,et al.  Adaptive switching circuits , 1988 .

[16]  Chee Kheong Siew,et al.  Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden Nodes , 2006, IEEE Transactions on Neural Networks.

[17]  Guang-Bin Huang,et al.  Convex incremental extreme learning machine , 2007, Neurocomputing.